I am trying to develop an android app to read emotion in real time using OpenCV and Tensoflow Lite. I have a CameraActivity that implements the camera access in my project and it also calls a facialEmotionRecognition class where the Intepreter is implemented. Everyhing looks fine and no eror messages on the code but when i run the project the camera opens but on detection of any image or human face it shuts down the app with the following error on logcat:
FATAL EXCEPTION: Thread-7
java.lang.IllegalArgumentException: Internal error: Failed to run on the given Interpreter: Can not open OpenCL library on this device - dlopen failed: library “libOpenCL.so” not found
Falling back to OpenGL
TfLiteGpuDelegate Invoke: GpuDelegate must run on the same thread where it was initialized.
Node number 68 (TfLiteGpuDelegateV2) failed to invoke.
at org.tensorflow.lite.NativeInterpreterWrapper.run(Native Method)
at org.tensorflow.lite.NativeInterpreterWrapper.run(NativeInterpreterWrapper.java:163)
at org.tensorflow.lite.Interpreter.runForMultipleInputsOutputs(Interpreter.java:360)
at org.tensorflow.lite.Interpreter.run(Interpreter.java:319)
uk.ac.tees.a0278818.gaugehealth.emotionfile.FacialExpressionRecognition.recognizeImage(FacialExpressionRecognition.java:130)
uk.ac.tees.a0278818.gaugehealth.emotionfile.CameraActivity$1.onCameraFrame(CameraActivity.java:53)
org.opencv.android.CameraBridgeViewBase.deliverAndDrawFrame(CameraBridgeViewBase.java:509)
at org.opencv.android.JavaCameraView$CameraWorker.run(JavaCameraView.java:373)
at java.lang.Thread.run(Thread.java:1012)
Blow id my code :
public class FacialExpressionRecognition {
private static Interpreter interpreter;
private static int INPUT_SIZE;
private static int height = 0;
private static int width = 0;
private GpuDelegate gpuDelegate = null;
private static CascadeClassifier cascadeClassifier;
FacialExpressionRecognition(AssetManager assetManager, Context context, String modelPath, int inputSize) throws IOException {
INPUT_SIZE = inputSize;
// Set GPU for interpreter
Interpreter.Options options = new Interpreter.Options();
gpuDelegate = new GpuDelegate();
// Add Gpu to options
options.addDelegate(gpuDelegate);
// Now set number of threads to options
options.setNumThreads(8); // This should be set according to your phone
interpreter = new Interpreter(loadModelFile(assetManager, modelPath), options);
// If model is loaded print
Log.d(“facial_expression”, “model is loaded”);
// Now let's load the haarcascade classifier
try {
// define input stream to read classifier
InputStream is = context.getResources().openRawResource(R.raw.haarcascade_frontalface_alt);
// Create a folder
File cascadeDir = context.getDir("cascade", Context.MODE_PRIVATE);
// Now create a file in that folder
File mCascadeFile = new File(cascadeDir, "haarcascade_frontalface_alt");
// Now define output stream to transfer data to file we created
FileOutputStream os = new FileOutputStream(mCascadeFile);
// Now let's create buffer to store byte
byte[] buffer = new byte[4096];
int byteRead;
// read byte in while loop
// when it read -1 that means no data to read
while ((byteRead = is.read(buffer)) != -1){
// writing on mCascade file
os.write(buffer, 0, byteRead);
} // close input and output stream
is.close();
os.close();
cascadeClassifier = new CascadeClassifier(mCascadeFile.getAbsolutePath());
// if cascade file is loaded print
Log.d("facial_expression", "Classifier is loaded");
}
catch (IOException e){
e.printStackTrace();
}
}
public static Mat recognizeImage(Mat mat_image){
// Before predicting our image is not aligned properly
// we have to rotate it by 90 degrees for proper prediction
Core.flip(mat_image.t(), mat_image, 1);
// Start with our process; convert mat_image to gray scale image
Mat grayscaleImage = new Mat();
Imgproc.cvtColor(mat_image, grayscaleImage, Imgproc.COLOR_RGB2GRAY);
// Ste height and width
height = grayscaleImage.height();
width = grayscaleImage.width();
// define minimum height of face in original image
// below this size no face in original image will show
int absoluteFaceSize = (int)(height*0.1);
// now create MatOfRect to store face
MatOfRect faces = new MatOfRect();
// Check if cascadeClassifier is loaded or not
if (cascadeClassifier != null) {
cascadeClassifier.detectMultiScale(grayscaleImage, faces, 1.1, 2, 2,
new Size(absoluteFaceSize, absoluteFaceSize), new Size());
}
// now convert it to an array
Rect[] faceArray = faces.toArray();
// loop through each face
for (int i = 0; i < faceArray.length; i++){
// if you want to draw rectangle around the face
// input/output starting point and ending point color R G B alpha thickness
Imgproc.rectangle(mat_image, faceArray[i].tl(), faceArray[i].br(), new Scalar(0, 255, 0, 255), 2);
// Now crop face from original frame and grayscaleImage
Rect roi = new Rect((int)faceArray[i].tl().x,(int)faceArray[i].tl().y,
((int)faceArray[i].br().x)-(int)(faceArray[i].tl().x),
((int)faceArray[i].br().y)-(int)(faceArray[i].tl().y));
Mat cropped_rgba = new Mat(mat_image, roi);
// now convert cropped_rgba to bitmap
Bitmap bitmap = null;
bitmap = Bitmap.createBitmap(cropped_rgba.cols(), cropped_rgba.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(cropped_rgba, bitmap);
// resize bitmap to (48, 48)
Bitmap scaledBitmap = Bitmap.createScaledBitmap(bitmap, 48, 48, false);
// now convert scaledBitmap to byteBuffer
ByteBuffer byteBuffer = convertBitmapToByteBuffer(scaledBitmap);
// now create an object to hold output
float[][] emotion = new float[1][1];
// now predict with bytebuffer as an input emotion as an output
interpreter.run(byteBuffer, emotion);
// if emotion is recognized print value of it
// define float value of emotion
float emotion_v = (float)Array.get(Array.get(emotion, 0), 0);
Log.d("facial_expression", "Output: " + emotion_v);
// create a function that return text emotion
String emotion_s = get_emotion_text(emotion_v);
// now put text on original frame(mat_image)
//
Imgproc.putText(mat_image, emotion_s + " (" + emotion_v + ")",
new Point((int)faceArray[i].tl().x + 10, (int)faceArray[i].tl().y + 20),
1, 1.5, new Scalar(0, 0, 255, 150), 2);
}
// After prediction rotate mat_image -90 degrees
Core.flip(mat_image.t(), mat_image, 0);
return mat_image;
}
private static String get_emotion_text(float emotionV) {
// create an empty string
String val = "";
// use if statement to determine val
if (emotionV >= 0 & emotionV < 0.5){
val = "Surprise";
} else if (emotionV >= 0.5 & emotionV < 1.5) {
val = "Fear";
} else if (emotionV >= 1.5 & emotionV < 2.5) {
val = "Angry";
}else if (emotionV >= 2.5 & emotionV < 3.5) {
val = "Neutral";
} else if (emotionV >= 3.5 & emotionV < 4.5) {
val = "Sad";
} else if (emotionV >= 4.5 & emotionV < 5.5) {
val = "Disgust";
} else {
val = "Happy";
}
return val;
}
private static ByteBuffer convertBitmapToByteBuffer(Bitmap scaledBitmap) {
ByteBuffer byteBuffer;
int size_image = INPUT_SIZE; //48
byteBuffer = ByteBuffer.allocateDirect(4 * 1*size_image * size_image*3);
// byteBuffer=ByteBuffer.allocateDirect(41size_imagesize_image3);
// 4 is multiplied for float input
//3 is multiplied for rgb
byteBuffer.order(ByteOrder.nativeOrder());
int intValues = new int[size_image*size_image];
scaledBitmap.getPixels(intValues, 0, scaledBitmap.getWidth(), 0, 0, scaledBitmap.getWidth(), scaledBitmap.getHeight());
int pixel = 0;
for (int i = 0; i < size_image; ++i){
for (int j = 0; j < size_image; ++j){
final int val = intValues[pixel++];
// now put float value to bytebuffer
// scale image to convert image from 0-255 to 0-1
byteBuffer.putFloat((((val >> 16) & 0xFF))/255.0f);
byteBuffer.putFloat((((val >> 8) & 0xFF))/255.0f);
byteBuffer.putFloat(((val & 0xFF))/255.0f);
}
}
System.out.println("Position before rewind: " + byteBuffer.position());
// Make sure to reset the position to the beginning of the buffer
byteBuffer.rewind();
System.out.println("Position after rewind: " + byteBuffer.position());
return byteBuffer;
}
private MappedByteBuffer loadModelFile(AssetManager assetManager, String modelPath) throws IOException {
AssetFileDescriptor assetFileDescriptor = assetManager.openFd(modelPath);
FileInputStream inputStream = new FileInputStream(assetFileDescriptor.getFileDescriptor());
FileChannel fileChannel = inputStream.getChannel();
long startOffset = assetFileDescriptor.getStartOffset();
long declaredLength = assetFileDescriptor.getDeclaredLength();
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
}
}
For Camera Activity:
public class CameraActivity extends org.opencv.android.CameraActivity {
private Mat mRgba;
private Mat mGray;
CameraBridgeViewBase cameraBridgeViewBase;
private FacialExpressionRecognition facialExpressionRecognition;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_camera);
cameraBridgeViewBase = findViewById(R.id.camera_view);
cameraBridgeViewBase.setCvCameraViewListener(new CameraBridgeViewBase.CvCameraViewListener2() {
@Override
public void onCameraViewStarted(int width, int height) {
mRgba = new Mat();
mGray = new Mat();
}
@Override
public void onCameraViewStopped() {
mRgba.release();
mGray.release();
}
public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
mGray = inputFrame.gray();
mRgba = inputFrame.rgba();
FacialExpressionRecognition.recognizeImage(mRgba);
return mRgba;
}
});
if(OpenCVLoader.initDebug()){
cameraBridgeViewBase.enableView();
}
// This will load Cascade Classifier and the model
// This will only happen one time when the CameraActivity is started
try {
int inputSize = 48;
facialExpressionRecognition = new FacialExpressionRecognition(getAssets(), CameraActivity.this,
"model300.tflite", inputSize);
} catch (IOException e){
e.printStackTrace();
}
// getPermission();
}
@Override
protected void onResume() {
super.onResume();
cameraBridgeViewBase.enableView();
}
@Override
protected void onDestroy() {
super.onDestroy();
cameraBridgeViewBase.disableView();
}
@Override
protected void onPause() {
super.onPause();
cameraBridgeViewBase.disableView();
}
@Override
protected List<? extends CameraBridgeViewBase> getCameraViewList() {
return Collections.singletonList(cameraBridgeViewBase);
}
@Override
public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
super.onRequestPermissionsResult(requestCode, permissions, grantResults);
// Ensure that this result is for the camera permission request
if (requestCode == 101 && grantResults.length > 0 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
// The camera permission was granted, enable the camera view
cameraBridgeViewBase.enableView();
}
}
}
Please, any assistance will be appreciated as i have been stuck with this for weeks and my deadline is closing in. Thank you