Image Preprocessing on a tflite model in Kotlin

TensorFlow General Discussion
, ,
1

I trained a custom ResNet image binary classification model and converted it to a .tflite model using

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model("path to my saved_model/") 
tflite_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

I used a python script to add metadata like:

input_normalization = _metadata_fb.ProcessUnitT()
input_normalization.optionsType = (
    _metadata_fb.ProcessUnitOptions.NormalizationOptions)
input_normalization.options = _metadata_fb.NormalizationOptionsT()
input_normalization.options.mean = [127.5]
input_normalization.options.std = [127.5]
input_meta.processUnits = [input_normalization]
input_stats = _metadata_fb.StatsT()
input_stats.max = [255]
input_stats.min = [0]

I tested the resulting .tflite model with the Tensorflow interpreter class, which gave me accurate results.

# Load the TFLite model
interpreter = tf.lite.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()

# Get input and output tensors
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Load and resize an image
img_path = "path_to_image.jpg"
img = Image.open(img_path).resize((512, 512))

img_array = image.img_to_array(img)
img_array = np.expand_dims(img_array, axis=0)

# Run inference
interpreter.set_tensor(input_details[0]['index'], img_array)
interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])

predicted_class = np.argmax(output_data[0])

I’m using Tensorflow/MediaPipe’s sample image classification code for Android. I was able to insert my .tflite model and run the code appropriately. However, the results are always the same label, with a high probability %. When I went back into the metadata and changed the output max and min to 1.0 and 0.0, respectfully, the output label changed but was always the same label, with a high probability. I think something is going wrong with the image preprocessing in the Android code. When I’m testing via Python, I always have the expand the dimensions of the image array before passing the image for inference. I don’t think the Android sample is doing this, nor have I seen any example of anyone else doing it either. How do I make sure that the livestream image is preprocessed correctly before using it for inference?

This is how the sample code is grabbing a frame and setting it up:

        val bitmapBuffer =
            Bitmap.createBitmap(
                imageProxy.width,
                imageProxy.height,
                Bitmap.Config.ARGB_8888
            )

        imageProxy.use {
            bitmapBuffer.copyPixelsFromBuffer(imageProxy.planes[0].buffer)
        }
        imageProxy.close()
        val mpImage = BitmapImageBuilder(bitmapBuffer).build()
        classifyAsync(mpImage, imageProxy.imageInfo.rotationDegrees, frameTime)