TFX Object Detection

smedegaard · November 11, 2021, 8:26am

@tensorflow_team

I’ve been reading up on TFX and wanted to make a pipeline for object detection, using an “off-the-shelf” model from Model Hub or the object detection API. I’ve also had a look at the TF Lite model builder.

But (as far as I can see) all of these options has a snag that makes it unusable in the context of a TFX pipeline.

The closest I’ve gotten is with a Efficientdet feature-vector from Model Hub:

My question:
Is there an easy way to retrain a Tensorflow object detection model in a TFX pipeline at this moment in time?

regards,
Anders

Ashwini_Gadag · November 18, 2021, 9:19am

Here is the sample example on image classification for TFX.

Train and serve a TensorFlow model with TensorFlow Serving

smedegaard · November 18, 2021, 9:54am

Yes, I’m aware of that. This question is about object detection.

robertcrowe · November 18, 2021, 8:16pm

Which part of object detection is sufficiently different from image classification that you’re having problems?

smedegaard · November 19, 2021, 4:20pm

Hi @Robert_Crowe,
Thanks for reaching out on this issue.

My main problem is that I have not succeeded in taking a pre-built model from either Tensorflow Hub or Tensorflow Object Detection API, or anyother place.

As stated on another thread on this forum:
https://tensorflow-prod.ospodiscourse.com/t/tensorflow-hub-transfer-learning-for-object-detection/4952/8

the Object Detection fine tuning story on TFHub is not perfect. Most of the OD models that were published don’t have the proper api to be fine tuned easily (like the image classification ones you mentioned).

As you know the TFX trainer module expects the function _build_keras_model() to return a compiled Keras model.

Model HUB approach

If I’m not mistaken, I can’t use tf.keras.Sequential() (as in the above tutorial) because there is more than one output from object detection models. The class, and the box.

I tried a naive approach with the functional API:

    img_inputs = keras.Input(shape=IMAGE_SIZE + (3,))
    hub_model = hub.KerasLayer(model_handle, trainable=do_fine_tuning)
    outputs = hub_model(img_inputs)
    model = keras.Model(inputs=img_inputs, outputs=outputs, name="obj_det_model")

    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
    model.compile(optimizer="adam", loss=loss_fn, metrics=["accuracy"])

This got me a bit further, but will not work because of the loss function. I need a combination of the Huber and Focal functions. But implementing the loss function is a bit over my level. Even if I felt comfortable doing it, it’s not quite an “of-the-shelf” solution.

Object Detection API Approach

The Tensorflow object detection API has some very good notebook tutorials but the model that is built is of type object_detection.meta_arcitectures.ssd_mera_arch.<MODEL>Arch. I have not found a way to get from that to a Keras model.

Trying to load a saved model using tf.keras.models.load_model() gives me a <tensorflow.python.saved_model.load.Loader._recreate_base_user_object.<locals>._UserObject not a keras model.

Related(?) Github issue github.com/tensorflow/tensorflow/issues/33870

Training from scratch

I’m currently following the instructions on training a model from scratch with the object detection API.

Recap

So I have tried quite a few things. I don’t see an easy way to take a predefined object detection model and use it in a TFX pipeline. I would be happy to be proven wrong with an example.

Regards,
Anders

smedegaard · November 19, 2021, 7:07pm

Update

I finished training a version of Centerdet with the Object Detection API. It produces checkpoints, no saved model. I don’t know how to convert from checkpoints to the TF >2.5 version of saved_model.

AutoML

I’ve also tried the AutoML repository. But it also produces the old version of saved_model. (there is no keras_metadata.pb file)

robertcrowe · November 29, 2021, 7:01pm

After training, are you calling model.save()?

smedegaard · November 29, 2021, 7:50pm

AutoML uses the TF v1 method b = tf.saved_model.Builder(output_dir)

https://github.com/google/automl/blob/39c39e5b635f73e86082bff3deddefecfbc0a123/efficientdet/model_inspect.py#L338

Object Detection API
Also seems to use the deprecated tf.saved_model.save

github.com/tensorflow/models/research/object_detection/exporter_lib_v2.py#L278

Model Hub

I have not come so far the I could call model.save() on a trained model.

jesusbasea · December 2, 2021, 8:55am

Hello. I’m also interested in integrate Object Detection API in TFX.
More in general, is it possible using tfx trainer with custom trainer loop where there is not a compiled keras model?? ( Writing a training loop from scratch | TensorFlow Core)

Thanks!!

vighneshbirodkar · December 14, 2021, 4:35pm

@smedegaard As you have found out exporter_main_v2 is the script we recommend for exporting your model.

Also, all the meta arch (like SSD etc) inherit from a keras Layer. This may be of use to you

github.com

tensorflow/models/blob/master/research/object_detection/core/model.py#L71


      
          import abc
          import six
          import tensorflow.compat.v1 as tf
          
          from object_detection.core import standard_fields as fields
          
          
          # If using a new enough version of TensorFlow, detection models should be a
          # tf module or keras model for tracking.
          try:
            _BaseClass = tf.keras.layers.Layer
          except AttributeError:
            _BaseClass = object
          
          
          class DetectionModel(six.with_metaclass(abc.ABCMeta, _BaseClass)):
            """Abstract base class for detection models.
          
            Extends tf.Module to guarantee variable tracking.
            """

Also note that an OD API model can be fine tuned using any of the checkpoints from model zoo, does that work for you ?

I was not aware of tf.saved_model.save being deprecated ? Can you send me a pointer to that ?

Unfortunately an object detection models will always have to output multiple values, but we can make some changes to the exporter if it can help your use case.

Thanks
Vighnesh

smedegaard · December 15, 2021, 7:48am

@vighneshbirodkar thank you for your reply. I’ll look into this. I might have been confused about tf.savedmodel.save(). But I’ve run into errors messages complaining about a missingkeras_metadata.pb file, that apparently is only created by the new API?

Thruth be told, I just wanted to make a working TFX pipeline for object detection, but it’s proven to be much more difficult than I expected.

vighneshbirodkar · December 15, 2021, 4:54pm

Could you share the following with us?

The code you used for your TFX pipeline.
The full error log of your program?

Thanks
Vighnesh

smedegaard · December 28, 2021, 10:21am

@vighneshbirodkar thank you for your help.
I’ve had to put this project on pause for now.

If you are interested, the code can be found here. Anders Smedegaard Pedersen / objdet_ml_pipeline · GitLab
But it shows that I’ve been trying a lot of different ways, and I’m not finished yet.

As Crowe hinted to earlier in this thread, it should be trivial to take one of the classifications examples from the Tensorflow website and convert it to an object detection pipeline. But I have to found that to be true. So any example of taking an “off-the-shelve” object det. model and using it in a TFX pipeline would greatly appreciated.

chrisjan.twigt · November 7, 2022, 8:47am

Hi Smedegaard,

Last year I have tried to do what you are doing. To build a TFX pipeline for object detection.
Thinking “how hard can it be”. It is very hard. Be aware that Object Detection can not be tested
in the same way as Classification. Classification is easy: Image in - label out.
Object detection output is much more complex: Image in - N x (label + coordinates) out.
So your model has a very different output signature than for classification problems.

You’ll have to change the way you interact with the tensorflow-serving instance.
You’ll probably want a tensorflow-serving instance with GPU support. Even firing that up
is quite a challenge. TFX does not support it out of the box.

To my knowledge there is not a single code example for what you are getting into.
I remember wasting a lot of time trying to make it work and in the end I failed to export
my model with the right signature to make it work with my tensorflow serving instance.

Topic		Replies	Views
Tensorflow HUB - transfer learning for object detection General Discussion model_garden , tfhub , help_request	12	7118	April 25, 2022
Beginner questions after training my first custom object detection model General Discussion models , help_request	3	1063	November 16, 2021
Object detection with TensorFlow 2 Object detection API General Discussion education	6	1222	February 9, 2024
Object Detection Android App creates error with model from tflite-model-maker (it had worked for many weeks a until a few weeks ago) General Discussion android , tflite , model_maker , help_request	26	11041	May 23, 2023
Does re-training an Object-Detection SSD Network give a different 'format' of model? General Discussion models , keras , model_garden , training	5	1344	July 1, 2021