Hello All
I am looking for documentation on how TFServing can be made to work with custom devices/accelerators. The link below talks about custom-ops, but not custom devices:
For TensorFlow, we have registered our accelerator and op_kernels supported for that device. Then we work with the ops that get offloaded for that device. TF will load custom-device when we set the device as tf.device(<custom_device_name>) and then it loads the shared object file from third party folder.
We tried to build tf-serving with this locally built TF but ran into few bazel issues. But after linking our library statically, we can at least build tensorflow-model-server. However, we are still seeing the ops being placed on CPU even though some of them are eligible to be placed on the device.
Any suggestions/links/documentation on how TFServing can work with custom devices is appreciated. One suggestion was to use pluggable interface for TF, but not sure if/how that would work with Serving.
Thanks and Regards
Prashantha
Hello,
I am also interested in this question and would appreciate a definitive answer from the team/community as I can’t answer myself.
I am not familiar with TFServing, so when hearing about it after having started the writing of Pluggable Device I was scared that TFServing was using its own computing core/backend and would make the Pluggable Device obsolete in a deployment scenario.
I admit that so far the documentation I read about TFServing participated to the confusion:
out of the box integration with TensorFlow models
What does imnply?
It is clear that there is compatibility with SavedModel, however are we talking about TFServing having a dedicated computing architecture or does it use the traditional Tensorflow runtime behind the scene?
After a quick look at the sources I have the impression that TFServing provides easy server development with a REST API that uses Tensorflow runtime as a backend for running models.
If that is really the case then Pluggable Device would be compatible with TFServing, as long as they are installed on the server that running TFServing.
An official confirmation would be a more satisfying answer 
I was hoping Pluggable Device would be compatible with TFServing, installed both pluggable device first and then TFServing on the same system. However, TFServing is not recognizing the device and no device placement takes place with Serving. It appears that there is no ABI compatibility between Model Server and plugin. I did not have any success in finding documentation/instructions to make it work. I would appreciate any information on serving support for pluggable device. Thanks