# \#serving

**URL:** https://discuss.ai.google.dev/tag/serving/22.md

[Latest](https://discuss.ai.google.dev/latest.md) · [Categories](https://discuss.ai.google.dev/categories.md) · [Tags](https://discuss.ai.google.dev/tags.md)

---

## [Why does the warmup process use only 1 CPU core when loading a model in TensorFlow Serving? How can this be fixed?](https://discuss.ai.google.dev/t/why-does-the-warmup-process-use-only-1-cpu-core-when-loading-a-model-in-tensorflow-serving-how-can-this-be-fixed/42785)

<div class="topic-metadata">

**Author:** [@Bakai\_Zhamgyrchiev](https://discuss.ai.google.dev/u/Bakai_Zhamgyrchiev)\
**Replies:** 0\
**Last updated:** [October 11, 2024, 9:36am UTC](https://discuss.ai.google.dev/t/why-does-the-warmup-process-use-only-1-cpu-core-when-loading-a-model-in-tensorflow-serving-how-can-this-be-fixed/42785 "2024-10-11T09:36:14Z")

</div>

We are facing an issue where loading a new version of a model in TensorFlow Serving takes a long time during the warmup process (in our case, 10 minutes). This is a problem for us. While investigating the issue, we foun…

---

## [Skipping loop optimisation error in tensorflow\_serving](https://discuss.ai.google.dev/t/skipping-loop-optimisation-error-in-tensorflow-serving/27842)

<div class="topic-metadata">

**Author:** [@Prajwal\_1](https://discuss.ai.google.dev/u/Prajwal_1)\
**Replies:** 1\
**Last updated:** [December 28, 2023, 5:22am UTC](https://discuss.ai.google.dev/t/skipping-loop-optimisation-error-in-tensorflow-serving/27842 "2023-12-28T05:22:52Z")

</div>

Hi team, I am getting the following message when testing a model with tensorflow\_serving, can u please help me debug and understand what it means 2022-02-13 23:20:52.460513: I tensorflow\_serving/model\_servers/server\_c…

---

## [Building tensorflow serving 2.4.1](https://discuss.ai.google.dev/t/building-tensorflow-serving-2-4-1/32263)

<div class="topic-metadata">

**Author:** [@leinades](https://discuss.ai.google.dev/u/leinades)\
**Replies:** 1\
**Last updated:** [August 3, 2023, 8:14am UTC](https://discuss.ai.google.dev/t/building-tensorflow-serving-2-4-1/32263 "2023-08-03T08:14:12Z")

</div>

I am attempting to build tensorflow-serving on an Ubuntu 18.04 LTS from source using the official documentation The build is running on an machine. I am running into problems with missing dependencies or errors concer…

---

## [Embeding the TensorFlow model vs asking to TensorFlow serving server](https://discuss.ai.google.dev/t/embeding-the-tensorflow-model-vs-asking-to-tensorflow-serving-server/30847)

<div class="topic-metadata">

**Author:** [@byounguk.lee](https://discuss.ai.google.dev/u/byounguk.lee)\
**Replies:** 0\
**Last updated:** [July 20, 2023, 8:39am UTC](https://discuss.ai.google.dev/t/embeding-the-tensorflow-model-vs-asking-to-tensorflow-serving-server/30847 "2023-07-20T08:39:35Z")

</div>

Hi, I want to use TensorFlow model in my application server. I have to decide how I use it. \[My question\] My questions is whether “Asking TensorFlow Serving server is always better than Embedding the TensorFlow model …

---

## [Serving Stable Diffusion in TF Serving](https://discuss.ai.google.dev/t/serving-stable-diffusion-in-tf-serving/28479)

<div class="topic-metadata">

**Author:** [@Chansung](https://discuss.ai.google.dev/u/Chansung)\
**Replies:** 1\
**Last updated:** [January 16, 2023, 11:35am UTC](https://discuss.ai.google.dev/t/serving-stable-diffusion-in-tf-serving/28479 "2023-01-16T11:35:22Z")

</div>

Hi folks, I am happy to share the project that I have done with @Sayak\_Paul. In this project, our main focus is to serve Stable Diffusion in TF Serving (V1 and V2 as well). To this end, we made three SavedModels for tex…

---

## [Tensorflow serving in Kubernetes deployment fails to predict based on input json (text based messages) - Output exceeds the size limit error](https://discuss.ai.google.dev/t/tensorflow-serving-in-kubernetes-deployment-fails-to-predict-based-on-input-json-text-based-messages-output-exceeds-the-size-limit-error/31691)

<div class="topic-metadata">

**Author:** [@duttab49](https://discuss.ai.google.dev/u/duttab49)\
**Replies:** 3\
**Last updated:** [November 7, 2022, 7:35am UTC](https://discuss.ai.google.dev/t/tensorflow-serving-in-kubernetes-deployment-fails-to-predict-based-on-input-json-text-based-messages-output-exceeds-the-size-limit-error/31691 "2022-11-07T07:35:46Z")

</div>

I have created tensorflow saved model using tf.keras.models.save\_model as below - tf.keras.models.save\_model( text\_model, export\_path, overwrite=True, include\_optimizer=True, save\_format=None, si…

---

## [What's the best way to import all required protobufs to compile PredictionService of TF-Serving?](https://discuss.ai.google.dev/t/whats-the-best-way-to-import-all-required-protobufs-to-compile-predictionservice-of-tf-serving/30497)

<div class="topic-metadata">

**Author:** [@jeongukjae](https://discuss.ai.google.dev/u/jeongukjae)\
**Replies:** 1\
**Last updated:** [September 9, 2022, 1:56am UTC](https://discuss.ai.google.dev/t/whats-the-best-way-to-import-all-required-protobufs-to-compile-predictionservice-of-tf-serving/30497 "2022-09-09T01:56:43Z")

</div>

Hi, I have a question about importing protobufs to develop a client for TensorFlow Serving. What is the official/best way to import all required protobufs and compile them for various languages? I want to import TF-Serv…

---

## [Tensorflow serving GRPC mode](https://discuss.ai.google.dev/t/tensorflow-serving-grpc-mode/32412)

<div class="topic-metadata">

**Author:** [@Dariusz\_Sciebura](https://discuss.ai.google.dev/u/Dariusz_Sciebura)\
**Replies:** 0\
**Last updated:** [August 26, 2022, 4:44pm UTC](https://discuss.ai.google.dev/t/tensorflow-serving-grpc-mode/32412 "2022-08-26T16:44:42Z")

</div>

Hi, I’m deploying a simple CPU model on tensorflow serving server (model is just a basic adder of 2 tensors, nothing heavy in particular). I created a test script in python in which I prepare and send a dozen of requests…

---

## [TFServing support for custom devices](https://discuss.ai.google.dev/t/tfserving-support-for-custom-devices/27621)

<div class="topic-metadata">

**Author:** [@pksubbarao](https://discuss.ai.google.dev/u/pksubbarao)\
**Replies:** 2\
**Last updated:** [August 11, 2022, 8:52pm UTC](https://discuss.ai.google.dev/t/tfserving-support-for-custom-devices/27621 "2022-08-11T20:52:57Z")

</div>

Hello All I am looking for documentation on how TFServing can be made to work with custom devices/accelerators. The link below talks about custom-ops, but not custom devices: For TensorFlow, we have registered our acc…

---

## [Direct loading/inference on model created in Vertex AI](https://discuss.ai.google.dev/t/direct-loading-inference-on-model-created-in-vertex-ai/31980)

<div class="topic-metadata">

**Author:** [@iordan](https://discuss.ai.google.dev/u/iordan)\
**Replies:** 1\
**Last updated:** [July 8, 2022, 6:23am UTC](https://discuss.ai.google.dev/t/direct-loading-inference-on-model-created-in-vertex-ai/31980 "2022-07-08T06:23:58Z")

</div>

I’m not familiar with TF Serving, so I would appreciate any information or direction on the problem I’m facing. I’ve trained a classification AutoML model on Vertex AI.Then I’ve downloaded the model locally and I am try…

---

## [Broken link on TF Serving](https://discuss.ai.google.dev/t/broken-link-on-tf-serving/27290)

<div class="topic-metadata">

**Author:** [@lucasfariaslf](https://discuss.ai.google.dev/u/lucasfariaslf)\
**Replies:** 1\
**Last updated:** [May 5, 2022, 4:46am UTC](https://discuss.ai.google.dev/t/broken-link-on-tf-serving/27290 "2022-05-05T04:46:07Z")

</div>

I found a broken link to TF Serving documentation on this tutorial page. The link is: Serving Models | TFX | TensorFlow.

---

## [Tf serving docker not working](https://discuss.ai.google.dev/t/tf-serving-docker-not-working/27315)

<div class="topic-metadata">

**Author:** [@guen\_gn](https://discuss.ai.google.dev/u/guen_gn)\
**Replies:** 7\
**Last updated:** [April 4, 2022, 3:09am UTC](https://discuss.ai.google.dev/t/tf-serving-docker-not-working/27315 "2022-04-04T03:09:59Z")

</div>

I have following saved\_model dir structure. The saved\_model\_cli is working correctly with the path but docker command not. Note this is following the book example and apparently docker part of example not working. test.…

---

## [Need help compiling TF Serving with custom TF kernels](https://discuss.ai.google.dev/t/need-help-compiling-tf-serving-with-custom-tf-kernels/32264)

<div class="topic-metadata">

**Author:** [@Juan\_Villamizar](https://discuss.ai.google.dev/u/Juan_Villamizar)\
**Replies:** 5\
**Last updated:** [March 20, 2022, 6:24pm UTC](https://discuss.ai.google.dev/t/need-help-compiling-tf-serving-with-custom-tf-kernels/32264 "2022-03-20T18:24:57Z")

</div>

Hey, I am looking for some help to debug a compilation issue for Serving. I have a TensorFlow fork with some custom kernel that compiles and runs without an issue. Unfortunately when I try to compile Serving with this sa…

---

## [Tensorflow serving latency spikes](https://discuss.ai.google.dev/t/tensorflow-serving-latency-spikes/32561)

<div class="topic-metadata">

**Author:** [@shayan\_sadeghieh](https://discuss.ai.google.dev/u/shayan_sadeghieh)\
**Replies:** 2\
**Last updated:** [February 16, 2022, 4:25pm UTC](https://discuss.ai.google.dev/t/tensorflow-serving-latency-spikes/32561 "2022-02-16T16:25:40Z")

</div>

Hi there, I’ve been running into issues with arbitrary spikes in latency when using tensorflow serving. I noticed the issue for tensorflow decision forest (TFDF) gradient boosted tree models, TFDF random forest models a…

---

## [About get tensorflow serving image](https://discuss.ai.google.dev/t/about-get-tensorflow-serving-image/31778)

<div class="topic-metadata">

**Author:** [@Haoran\_Jiang](https://discuss.ai.google.dev/u/Haoran_Jiang)\
**Replies:** 3\
**Last updated:** [November 24, 2021, 11:05pm UTC](https://discuss.ai.google.dev/t/about-get-tensorflow-serving-image/31778 "2021-11-24T23:05:04Z")

</div>

Hi guys, I am using tensorflow training model recently and I find that I can’t get tensorflow serving image by docker. Because I am using distributed device of my campus and I can’t run “sudo” command, neither any other …

---

## [Tensorflow Serving how to filter output?](https://discuss.ai.google.dev/t/tensorflow-serving-how-to-filter-output/24705)

<div class="topic-metadata">

**Author:** [@Toxic\_Cat](https://discuss.ai.google.dev/u/Toxic_Cat)\
**Replies:** 1\
**Last updated:** [November 21, 2021, 11:05am UTC](https://discuss.ai.google.dev/t/tensorflow-serving-how-to-filter-output/24705 "2021-11-21T11:05:55Z")

</div>

Hello all. I’m using SSD MobileNet V2 320x320 with Tensorflow Serving:latest-gpu on Nvidia 1060 6GB card Sending a lot of requests by REST API output\_body = {"inputs": \[{"b64": input\_image}\]} latency: ~1966.144561767…

---

## [Performance overhead of tensorflow custom ops](https://discuss.ai.google.dev/t/performance-overhead-of-tensorflow-custom-ops/29208)

<div class="topic-metadata">

**Author:** [@Tony\_Wang](https://discuss.ai.google.dev/u/Tony_Wang)\
**Replies:** 1\
**Last updated:** [October 29, 2021, 9:36am UTC](https://discuss.ai.google.dev/t/performance-overhead-of-tensorflow-custom-ops/29208 "2021-10-29T09:36:17Z")

</div>

Hi all – I have written some tensorflow custom ops that replace stock tensorflow kernels. When benchmarking the kernels in isolation in a C++ test bench my kernels have comparable performance with tensorflow kernels. How…

---

## [Build Serving Image with Batching Inference Request and How to check if its worked](https://discuss.ai.google.dev/t/build-serving-image-with-batching-inference-request-and-how-to-check-if-its-worked/27572)

<div class="topic-metadata">

**Author:** [@gabbygab1233](https://discuss.ai.google.dev/u/gabbygab1233)\
**Replies:** 0\
**Last updated:** [October 5, 2021, 7:00am UTC](https://discuss.ai.google.dev/t/build-serving-image-with-batching-inference-request-and-how-to-check-if-its-worked/27572 "2021-10-05T07:00:08Z")

</div>

How to test if the batching request work? Create Serving Image docker run -d --name serving\_base tensorflow/serving Batchin\_Parameters txt file max\_batch\_size { value: 32 } batch\_timeout\_micros { value: 5000 } pad\_vari…

---

## [Build Serving Image with Multiple Models](https://discuss.ai.google.dev/t/build-serving-image-with-multiple-models/31971)

<div class="topic-metadata">

**Author:** [@gabbygab1233](https://discuss.ai.google.dev/u/gabbygab1233)\
**Replies:** 5\
**Last updated:** [October 4, 2021, 6:17pm UTC](https://discuss.ai.google.dev/t/build-serving-image-with-multiple-models/31971 "2021-10-04T18:17:25Z")

</div>

I don’t have a problem when creating my own serving image in docker using 1 model but when I try to build a serving image with multiple models it doesn’t work. Here’s the command I used Create Serving Image docker ru…
