Tf serving docker not working

guen_gn · November 9, 2021, 12:02am

I have following saved_model dir structure. The saved_model_cli is working correctly with the path but docker command not. Note this is following the book example and apparently docker part of example not working.
test.npy test input feeds 3 instances of mnist 28x28 images to serving.

DIR STRUCT:

root@nonroot-Standard-PC-i440FX-PIIX-1996:~/dev-learn/gpu/tflow/tensorflow/tflow-2nded# tree p297
p297
├── 0001
│   ├── assets
│   ├── saved_model.pb
│   └── variables
│       ├── variables.data-00000-of-00001
│       └── variables.index
├── assets
├── keras_metadata.pb
├── saved_model.pb
└── variables
    ├── variables.data-00000-of-00001
    └── variables.index

CLI:

    saved_model_cli run --dir p297/0001 --tag_set serve --signature_def serving_default --inputs 
flatten_input=test.npy

output of cli (OK)

2021-11-08 15:57:11.458910: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-08 15:57:11.460906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 31740 MB memory:  -> device: 0, name: Device 738c, pci bus id: 0000:00:07.0
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/saved_model_cli.py:445: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
INFO:tensorflow:Restoring parameters from p297/0001/variables/variables
2021-11-08 15:57:11.961912: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
2021-11-08 15:57:12.671939: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
2021-11-08 15:57:12.686081: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
Result for output key dense_2:
[[3.65022061e-05 2.47262960e-05 6.37578269e-05 2.08125566e-05
  7.02261750e-05 1.18829332e-01 4.30839646e-05 2.72816449e-01
  1.48763061e-02 5.93046188e-01 6.37045162e-07 3.02645799e-06
  3.80635288e-06 4.22269186e-05 6.12226586e-06 4.33605646e-06
  7.58367719e-07 3.06199559e-06 5.42078442e-06 2.38417056e-06
  4.93106018e-06 7.25027712e-06 1.24132812e-05 1.24427579e-05
  1.15528803e-06 4.87520847e-05 1.68714314e-06 8.28819338e-07
  2.23448342e-06 9.11506140e-06]
 [1.64286757e-04 3.39760754e-06 9.66621459e-01 1.61247503e-04
  1.02249524e-02 7.44288286e-07 2.27110237e-02 3.17756710e-10
  1.12835020e-04 3.98790796e-08 4.89366492e-10 1.29820976e-09
  9.63464334e-13 5.25168797e-10 9.53418247e-11 4.82374418e-10
  2.60897762e-11 2.75338996e-12 3.59164387e-09 7.25419169e-11
  5.41757861e-10 8.33503266e-10 1.25494719e-11 4.14474233e-09
  3.54530544e-10 7.37128275e-11 4.25408209e-10 7.22836443e-11
  6.95292546e-10 2.50320233e-11]
 [1.12954825e-04 9.99067128e-01 5.11940962e-05 5.15281979e-04
  2.25681084e-04 7.90114484e-07 2.39875553e-05 1.08763277e-06
  6.77304740e-07 8.83030324e-08 3.44932758e-08 3.36465895e-08
  1.49800372e-09 2.67271858e-08 7.71939810e-08 1.00484840e-07
  5.05017761e-09 4.95800823e-09 1.20519260e-07 1.41827059e-07
  1.35087987e-07 3.95625591e-07 4.27236913e-08 4.71499533e-08
  3.37586954e-08 1.78841599e-08 1.00948716e-08 4.40149028e-10
  2.45303022e-09 2.79508638e-09]]

DOCKER:

    MODEL_NAME=p297
    docker pull tensorflow/serving
    docker run -it --rm -p 8500:8500 -p 8501:8501 \
        -v "$MODEL_NAME:/models/$MODEL_NAME" \
        -e MODEL_NAME=$MODEL_NAME \
        tensorflow/serving

output of docker instance (fail):

Using default tag: latest
latest: Pulling from tensorflow/serving
Digest: sha256:6651f4839e1124dbde75ee531825112af0a6b8ef082c88ab14ca53eb69a2e4bb
Status: Image is up to date for tensorflow/serving:latest
docker.io/tensorflow/serving:latest
2021-11-09 00:01:43.627821: I tensorflow_serving/model_servers/server.cc:89] Building single TensorFlow model file config:  model_name: p297 model_base_path: /models/p297
2021-11-09 00:01:43.628201: I tensorflow_serving/model_servers/server_core.cc:465] Adding/updating models.
2021-11-09 00:01:43.628227: I tensorflow_serving/model_servers/server_core.cc:591]  (Re-)adding model: p297
2021-11-09 00:01:43.629366: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:268] No versions of servable p297 found under base path /models/p297. Did you forget to name your leaf directory as a number (eg. '/1/')?
2021-11-09 00:01:44.629551: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:268] No versions of servable p297 found under base path /models/p297. Did you forget to name your leaf directory as a number (eg. '/1/')?
2021-11-09 00:01:45.629769: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:268] No versions of servable p297 found under base path /models/p297. Did you forget to name your leaf directory as a number (eg. '/1/')?
^X2021-11-09 00:01:46.629997: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:268] No versions of servable p297 found under base path /models/p297. Did you forget to name your leaf directory as a number (eg. '/1/')?

I logged onto container using
docker-exec -it <C_ID> /bin/bash
and inspected p297/0001 folder but turns out to be empty.

sebastian-sz · November 9, 2021, 7:19am

I think the conflict could be due to the fact that you have both leaf directory 0001 and actual Saved Model in your Model directory. Can you try moving 0001 to a separate folder? Or Saved Model to 0002 for example?

guen_gn · November 9, 2021, 3:51pm

I am not sure why it would interfere? It saved that way when I saved as saved_model.

sebastian-sz · November 10, 2021, 6:25am

Yes, but the docker expects that you pass a versioned model, aka in a directory like 0001. In your directory with models you are keeping 1 versioned and 1 unversioned, so maybe this could be a problem.

guen_gn · November 12, 2021, 10:40pm

[post removed by moderator]

sebastian-sz · November 13, 2021, 12:18pm

@guen_gn
I feel like this is a bit of exaggeration: I only suggested moving few files to a separate folder, because I used to have this problem in the past.

My suggestion was not meant to be bad or ill-motivated, but if you feel that way I’ll leave you to it.

Abdul_Mohammed · March 15, 2022, 10:05pm

Hi Sebastian. I am having the same problem. I was hoping you would have some ideas. My dir structure is as follows:

guen_gn:

kcb_models
├──1
        ├── assets
        ├── saved_model.pb
        ├── keras.metadata.pb
        └── variables
                   ├── variables.data-00000-of-00001
                   └── variables.index

Thanks

estiller_dev · April 4, 2022, 3:09am

thanks. this solved my problem. For some reason, running it in docker expects the folder to be named 0001 but running it in tf model server is fine with 1.