# How to properly deploy Keras models for inference in Python? **URL:** \ **Category:** General Discussion\ **Tags:** keras, models, help\_request\ **Created:** [March 24, 2022, 1:27am UTC](https://discuss.ai.google.dev/t/how-to-properly-deploy-keras-models-for-inference-in-python/32565 "2022-03-24T01:27:05Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![apcamargo](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@apcamargo](https://discuss.ai.google.dev/u/apcamargo)\ **Post date:** [March 24, 2022, 1:27am UTC](https://discuss.ai.google.dev/t/how-to-properly-deploy-keras-models-for-inference-in-python/32565/1 "2022-03-24T01:27:05Z")
I’m deploying a big Keras model to production and it’s not very clear to me if I should do anything to it to make it more efficient for inference. In tf-v1 I used to prepare frozen graphs, but this is being deprecated in tf-v2 (as far as I understand). Right now, I’m just loading the model and using the `.predict()` method to perform inference on `.tfrec` files. ```python from tensorflow import keras model = keras.models.load_model('path/to/location') model.predict(get_dataset(tfrecord_list, batch_size), verbose=0) ``` This model won’t be trained anymore, so I wanted to understand if there are any production-specific steps that I should do to increase the model computational performance (if possible). Thanks! ---
**Author:** ![Kiran\_Sai\_Ramineni](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/kiran_sai_ramineni/32/8859_2.png) [@Kiran\_Sai\_Ramineni](https://discuss.ai.google.dev/u/Kiran_Sai_Ramineni)\ **Post date:** [March 25, 2022, 8:35am UTC](https://discuss.ai.google.dev/t/how-to-properly-deploy-keras-models-for-inference-in-python/32565/3 "2022-03-25T08:35:12Z")
Hi @apcamargo please refer to this [documentation](https://www.tensorflow.org/tfx/tutorials/serving/rest_simple#serve_your_model_with_tensorflow_serving) to know how to deploy sever you model using TensorFlow serving. ---
**Author:** ![apcamargo](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@apcamargo](https://discuss.ai.google.dev/u/apcamargo)\ **Post date:** [March 25, 2022, 3:55pm UTC](https://discuss.ai.google.dev/t/how-to-properly-deploy-keras-models-for-inference-in-python/32565/5 "2022-03-25T15:55:26Z")
HI @Kiran_Sai_Ramineni. TensorFlow serving is aimed towards web services, right? My model will be distributed within a Python package, so I’m not sure if that’s the way to go ---
**Author:** ![Bhack](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/bhack/32/6388_2.png) [@Bhack](https://discuss.ai.google.dev/u/Bhack)\ **Post date:** [March 25, 2022, 5:07pm UTC](https://discuss.ai.google.dev/t/how-to-properly-deploy-keras-models-for-inference-in-python/32565/6 "2022-03-25T17:07:30Z")
Other then optimizing the model itself you can try to jit\_compile your model: > **[Classifying CIFAR-10 with XLA  |  OpenXLA Project](https://openxla.org/xla/tf2xla/tutorials/autoclustering_xla)** > **[GitHub - tensorflow/model-optimization: A toolkit to optimize ML models for deployment for...](https://github.com/tensorflow/model-optimization)** > > A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning. ---
**Author:** ![apcamargo](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@apcamargo](https://discuss.ai.google.dev/u/apcamargo)\ **Post date:** [March 29, 2022, 9:43pm UTC](https://discuss.ai.google.dev/t/how-to-properly-deploy-keras-models-for-inference-in-python/32565/8 "2022-03-29T21:43:26Z")
Thanks, @Bhack!. `jit_compile` doesn’t work with my model. I still haven’t figured out why. ---
**Author:** ![Bhack](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/bhack/32/6388_2.png) [@Bhack](https://discuss.ai.google.dev/u/Bhack)\ **Post date:** [March 29, 2022, 9:54pm UTC](https://discuss.ai.google.dev/t/how-to-properly-deploy-keras-models-for-inference-in-python/32565/9 "2022-03-29T21:54:36Z")
Generally you have a message in the log about something not supported. ---
**Author:** ![apcamargo](https://avatars.discourse-cdn.com/v4/letter/a/58956e/32.png) [@apcamargo](https://discuss.ai.google.dev/u/apcamargo)\ **Post date:** [March 30, 2022, 11:34pm UTC](https://discuss.ai.google.dev/t/how-to-properly-deploy-keras-models-for-inference-in-python/32565/11 "2022-03-30T23:34:06Z")
I’m getting a `InvalidArgumentError` error. ```auto InvalidArgumentError: Graph execution error: Detected at node 'StatefulPartitionedCall' defined at (most recent call last): … ``` I’m not posting the whole log here because that’s out of topic. I’m using a complex custom layer that is probably causing this. I’ll try to figure out the root of the problem. ---
**Author:** ![Bhack](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/bhack/32/6388_2.png) [@Bhack](https://discuss.ai.google.dev/u/Bhack)\ **Post date:** [March 31, 2022, 12:05am UTC](https://discuss.ai.google.dev/t/how-to-properly-deploy-keras-models-for-inference-in-python/32565/12 "2022-03-31T00:05:16Z")
You can limit the compilation scope on critical functions with `@tf.function(jit_compile=True)`: > **[Use XLA with tf.function  |  OpenXLA Project](https://openxla.org/xla/tf2xla/tutorials/jit_compile?hl=en)** But I suggest also to profile your model to understand what happens: > **[Optimize TensorFlow performance using the Profiler  |  TensorFlow Core](https://www.tensorflow.org/guide/profiler)**