# Retracing with Distributed Training

**URL:** <https://discuss.ai.google.dev/t/retracing-with-distributed-training/29059>\
**Category:** General Discussion\
**Tags:** gpu, distributed-training\
**Created:** [June 29, 2022, 2:26am UTC](https://discuss.ai.google.dev/t/retracing-with-distributed-training/29059 "2022-06-29T02:26:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![King\_Gee](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/king_gee/32/12638_2.png) [@King\_Gee](https://discuss.ai.google.dev/u/King_Gee)\
**Post date:** [June 29, 2022, 2:26am UTC](https://discuss.ai.google.dev/t/retracing-with-distributed-training/29059/1 "2022-06-29T02:26:39Z")

</div>

Hi guys,

I am trying a custom model with distributed training on multiple GPU with tf.function(). The graph tends to compile at every call. To solve the issue i passed the input\_signature argument with specified tf.TensorSpec() on the tf.function() which works fine for 1 gpu, however when i use multiple gpus, it returns the error ‘Perreplica does not have dtype’.

Please any idea, how i an solve this problem?

---

<div class="post-metadata">

**Author:** ![Jetti\_Bharat](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/jetti_bharat/32/1510_2.png) [@Jetti\_Bharat](https://discuss.ai.google.dev/u/Jetti_Bharat)\
**Post date:** [October 11, 2024, 8:42am UTC](https://discuss.ai.google.dev/t/retracing-with-distributed-training/29059/3 "2024-10-11T08:42:26Z")

</div>

Hello @King_Gee

Thank you for using TensorFlow  
In the training step, add following line to avoid the perreplica issues for dtypes, as mentioned in the [documentation](https://www.tensorflow.org/api_docs/python/tf/distribute/Strategy#experimental_local_results)  
`per_replica_inputs = strategy.experimental_local_results(inputs)`  
and give the per\_replica\_inputs to model as a list.