# How to do parallel batch inferencing?

**URL:** <https://discuss.ai.google.dev/t/how-to-do-parallel-batch-inferencing/31884>\
**Category:** General Discussion\
**Tags:** keras, tensorflow, models, help\_request\
**Created:** [July 2, 2022, 10:24pm UTC](https://discuss.ai.google.dev/t/how-to-do-parallel-batch-inferencing/31884 "2022-07-02T22:24:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![IamExperimenting\_Now](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/iamexperimenting_now/32/11830_2.png) [@IamExperimenting\_Now](https://discuss.ai.google.dev/u/IamExperimenting_Now)\
**Post date:** [July 2, 2022, 10:24pm UTC](https://discuss.ai.google.dev/t/how-to-do-parallel-batch-inferencing/31884/1 "2022-07-02T22:24:34Z")

</div>

Hi Team, I have built the tf2 model which does image classification. Currently, when I do batch inference for 50 images it’s takes 42secs. And I does sequential inference. I would like to do parallel inference and reduce the inference because I got 70K images to do inference as batch.

Can anyone please help me here to solve the problem?

Please find me script below,

```auto
import tensorflow as tf, numpy as np
from PIL import Image, ImageOps

model = tf.keras.models.load_model(‘export/model’)

image = ‘new_image.png’
image = Image.open(image).convert(‘RGB’)
image = ImageOps.exif_transpose(image)
image = np.array(image.resize((224,224)))
image = np.reshape(image,(1,224,224,3))
prediction_result = model.predict(image)

```

Can anyone please help me to do parallel inference.

Currently, I use for loop to iterate the prediction sequentiality.

#tf2 #Keras #inference #Tensorflow #batchinference

---

<div class="post-metadata">

**Author:** ![Lance\_N](https://avatars.discourse-cdn.com/v4/letter/l/a4c791/32.png) [@Lance\_N](https://discuss.ai.google.dev/u/Lance_N)\
**Post date:** [July 11, 2022, 4:29am UTC](https://discuss.ai.google.dev/t/how-to-do-parallel-batch-inferencing/31884/3 "2022-07-11T04:29:42Z")

</div>

You need to create a Tensorflow Dataset class that reads your images and yields them in a sequence. Keras models expect to iterate through thousands of images using an object that reads a batch of images, collects the batch into a sequence, then returns that batch, then later starts on the next batch.

Learn the Dataset abstraction to achieve what you are trying to do with 70k images.

> **[tf.data.Dataset  |  TensorFlow v2.16.1](https://www.tensorflow.org/api_docs/python/tf/data/Dataset)**
>
> Represents a potentially large set of elements.