Calculating throughout (image/second) of a model

TensorFlow General Discussion
, ,
1

I would like to know how can I calculate the throughput of CNN models.

3

In the context of serving a model it is impacted by different factors:

5

The throughput will depend on:

  • The hardware that you’re running on
  • What else is running on that hardware
  • Whether you’re doing online or batch inference
  • The configuration of your serving framework (such as TF Serving or the BulkInferrer component)

If you can pick a single set of hardware and configuration then you can make comparisons of different models. This is not unlike other types of throughput estimation.