Pipeline parallel reading and training data

TensorFlow General Discussion
,
1

I have a classification task, the data is about 10 million files, each file corresponds to a classification marker, from these 10 million files, each file can be loaded with a 1000 dimensional numeric vector, now I want to build a model for Pipeline parallel reading and training data, is there any readymade tensorflow framework I can refer to?

3

Hi @wu_Heng Welcome to The Tensorflow Forum ,

To parallelize training of a model with multiple inputs and outputs in TensorFlow, you can leverage functionalities through the tf.data API and other tools that can be combined to achieve this.

You can refer to these articles; they might be helpful.
Pretrained model
tf.data: Build TensorFlow input pipelines
Distributed training with Keras

Thank You !