# Writing multiple tf.train.Example to a TFRecord

**URL:** <https://discuss.ai.google.dev/t/writing-multiple-tf-train-example-to-a-tfrecord/28252>\
**Category:** General Discussion\
**Tags:** datasets, help\_request\
**Created:** [June 14, 2022, 2:55pm UTC](https://discuss.ai.google.dev/t/writing-multiple-tf-train-example-to-a-tfrecord/28252 "2022-06-14T14:55:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![neel\_g](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/neel_g/32/9896_2.png) [@neel\_g](https://discuss.ai.google.dev/u/neel_g)\
**Post date:** [June 14, 2022, 2:55pm UTC](https://discuss.ai.google.dev/t/writing-multiple-tf-train-example-to-a-tfrecord/28252/1 "2022-06-14T14:55:50Z")

</div>

This (unanswered) post accurately summarizes the question: [python - Writing TFRecords in batches - Stack Overflow](https://stackoverflow.com/questions/49768997/writing-tfrecords-in-batches)

I want to speed up a multi-thread script, so I wanted to look to append multiple `tf.train.Example` in a single list and then write them to the serial `TFRecord`. How can I accomplish that?

Writing thousands of `TFRecords` with the conventional method seems a bad idea for my dataset, potentially slowing down data reading and creating a bottleneck.

Cheers,  
N

---

<div class="post-metadata">

**Author:** ![aniruthraj](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/aniruthraj/32/660_2.png) [@aniruthraj](https://discuss.ai.google.dev/u/aniruthraj)\
**Post date:** [November 29, 2024, 1:49pm UTC](https://discuss.ai.google.dev/t/writing-multiple-tf-train-example-to-a-tfrecord/28252/3 "2024-11-29T13:49:45Z")

</div>

Hi @neel_g,

Apologize for the delay in response.  
To write TFRecords efficiently, I suggest to batch your `tf.train.Example` objects and write each batch to a TFRecord file which reduces computation overhead and balances memory usage and use `tf.io.TFRecordWriter` for serialization, adjusting the batch size for optimal performance.In addition, we can parallel process it with threading. Here’s a sample implementation [gist](https://colab.research.google.com/gist/aniruthraj/453588d3fd19aee99fd70915c7690f52/tfrecords.ipynb) for your reference.

Hope this helps.Thank You.