Custom Data generator in TF2

Abhijeet · January 17, 2022, 4:14pm

Can someone guide, how to include the Custom DataGenerator functionality from tf.keras with methods
keras.utils.Sequence ,
on_epoch_end, __len__, __getitem__ ,
`

in tensorflow alone?

Bhack · January 17, 2022, 6:14pm

I don’t understand your issue, do you have a minimized standalone example or Colab?

Abhijeet · January 18, 2022, 5:04pm

Yes, here is the link. Uses custom generator based on
keras.utils.Sequence
and some add-ons to fetch the cores.

Is there way to implement in native TensorFlow alone (no Keras)?

Bhack · January 18, 2022, 5:08pm

it is just really like a plain python class. Please check yourself the source code at:

github.com

keras-team/keras/blob/v2.7.0/keras/utils/data_utils.py#L411-L486


      
          class Sequence:
            """Base object for fitting to a sequence of data, such as a dataset.
          
            Every `Sequence` must implement the `__getitem__` and the `__len__` methods.
            If you want to modify your dataset between epochs you may implement
            `on_epoch_end`.
            The method `__getitem__` should return a complete batch.
          
            Notes:
          
            `Sequence` are a safer way to do multiprocessing. This structure guarantees
            that the network will only train once
             on each sample per epoch which is not the case with generators.
          
            Examples:
          
            ```python
            from skimage.io import imread
            from skimage.transform import resize
            import numpy as np

This file has been truncated. show original

Topic		Replies	Views
What is the difference between tf.data.Dataset and Data generators with Keras General Discussion datasets , keras , tfdata , learning , help_request	3	3521	March 8, 2022
[AutoKeras] Bug in using tf.Dataset for training Keras datasets , keras , help_request	3	1901	December 6, 2021
Keras custom models deteriorates after save and reload General Discussion models , keras , help_request	20	1931	August 19, 2021
Custom Dtype Objects in Tensors General Discussion help_dev , tfcore	1	2240	October 16, 2021
Custom Optimizer General Discussion api , docs , keras , help_dev	1	1382	October 8, 2022

Custom Data generator in TF2

Related Topics