Customize classification dataset for resnet

TensorFlow General Discussion
, ,
1

Hi there,

i would like to ask how to customize dataloader in tensorflow.

Now the image paths and labels are saved in csv-file as dataframe.
My first step is get the huge dataframe from many csv files and the images as data will be appended in the dataframe.
Then using a function i get these images and labels from the big dataframes.The images and labels are numpy arrays.
At last i feed the data direktly in model.fit(traindata, trainlbl). The problem comes out when the dataset too big becomes, the cpu will be overloaded and the SIGKILL9 will appear.
I have also tried to put data and label numpy array in tensor via tf.data.Dataset.from_tensor_slices((data, lbl)). And it runs also on cpu, when the data is too big, then also SIGKILL9.

So, could some one help or give me a hint about how to customize dataset in tensorflow?

best regards,
Chao

3

Hi @yc9614 ,

Welcome to the TensorFlow Forum!

You can try using tf.data.experimental.make_csv_dataset function which is a high-level interface for reading sets of CSV files. It supports column type inference and many other features, like batching and shuffling, to make usage simple.

Please refer to this page to explore other methods and its usability. Thank you.