TFDS on Colab and Kaggle with TPU

innat · September 1, 2022, 2:43pm

I have a related query similar like this

Document how to use TFDS on Colab with TPU

opened 11:24AM - 19 Apr 19 UTC

help contributions welcome

**What I need help with / What I was wondering** When trying to use TFDS on Goo…gle Colab with TPU acceleration, there's the following exception: ``` UnimplementedError: File system scheme '[local]' not implemented ``` **What I've tried so far** From e.g. `https://cloud.google.com/tpu/docs/quickstart` one can see that TPUs expect data to be stored on GCS. However there are examples using Keras+TPU on Colab which load data via `tf.keras.datasets`, such as: `https://colab.research.google.com/gist/ceshine/f196d6b030adb1ec3a8d0b50642709dc/keras-fashion-mnist-tpu.ipynb` **It would be nice if...** ...there was documentation on how to use TFDS with Keras using TPU on Colab.

but in general on both colab tpu and kaggle tpu, what are the best practices to use tensorflow-dataset with them?

Bhack · September 1, 2022, 4:49pm

Isn’t this similar to the “standard” TPU?

github.com/keras-team/keras-cv

Improving ImageNet-1k support

opened 04:11PM - 30 Aug 22 UTC

closed 01:47AM - 28 Feb 24 UTC

sayakpaul

stat:awaiting response from contributor stale

W.r.t the current support for [ImageNet-1k](https://github.com/keras-team/keras-…cv/blob/master/keras_cv/datasets/imagenet), we can improve things: * First, let's start leveraging TFDS. It significantly reduces the work expected to be done by a user. Let's walk through an example. First, the user needs to keep the `ILSVRC2012_img_train.tar` and `ILSVRC2012_img_val.tar` archives to this path: `gs://[BUCKET-NAME]/tensorflow_datasets/downloads/manual`. * One this is done, the user does the following: ```py import tensorflow_datasets as tfds data_dir = "gs://[BUCKET-NAME]/tensorflow_datasets" builder = tfds.builder("imagenet2012", data_dir=data_dir) builder.download_and_prepare() ``` `builder.download_and_prepare()` takes some time but it's lesser than what the current process of obtaining the initial TFRecords takes. * The the user can load the ImageNet-1k dataset with `tfds.load("imagenet2012", data_dir=data_dir)` and that is it. The above two points assume the user already has access to the GCS bucket and all the necessary privileges to write data into it. ## General recommendations W.r.t https://github.com/keras-team/keras-cv/blob/e607e05e7d73dda4fcc1f11eca11d1f71d83bee4/keras_cv/datasets/imagenet/load.py#L92 enable interleaved reading by setting `num_parallel_reads=tf.data.AUTOTUNE`. W.r.t https://github.com/keras-team/keras-cv/blob/e607e05e7d73dda4fcc1f11eca11d1f71d83bee4/keras_cv/datasets/imagenet/load.py#L113 enable prefetching of a few batches so that the accelerator doesn't have to wait by using `dataset.prefetch(tf.data.AUTOTUNE)`.

Topic		Replies	Views
Load tfrecord dataset from google drive when using TPU in colab General Discussion datasets , tpu , help_request	1	1685	September 20, 2021
Tensorflow code running on colab TPU but not on Kaggle/GCE TPU-VMs General Discussion kaggle , tpu , help_request	3	2192	February 28, 2022
Issue with TPUs on Google Colab when training BERT General Discussion nlp , tpu , help_request	0	1550	March 11, 2022
Problem of using tfds when try to load my super resolution dataset General Discussion datasets , help_request	14	1986	February 17, 2022
Image augmentation library working with TPU Show and Tell tpu , library	3	1650	July 25, 2022

TFDS on Colab and Kaggle with TPU

Related topics