Convert to tfrecord and write directly to an AWS S3 Bucket

TensorFlow General Discussion
, ,
1

I have some images that I would like to convert to TFRecord format and load them into an S3 Bucket on AWS. However, existing solutions I found requires saving the TFRecords locally first, then uploading the saved local files to S3. I’m wondering if there’s a way to convert to TFRecords and save the output as .tfrecord directly to an S3 bucket.

2

I think that if you check point 6 probably it could work also with an s3 url:

3

Unfortunately I got an File system scheme 's3' not implemented error when I tried that. The filename I used with tf.io.TFRecordWriter was s3://BUCKET-NAME/record-name.tfrecord.

4

Have you tried to install TF.io?

See:
https://github.com/tensorflow/tensorflow/issues/40302#issuecomment-918917472

5

I attempted the above with with tf.io 0.17 installed and got the same error (also tried with s3e and got the same).

7

@mihaimaruseac as you have replied on the mentioned ticket do you know something more?

8

For additional information, I’m using tensorflow 2.4.4, with OS Windows 10. For some reason tf.io.read_file works, even without installing tf.io, but other methods like tf.io.gfile.GFile' returns the same s3 not implemented` error