Transformers for image prediction

[Eugenia_Sol]

Hey!

I was wondering if Transformers can be use for image prediction?
For example: you’ve got a sequence of images and you want to predict what the next image in the sequence will be.

I’ve found plenty of examples for image classification (ViT), but not for prediction. (If you have some examples I appreciate the help)

Thanks

[8bitmp3]

@Eugenia_Sol

I’ve found plenty of examples for image classification (ViT), but not for prediction. (If you have some examples I appreciate the help)

Hi, welcome to the TF Forum!

@Eugenia_Sol

I was wondering if Transformers can be use for image prediction?

Great question.

There’s a fairly recent paper by Google Research from 2021 called VATT (Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text) that was published with TensorFlow 2 code.

• Code on GitHub: google-research/vatt at master · google-research/google-research · GitHub
• Paper: VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (https://arxiv.org/pdf/2104.11178.pdf)

Other research papers that may be related to this task:

• “VPTR: Efficient Transformers for Video Prediction” (2022) [2203.15836] VPTR: Efficient Transformers for Video Prediction
• “Anticipative Video Transformer” (2021) https://openaccess.thecvf.com/content/ICCV2021/papers/Girdhar_Anticipative_Video_Transformer_ICCV_2021_paper.pdf
• “Generative Video Transformer: Can Objects be the Words?” (2021) http://proceedings.mlr.press/v139/wu21h/wu21h.pdf
• “Vision Transformers for Dense Prediction” (2021)
  ICCV 2021 Open Access Repository, https://www.youtube.com/watch?v=baZgifARKWA