Fitting transformers model for text 2 text generation

TensorFlow General Discussion
,
1

Hi,

I am looking to fit a transformers model on a large amount of text and then to fine tune it for a specific task using transfer learning. I have looked at this tutorial Modelo de transformador para compreensão da linguagem  |  Text  |  TensorFlow

But there seems to be only one “fitting” procedure. Do you know if there is any example on fitting a transformers model first on raw text so that it learns the language and then to fit it on a specific task through transfer learning?

Many thanks,

Antoine

3

You could check KerasNLP tutorial:

1 Like