# Transformer model for language understanding with another Dataset

**URL:** <https://discuss.ai.google.dev/t/transformer-model-for-language-understanding-with-another-dataset/24172>\
**Category:** General Discussion\
**Tags:** nlp, datasets, help\_request\
**Created:** [October 3, 2021, 1:53pm UTC](https://discuss.ai.google.dev/t/transformer-model-for-language-understanding-with-another-dataset/24172 "2021-10-03T13:53:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![PK2021](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/pk2021/32/9080_2.png) [@PK2021](https://discuss.ai.google.dev/u/PK2021)\
**Post date:** [October 3, 2021, 1:53pm UTC](https://discuss.ai.google.dev/t/transformer-model-for-language-understanding-with-another-dataset/24172/1 "2021-10-03T13:53:59Z")

</div>

Hello all!

I have been reading the official guide here ([Modelo de transformador para compreensão da linguagem &nbsp;|&nbsp; Text &nbsp;|&nbsp; TensorFlow](https://www.tensorflow.org/text/tutorials/transformer)) to try and recreate the Vanilla Transformer in Tensorflow. I notice the dataset used is quite specific, and at the end of the guide, it says to try with a different dataset.

But that is where I have been stuck for a long time! I am trying to use the WMT14 dataset (as used in the original paper, Vaswani et. al.) here: [wmt14\_translate &nbsp;|&nbsp; TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/wmt14_translate#wmt14_translatede-en).

I have also tried Multi30k and IWSLT dataset from spacy, but are there any guides on how I can fit the dataset to what the model requires? Specifically, to tokenize it. The official TF guide uses a pretrained tokenizer, which is specific to the PR-EN dataset given.

```auto
model_name = "ted_hrlr_translate_pt_en_converter"

```

I am wondering, how I can use the TF (bert) tokenizer to tokenize the Spacy dataset? I have the code for PyTorch, unfortunately I do not know how to adapt it for Tensorflow. Any help would be greatly appreciated!

```auto
 import spacy
    spacy_de = spacy.load('de')
    spacy_en = spacy.load('en')

    def tokenize_de(text):
        return [tok.text for tok in spacy_de.tokenizer(text)]

    def tokenize_en(text):
        return [tok.text for tok in spacy_en.tokenizer(text)]

    BOS_WORD = '<s>'
    EOS_WORD = '</s>'
    BLANK_WORD = "<blank>"
    SRC = data.Field(tokenize=tokenize_de, pad_token=BLANK_WORD)
    TGT = data.Field(tokenize=tokenize_en, init_token = BOS_WORD, 
                     eos_token = EOS_WORD, pad_token=BLANK_WORD)

    MAX_LEN = 100
    train, val, test = datasets.IWSLT.splits(
        exts=('.de', '.en'), fields=(SRC, TGT), 
        filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and 
            len(vars(x)['trg']) <= MAX_LEN)
    MIN_FREQ = 2
    SRC.build_vocab(train.src, min_freq=MIN_FREQ)
    TGT.build_vocab(train.trg, min_freq=MIN_FREQ)

```

---

<div class="post-metadata">

**Author:** ![chunduriv](https://avatars.discourse-cdn.com/v4/letter/c/bbe5ce/32.png) [@chunduriv](https://discuss.ai.google.dev/u/chunduriv)\
**Post date:** [September 16, 2022, 2:33pm UTC](https://discuss.ai.google.dev/t/transformer-model-for-language-understanding-with-another-dataset/24172/3 "2022-09-16T14:33:24Z")

</div>

Please follow [Subword tokenizers &nbsp;|&nbsp; Text &nbsp;|&nbsp; TensorFlow](https://www.tensorflow.org/text/guide/subwords_tokenizer). Thank you