Reducing the parameter size of LaBSE(language-agnostic BERT Sentence Embedding) for practical usage

TensorFlow Show and Tell
,
1

To get good quality language-agnostic sentence embeddings, LaBSE is a good choice. But due to the parameter size(Bert-base size, but #param is 471M), it is hard to fine-tune/deploy appropriately in a small GPU/machine.

So I applied the method of the paper “Load What You Need: Smaller Versions of Multilingual BERT” to get the smaller version of LaBSE, and I can reduce LaBSE’s parameters to 47% without a big performance drop using TF-hub and tensorflow/models.

GitHub: GitHub - jeongukjae/smaller-labse: Applying "Load What You Need: Smaller Versions of Multilingual BERT" to LaBSE

Relative Links:

3

Nice work Jeon!!

Does the preprocessing model still works with your model? or is there still a need for it?
Did you think about publishing this to TFHub too?

1 Like
4

Thank you :slight_smile:

The preprocessing model is exported using the modified vocab file. So this model can be used with updated preprocessing model, not the original one. (You can check here make_smaller_labse.py#L37)

And I didn’t think about publishing this model, because I didn’t train the model, just patched it. Is it okay to publish?

6

Yes I think you should!!

Of course, mention the base model on the description and all.
I’d also publish the updated preprocessing model too to keep consistency.

2 Likes
8

Oh, then I will check the docs and send a PR publishing this model!

2 Likes
10

This is perfect!! Thanks!

Keep me posted, I’d love to try it on this Colab: Classify text with BERT  |  Text  |  TensorFlow

1 Like
12

Thanks!! I created a PR to upload this model.

3 Likes
13

Very good!! thanks for contributing to the community!

1 Like
15

and it’s live: Kaggle | smaller_LaBSE_15lang | Kaggle

well done!!

I’d update the documentation to link to your preprocessing too!
Great work!

3 Likes