Dropout in Tensorflow Hub Problem

TensorFlow General Discussion
, ,
1

Recently, I try to use bert to do some contrastive learning in tfhub. You know that dropout rate is a important param in contrastive learning (for nlp task), but I don’t find any method to set dropout rate in tfhub api. Is there some way to set dropout in tfhub?

Thanks!

3

Hi Ryan,

I don’t think you can change dropout as a parameter from the models on Hub.

What you could do instead, is create a model yourself (eg: Classify text with BERT  |  Text  |  TensorFlow) based on the BERT encoders on TFHub
and define the dropout rate you want. With this you have full control of the parameter.
does that work for you?

5

Thanks for your reply.
But in Contrastive Learning I must set dropout of the pretrained part. The tutorial Classify text with BERT | Text | TensorFlow can’t help me. Is there any solution or new features for Contrastive Learning in TFHub? (Reference: [2104.08821] SimCSE: Simple Contrastive Learning of Sentence Embeddings)

1 Like
8

Sorry Ryan, I don’t know if for text, only image (Google | resnet_v1 | Kaggle)

1 Like
10

I think e.g. in Albert V2 was removed:

1 Like