Fine-tuning a pre-trained model while replacing one of the pre-trained layers with a new PyTorchlayer

TensorFlow General Discussion
,
1

Hi everyone!

I want to fine-tune a pre-trained BERT model off of the official BERT repository. I want to replace one of the pre-trained dense layers with a custom PyTorch layer.

I’ve been trying to implement this, but I haven’t been able to figure it out. I learned that the get_assignment_map_from_checkpoint function computes the union of the current variables and checkpoint variables, and then the output of this function is passed into tf.train.init_from_checkpoint.

Any tips/advice/suggestions would be greatly appreciated. Thank you in advance for your help!

3

Hello @Conor_Warren

To my understanding, The possible way of doing this is to get the model checkpoints and identify the layer which you want to work on and then copy the weights of that layer and add them to replaced layer from pytorch.
The general way of working with tensorflow models in pytorch is to convert them to pytorch, verify the model with the similar accuracy inference and then work with the model in pytorch.

Thank you.