Token count decrease from 1 mil to 16k after fine tune

Google AI Studio
,
1

I fine tune gemini flash 1.5 on google studio by upload csv file as dataset. After 20 hours of fine tuning, the ui show token count of 16,2348 tokens.
Screenshot from 2024-10-15 09-56-06
Switching back to gemini flash 1.5 show 1.5 mil tokens correctly.

The config I use to train the model is batch size 32, epochs 50 and learning rate is 0.005

2

Hi @Flynn_Tran,

Welcome to the forum!!

There is some limitation of tuned models : The input limit of a tuned Gemini 1.5 Flash model is 40,000 characters. Follow this doc.

4

I thought that is the limitation of dataset when training, not limitation of the tuned model. Thank you very much!