Wildly inaccurate results after fine-tuning

Gemini API
,
1

I have this classification task which I want to employ Gemini to help with. The job is to classify a company into one of 30 industries from an input of a brief description. I have a lengthy system instruction on how to do this as well as the list of industries along with keywords to help with the classification.

Gemini 1.5 Flash 002 does a really good job of this just off the instructions, about 95% accuracy I would say. My thinking was that I would be able to bump this up to 99-100% with a few fine-tuning examples. So I created fine-tuned model off Gemini 1.5 Flash Tuning 001 and gave it the instructions and 100 examples.

Chatting to the the tuned model was a huge disappointment. Wildly inaccurate, maybe 30% and it even started making up its own industries (unhelpful). What am I missing here? It’s like it payed very little attention to the lengthy system instruction after tuning.

How should I continue? Can I fix the tuned model somehow or should I focus on the non-fine-tuned version. I’d be happy to use the 95% accurate model but I need to classify a ton of data through the API and I don’t want pass the instructions each time, it uses tokes too inefficiently.

Thanks

2

Hi @wije

Welcome to forum!

  • Perform fine-tuning again with a larger dataset, ensuring all 30 classes are included and represented evenly across the examples.
  • Test different epochs and learning rates to optimize the model’s performance.
  • Explore few-shot prompting by providing example cases directly to the model during inference.
  • You can use context caching for passing repeated instructions. Here is the link

Thanks

4

How do people know how to tuning effectively through API?
Add a description of the tuning features to the documentation.
Features of tokenization, loss function, regularization, LoRa parameters, scheduler, tuning method.
And generally make it customizable.

And finally correct the counting of characters.

6

@Susarla_Sai_Manoj Upping the number examples to the full 500 helped quite a bit! Thanks

1 Like