Hello everybody,
Let’s say I have 100 examples (input/output pairs) where the input is a sentence and the output is a JSON object. Which approach is better in terms of the quality of responses: using these examples for many-shot learning with Gemini or fine-tuning Gemini on these examples? In which case will the model be able to catch patterns more effectively? isn’t it harder to train and get the desired parameters via fine-tuning than to make the model learn patterns with many-shot learning? (I think so). Please focus on the accuracy of the outputs, disregarding time and computational resources.
Thanks in advance!
While fine-tuning generally leads to greater performance gains as it allows the model to adapt its parameters to your specific task, it does require more computational resources, time, and can be prone to overfitting with limited data.
Few-shot learning, on the other hand, is attractive for its simplicity and efficiency, as it doesn’t require altering the model’s weights. However, its performance hinges largely on the knowledge acquired during pre-training and the design of your prompts.
Given your limited dataset of 100 examples, I would lean towards experimenting with few-shot learning first. However, the definitive answer requires empirical evaluation tailored to your specific task and data.
1 Like
I agree that a fine-tune is generally better, and uses less input tokens than multi-shot.
However, with only 100 training points, I am not sure you will get much mileage out of a fine-tune. You usually need thousands of examples, across various cases, and each case has multiple variants of the same type.
So if you don’t mind input token usage, maybe you can try most of your 100 examples, or maybe all of them, and see what happens.
But if your inputs are huge, you may have to resort to a fine-tune anyway.