Fine-tuning Gemini works via AI Studio, but not via REST API

Gemini API
1

A question on fine-tuning a Gemini text model.

  • When using AIStudio w/ structured data, everything works for me as expected: I can upload a CSV of 50-100 examples, fine-tune the model, and when I eval on test data I get coherent output.

  • When I do the same thing using google.generativeai.create_tuned_model  |  Google AI for Developers  |  Google for Developers, providing input (unstructured) via the JSON upload, I get pretty gibberish results from the fine-tuned model–it feels like the results from a very un-tuned LLM. (Like, random repeating numbers, snippets of text repeating, etc.)

I assume that AIStudio, under the hood, is doing something different for structured data than I am doing with my REST calls. Is it well documented somewhere? Any suggestions?

1 Like
2

Hey there!

What does your API call look like? If you’re sending unstructured data, that may be why you’re encountering issues. That’s an easy fix though! All we gotta do is figure out how to structure the data properly so you can make successful calls with the REST API.

1 Like
3

Thanks, Macha.

The particular inference I am doing is something where I have a bunch of input fields and a single output. So in AI Studio, I have columns like color, shape, texture (for example) and a single output column.

I’m sending data like below, where inputs is like ["color: red", "shape: triangle", "texture: soft"]:

        examples = [
            {
                "text_input": "\n".join(inputs),
                "output": output,
            }
            async for inputs, output in get_examples()
        ]

    operation = genai.create_tuned_model(
        source_model=base_model[0].name,
        training_data=examples,
        id=args.model,
        # TODO: tune the below:
        epoch_count=100,
        batch_size=4,
        learning_rate=0.001,
        temperature=0.0,
    )

What am I wondering is whether that’s, like, correct? :slight_smile:

Thanks!

2 Likes
4

Ah, Okay.

So, just to inform you, you are not calling the REST API, but the Python API :wink:.

Think of the REST API as the “base” schema for which any programming language can interact with their toolset. Everything stems from the REST API, and all the other APIs basically wraps over the REST to either make them easier to call, or to have them fit the programming language better.

That being said, I don’t see anything too out of the ordinary with your code or structure.
I’m looking at these two docs:

and
https://ai.google.dev/gemini-api/docs/model-tuning/python

If you are not getting errors, but are receiving outputs that are janky, we need to take a look at the tuning data.

I could see the problem could be either 1 of 2 things:

The first one would be with async. I could be wrong here, but I’m very wary about how it’s being called here. Essentially, async is telling the computer to keep running the rest of the code even if this function isn’t finished executing. So, by calling it in here:

It’s possible that you’re giving the tuned model incomplete, jarbled, or empty data, because the operation is trying to run things before it has the completed value that’s inside training_data. However, I’m surprised that something like this wouldn’t throw an error if it was incorrect. Still, I think it’s worth looking into.

Otherwise, it’s possible you may be encountering issues because of how you trained the model, and how you use it after tuning.

Based off of your data, every call to the model will need the exact format. So, for your case, you’d need to make sure you structure your inputs for your tuned model precisely like this:

color: [value]
shape: [value]
texture: [value]

Otherwise it would barf.

And I am talking about inputting to the model after it’s tuned. Tuning essentially locks you into the desired structure/format you want, so if the above is the structure you intended, make sure nothing you input deviates from that.

And finally, remove temperature=0.0,, just to be on the safe side, as I do not currently see that in the docs as a legitimate parameter (yet).

Let me know if this helps, or if you have any other questions!

1 Like
5

Thanks, Macha.

I’m aware of the difference between REST APIs and Python clients to REST APIs. :wink:

I believe the problem here, just for others who stumble upon this, is that my production code was calling the fine-tuned model with the existing prompt + few-shot examples I had been using for the untuned models. But this causes the fine-tuned models to misbehave as described. (I’m not sure exactly why, but I assume the fine-tuned models are using SPT or some other PEFT technique, and continuing to provide the prompt somehow doesn’t work right.)

So this had nothing to do with training via the API vs the GUI, and everything to do with the fact that fine-tuned models need only an example for inference, not any few-shot prior examples or prompting.

Thanks for the help!

1 Like