So I went ahead and actually fine-tuned gemini pro model. My question now, how to do i check if my new fine-tuned model can make predictions based off of the dataset i finetuned it with? Thank you very much
As a rule of thumb, you split your overall data into a bucket with 80% of it and use that 80% to train (fine tune) the model. Then you use the remaining 20% to validate that the model is now performing significantly better compared to a baseline model with no training.
thats not what im asking. I think you misunderstood my question
I apologize for the misunderstanding. Perhaps you can say one or two more sentences explaining what the question is really about?
1 Like
Its ok. Thanks for trying to help. My questions is, I have fine tuned my Gemini model via Vertex AI platform. I fine tuned that model to give me soda consumption predictions by every state in America. I am using Flask as my backend. So im guessing I need to somehow embed the code provided by Vertex AI (results after fine tuning) into my Flask logic. When I fine tuned the Gemini model I used a dataset of 4 thousand rows and about 13 columns with soda consumption percentages per state/region. But for now I just want to make sure I get the Flask logic correctly to interact with my Vertex AI fine-tuned Gemini model. but before I start doing logic in flask, im wondering whats the way to test if my new finetuned model can actually predict soda sales. For example, in Vertex, i see that we can interact with the new tuned model via a prompt. but would prompting really be the way to test my new models prediction capability? Because when i use the prompt, it doesnt really predict consumption percentages (as I had fine tuned it with real government data). Or do i first need to do the Flask logic to see if my fine tuned model can predict by state. When i did the fine tuning i gave it thousands of examples, so the gemini model should be able to make some sort of prediction right?
I hope I have it right this time. During fine tuning, the model is exposed to a large number of datapoints paired with a particular desired response, which in this case should be âconsumption prediction for the next 6 monthsâ (or whatever period you are interested in). The expected model output is given to the model during fine tuning.
After fine tuning, prompts that match the inputs used in training and asking for the exact same target response (consumption prediction for the next 6 months) should perform significantly better than the same prompt used in an untrained model. To get a handle on how much better, it is useful to partition the overall dataset into a training and validation subset, and that is not a firm requirement.
In any case, even single prompts against your finetuned model should exhibit the improvement. The only important thing is, the provided data must be of the kind used in the training, and the expected prediction the prompt is asking for must be what was labeled âoutputâ during the training.
1 Like
interesting. yes, I had split my data into validation and training sets. I think i was holding off the test-sets until after. So I guess now I must apply the test sets via prompting? How would I exactly implement the testsets. I do have the JSONL format and JSON format of my original dataset. Am I suppose to copy/paste a row into the prompt (In Vertex AI screenshot of prompt page of my fine tuned model) and expect a prediction? So for example, right before tuning, i had to âinstructâ all my rows in order to tune Gemini. So an example row (from JSONL) would look something like this {user: // input text: Year, Text: How often soda consumed: blah blah, context: blah blah output text: 'Data_Value,: 23%} So, in my dataset, one row would have a percentage of soda consumption of that state. Its historical data. So I fine tuned Gemini four thousand rows (13 columns including engineered data). So how exactly do I prompt that? Do i just copy paste a row (of JSONL) as i listed above? Plus my app is to display predictions via a US map/heatmap. There will be no user input, just predictions for the next year. And yes I remember when I was finetuning my model that it asked me how my data was âtimedâ: by weeks, months, years- i remember that clearly. Also I just saw that you can finetune a model via Googls Studio AI. I tuned mine via Veretex because thats where i created a bucket for my dataset. Im gonna check out Google Studio AI-but whats the difference?
I have been using AI Studio. One difference I noticed from the Vertex screenshot you posted is that you have Freeform prompts available; they were removed from AI Studio during a recent update.
You donât lose the quality of the validation dataset when you use it to check the model performance, it is not âlearningâ from it. If you start a new chat session, context gets wiped out and you can re-test with validation data.
Either Vertex or AI Studio should give improved answers by fine tuning, the backend model is the same (with some naming convention differences). It sounds like you want the model to do some consolidation as well, as in give US total responses having been trained on per-state data. I havenât tried this specifically, one approach I think could work is
- in the prompt, give it a couple of rows from the validation dataset (excluding the target value, consumption)
- then ask it to predict the consumption for the states provided in the prompt AND to extrapolate the consumption for the other states not provided in the prompt
- you can also give it the instruction to add up the states, but large language models are not that good with math, you might need to give it a tool function to add 50 numbers
Hope that helps
1 Like
yeah, that prompt question you saw on the screenshot, was a sort of consolidation question where im asking for a combined soda consumption in america-that was just a random prompt. What I did now in Vertex prompt, was ask it for prediction for individual state and for some states it was able to give me a number, such as -3% to -5% for future prediction. So the model does well/ok when you asked for individual state. But the -3%, it says it got that prediction based off of soda consumption from 2024/2023. In the data i tuned it with, was dated all the way back from 2010 and so on. So I feel like its not even using the dataset i tuned it with. But this was the result from just a normal language text. Right now Im going to take your advice and provide it with a row or some rows from my validation set to see if I get a different number. I will post that here shortly. Thanks for all your input. If it wasnt for you , id be SUPER LOST.
So I went ahead and prompted the tuned model as you instructed. And got back some great results. Using Vertex AI prompt I inputed only two VALIDATE rows from my dataset (But I had a side question: why did i need to provide 2 VALIDATE rows instead of 2 TEST rows?) and then asked it to predict off of those two rowsâ states AND asked it to provide predictions off of other states. And it did so. It even randomly picked data (target value/Data_Value) to compare years from 2013 and 2019 without me asking it to. So I can see that its working/reading with the tuned data I had tuned it with before. I dont know if you can see from the screenshots but it gave me predictions in percentages (just like in the target value/Data_Value). So its all good. So does this mean my model/app is ready for deployment/production. I had a second question since I just learned about this yesterday: I saw this seciont/option in Vertex where I can Deploy To Endpoint/Deploy Your Model. What is that all about? It has this little description under it: Endpoints are machine learning models made available for online prediction requests. Endpoints are useful for timely predictions from many users (for example, in response to an application request). You can also request batch predictions if you donât need immediate results.
So, when I went ahead and created this new endpoint, I used my already tuned model to create this new endpoint. So whats the difference between my original endpoint (tuned model) to this new "Deploy To Endpointâ endpoint? In addition to my tuned model I wanted to my web app to display live/real time soda consumption predictions, its this what this new endpoint would be doing? (since I think thats what I understood from the little description posted under the âDeploy To Endpointâ button.
1 Like
I wouldnât fine-tune for knowledge, or for any sort of prediction. Use RAG for knowledge, mathematical models for prediction, and fine-tunes for tone or formatting issues.
1 Like
Hey Curt, I appreciate your input. At the moment I have already fine-tuned Gemini. What do you mean âuse mathematical models for predictionâ? Are you saying to use a different model from Gemini? Im confused. My web app is a soda consumption prediction. I tuned gemini with a dataset of 4K rows and 13 columns. I also want to integrate a live API to strengthen the prediction accuracy. The only mathematical input that I can think about regarding your comment, is doing regression task while preparing(cleaning, inputting blank data, etc) my dataset. But thats all been done.
The mathematical model can take different forms. Iâm thinking of a time series ⌠soda anmount over time ⌠so a Kalman filter or an Alpha-Beta filter could help with your predictions.
There are ML architectures that can handle time series, like RNNâs. But I would try traditional techniques first before going there. Unless you want to have fun trying to create a working ML/AI solution. 
2 Likes
If you need help creating a ML model for this you can reach out to me for help.
1 Like
Curt and user: âgrandell1234â, thanks for both your input. Curt, you are totally right. A time series model (such as ARIMA, SARIMA, or a transformer-based model) would probably be the best for predicting soda consumption (But I cannot use those models alone without using Gemini for the Gemini contest). If i would of known earlier that Gemini alone would not be the most efficient for forecasting tool, I would of reconsidered my approach. Although I do remember doing enough research before approaching my prediction project using Gemini. Maybe during my research thats when I was introduced to fine-tuning gemini with current dataset. Since Ive spent many weeks(feature engineering, data cleaning, splitting, filling in blanks, instructing data, research, etc) to where I am now in my project, I was considering possibly integrating a time-series model to my project along with my fine-tuned Gemini-(but I will explain why maybe integrating a time-series model may not be needed). The only way I can think about integrating that is by having those two models (Gemini and Time Series) point to the same deployed endpoint (which Ive already done). Its my first time working in AI/Machine Learning so this is all new stuff to me. There is another (hybrid) approach where I can have Gemini handle non temporal data (such as GeoLocation) and feed that into a time-series model that would handle temporal data(no clue how to do that yet). But I think that Hybrid approach would only complicate things. Because I think its best if my fine-tuned Gemini model can find/see relationships between temporal data and non-temporal data in order to predict better. For example, in my dataset of 4-thousand rows and 13 columns (the set used to fine-tune Gemini), I did some feature engineering (a combination of temporal and non temporal data). So after I fine-tuned Gemini with that dataset alone, as you can see in my previous screenshot, I was reasonably happy with the results of prediction. For example, as user [OrangiaNebula] suggested earlier that, in Vertex AI prompt console, I should prompt my fine-tuned-gemini model with a couple of-rows from my -validate-data-set and ask the finetuned-gemini-model to do some predictions. If you can see my earlier post, it was even able to predict by state. To give you more info about my dataset that i used for fine-tuning Gemini, its all historical data from only high school students (grades 9-12) regarding soda consumption from every state in America dating as early as 2007-2019-from government source-website. (Ofcourse If I have more time (Im working alone) Id like to get other datasets from different age groups to strengthen predictions). The fine-tuned-gemini model gave me predictions from different states of declines in soda consumption, some states had steeper decreases, while others had less decreases. So to double check this prediction, I used Perplexity.ai (just a rival of ChatGPT) to predict to me (based off of historical data) what soda consumption looked like for many states in America, and it also said, in general there was a decline overall for many states. And yes I want to make my app smarter. So thats where I am right now (before starting the frontend/backend of my web app). Soon I may start looking at hyper-tuning my fine-tuned model in Vertex (I mad the mistake of not looking into hypertuning before deploying to an endpoint-i just didtn wanna mess with the already default settings). Ive never hypertuned before. Also, the reason why I chose the Language flavored Gemini over the Multimodal flavored Gemini (see attachment) is because I only have one dataset with temporal/non-temporal data. I dont have any other media right now that I have to consider fine tune it with. Please let me know your inputs. And also when I deployed to an endpoint, it says on Vertex, that deploying to an endpoint is for online predictions lol. So that really confused me. Do online predictions mean exactly the type of predictions Im trying to accomplish with my web app?
1 Like
grandell1234, please see my post above
I saw it, donât worry about pinging me, I have read every post created on this forum since it was created.
I am familiar with perplexity
I am not sure, do you have some documentation links or screenshots?
heres what I mean. The bottom endpoint: âtuned_2varsaddedâ was my first finished fine-tuned gemini. Then the endpoint above 'from_tuned_to_deploy_to_endpoint" is the result of deploying my fine-tuned model to an endpoint.
Curt, I appreciate the info on these advanced âtime-series-forecastingâ techiques which can be applied with tensorflow in the near future. I think what I need to do first is, somehow get the predictions im getting from my finetuned gemini model to be displayed individually by state (example California, Texas, etc) using Flask. Once I get that going, im assuming that those predicted numbers by gemini can then be refined with those advanced techniques you mentioned earlier. So my current question is: how do i get/do the logic for individual states to display their unique prediction number of soda consumption? Is this logic written on Flask or done somehow inside Vertex? And I think whats even a more basic question is, whats the code to only just connect my current already deployed endpoint to my backend flask file, would be great! I think what would be great from the community. I will work on that now and possibly look at some hypertuning and let you guys know my update. Once I get the app fully communicating with geminis prediction numbers, I will go back to refine with advanced methods, but since project is due August 12, im trynna at least make a full basic connection from endpoitn to backend/frontend. Since all the advanced techniques will be done with tensorflow, id first need to get those gemini numbers rolling in. Thanks community!!
Normally you just make API calls to whatever AI model you want to use. This sends data over the internet, and you get your AI response back. It works in pretty much every programming language or framework.