Will there be a evaluation framework for Google AI models?

Gemini API
1

Hi!
Going into production with AI models requires careful assessment of the expected performance. Will there be a framework we can leverage to evaluate the Google LLMs?
I am sure this will be useful for researchers as well.

5 Likes
2

This sounds interesting. I wonder we can port the existing evals.

1 Like
3

In Vertex AI on Google Cloud in the Model Development section, there is an Experiment window that allows you to run experiments and then have the ability to track, visualize, and compare them. Is this sort of what you are looking for?

4 Likes
4

This is very interesting and thanks for sharing the link.
From a first quick screening of the extensive documentation this looks very interesting for fine-tuning and other model training jobs. Looking forward to learn more!

3 Likes