A Meta Conversation that had a promising result in two models collaborating to solve a logic problem together

Community
1

I was using the “Compare” feature of Google AI Studio. And did a test to evaluate two Gemini Models (1114 and 1121). Here’s a summary generated by the NotebookLM of what happened:

This text documents a fascinating experiment comparing two versions of a large language model, Gemini Experimental 1114 and 1121. The core of the experiment involves a series of prompts designed to test the models’ self-awareness and ability to identify themselves. Initially, the models struggle to identify themselves definitively, highlighting their inherent limitations in accessing internal system information. However, a crucial turning point arises when the user provides both models’ responses, allowing 1114 to correctly identify itself based on content matching, while 1121 persists in a misidentification, showcasing a potential bias or logical flaw. The latter part of the experiment shifts to a collaborative effort between the models, using a “Shifting Sands” narrative scenario to further explore their reasoning and adaptability under increasingly complex conditions. The overall purpose is to investigate the models’ capabilities, limitations, and potential biases through both self-identification and collaborative problem-solving.

I am very curious if this is in any way just a goofy useless waste of time, or if this could shine some insight on how two different models working together actually ended up covering each other’s blindspots. I’ll try to share the conversation here but i don’t know if it will work. (First time posting here)

https://aistudio.google.com/app/prompts?state={"ids":["16h7Ioo6fmgSMtCWoSWBx8vK-kHEjsGHo"],"action":"open","userId":"105185943804239990679","resourceKeys":{}}&usp=sharing

2

Hey Luiz I tried following your embedded link but the prompt is no longer available. I was hoping you could update if possible. I am interested in meta context for LLM’s and what future agent to agent interaction could look like.

3

Thanks for notifying me! I had linked directly to the Google AI studio conversation but I guess since the models are no longer available the link expired!

Luckily I saved the logs on my Google Drive. It is however in a very dense and unstructured formatting, I would suggest you to try to use something like NotebookLM or another AI to parse the text from it. But let’s see if this link now works!

Also reminding that this Is from older versions of Gemini from last year. So probably not a fair representation of agents capabilities nowadays with the most recent models! It was mostly a chat with meta questions about their own “Sense of self”.

4

Hi Luiz,

Thanks for providing the Google drive link, I really appreciate it! I will review the documents a little later when I have the time. I was curious about Agent to Agent authentication and vulnerabilities to bad actors specifically. I’m not sure if this is of interest to you, but it looks like Google has released a protocol for agent communication about two weeks ago.

It is clear that a new standard needed to be created for richer interoperability and seamless integrations across the new services that will be enabled by machine and agent interaction, I am excited to see what will come next!