Image caching leads to wrong behavior of gemini?

Gemini API
,
1

I am doing a task that gemini-2.0-flash should analyze a user input image according to some rules. For gemini’s better understanding, I tried to use explicit caching to cache some few-shot images. The user input image is put in genai.Content{}.

The overall prompt structure is as below.

genai.Contents: {
    rules 1,
    user input image,
    rules 2
}

cache: {
    few-shot 1 description,
    few-shot 1 image,
    few-shot 2 description,
    few-shot 2 image
}

However, I found that after caching the few-shot images, gemini cannot correctly recognize the user input image. It always analyzes the last few-shot image, or even the “image“ that doesn’t exist.

Gemini works fine if I also put the few-shot images in genai.Content{} (no caching), so I think there may be something wrong when caching is used.

Has anyone run into the similar problem before?

3 Likes
3

Yes. I am having the exact same issue. It affects all the models, not just Gemini 2.0. I’ve reported via multiple channels but currently I don’t have any resolution.

There’s more info on this GitHub issue: Critical bug: Vertex API with context cache leaks state between generate_content calls · Issue #1634 · googleapis/python-genai · GitHub

I also reported via Google VRP (since this has security/privacy implications) and this is the internel Google ticket, but the Google reviewer didn’t understand what I was saying. Google Issue Tracker (link only accessible within Google)

-Adrian

2 Likes
5

Thanks for the information. Hope Google will fix the problem soon.

7

Hi @hedy @Adrian_Cable This issue is now resolved. Please confirm if you are still facing it.

9

@Pannaga_J - the issue is resolved but I now need your assistance to help correct an injustice.

This issue was originally reported by me via the Google VRP (issu.ee/455616013). The reviewer did not understand the report and dismissed the concern as ‘intended behavior’ without looking into it. I am glad that after some prodding Google took the right course of action on this issue and took care of it quickly, but the optics of the sequence of events isn’t great:

  • Security researcher (me in this case) submits potentially serious security/privacy issue via Google VRP, following rules of responsible disclosure (i.e. no public reporting)
  • Google dismisses the researcher’s concerns, says it is ‘working as intended’ when that is clearly not the case
  • Researcher then has no other option but to publicly report the issue (via Google SDK GitHub) - not good that this has to be done, because public disclosure of potential security/privacy issues raises the chances of exploitation by bad actors
  • Google then in the background presumably recognizes the seriousness of the issue and fixes it, without acknowledging the researcher’s report and therefore bypassing the VRP bug bounty process

Just to be clear, I’m not saying there’s any deliberate bad faith on Google’s part here, but I hope you can see the above is not a good look for Google within the security community and I would like to nip this one in the bud and, hopefully, work with you to (1) get this one re-opened for bug bounty in the VRP program, (2) identify ways we can avoid this kind of thing from happening in the future, either to us or to others within the security research community. I’m very open to your suggestions.