Can't prompt Gemini with media hosted on S3?

Gemini API
,
1

Hey all, im in a weird predicament here and i dont know what to do from here. I’m unable to use the GoogleAIFileManager package to upload the media I need to due to its need to run server-side, which my NextJS project cant do because i’m doing this in a ChakraUI component, which can only run client side.

I’m trying to upload the video i’m using to Amazon S3, and that works fine. But when i put the S3 presigned URL into Gemini, it throws this error:

{
“error”: {
“code”: 400,
“message”: “Invalid or unsupported file uri: https://makethedocs-videos.s3.us-east-2.amazonaws.com/2024-06-02%2009-28-35.mp4…(Long URL)”,
“status”: “INVALID_ARGUMENT”
}
}

I’m not sure, but i suspect Gemini is blocking any domain that isn’t storage.googleapis.com (or wherever Gemini content gets uploaded to when using GoogleAIFileManager) for passing media into Gemini.

I need help quick. Will answer any questions needed or provide any extra info

1 Like
2

Correct, the Gemini API can only access media from one of two locations:

  • If you’re using AI Studio’s API - it must be a URI provided by the File API
  • If you’re using Vertex AI - it must be a GCS URI

You can’t use other URLs.

It isn’t clear to me, however, why you can’t run it client side (aside from it being a bad idea to use API keys client side). This is the GoogleAIFileManager class that is part of the generative-ai JavaScript package, right?

1 Like
3

Thanks for the reply! Follow up question, does it say that in the documentation anywhere, or is this an undocumented thing?

Also regarded not being able to use the GoogleAIFileManager package, it NEEDS to be run server side because it depends on fs and that can apparently only run server side. Not sure if this is true, but it’s the only conclusion I was able to draw as to what the hell is going on.

2 Likes
4

It is clearly documented as part of the Vertex AI Gemini API. It is less clear in the AI Studio Gemini API, however I’ve seen it stated somewhere… just can’t find it at the moment.

And yeah, looking at GoogleAIFileManager, I see what you mean.

You can still upload it using REST. See cookbook/quickstarts/file-api/sample.sh at main · google-gemini/cookbook · GitHub for an example of how to do so.

(As an aside - this is motivating me even more to get a Media Manager finished asap.)

1 Like
5

I tried to switch to the Vertex AI package, same issue with the Module not found: Can't resolve 'fs' error. Same error with the @google-cloud/storage package. Honestly running out of ideas on what to do. S3 seems to be the only package that doesnt get that FS error.

1 Like
6

You can use the Firebase packages to access Google Cloud Storage from a web client and to access Vertex AI.

See

(And I’d love to push LangChain.js at this point, but I haven’t finished the Media Manager which would let you store things on S3 and get a File API URI for it.)

1 Like
7

I finally figured out a solution that worked for me

  1. Client uploads video to S3, then calls to custom API with S3 file key
  2. Custom API downloads file from S3 into /tmp directory, then uploads the file to Gemini on the server side with @google/generative-ai/files package, returns with URI from Gemini
  3. Client side finally generates using @google/generative-ai package and passes in File URI provided by custom API

Better flow chart I made in Figma

horriblesolution
horriblesolution1367×617 61.3 KB

3 Likes
8

I’m currently struggling with something like this. I had already thought of saving into a /tmp folder, just didn’t know i would need the files package. About to try that. Hope it works