Get logprobs at output token level

Gemini API
,
1

:wave: Hey there, I was wondering if it was possible to get logprobs for each individual token in the output of a Gemini model.
In all the examples I find online (here), it seems I can only get avg_logprobs. Basically an average of logprobs for all output tokens. This defeats the purpose of more fine-grained control of what we receive and how we use it.
In my case specifically, I receive a complex JSON output and I need to compute model confidence (from logprobs) from some specific entries in the JSON only.
Essentially what OpenAI offers since day 1 (here).
Any hints please?

Addendum
It seems with gemini-1.5-flash-002 I can invoke the model with

generation_config = genai.GenerationConfig(response_logprobs=True)

This might go in the right direction according to docs, but it’s unusable. After 3 calls I get

ResourceExhausted: 429 Unable to submit request because you've reached the maximum number of requests with logprobs you can make per day. Remove logprobs from the request or try again tomorrow.
4 Likes
3

is there any updates?

4

Unfortunately not that I know of

5

Would love an update on this too

Logan had posted that this would be available ~4 months ago (x.com). But can’t seem to be find anything in the documentation.

6

Hi there,

Using an enabled responseLogprobs setting in the generationConfig and using models gemini-1.5-pro-002, gemini-1.5-flash-002 gemini-2.0-flash-exp, or gemini-exp-1206 gives me an HTTP 400 as response:

{
  "error": {
    "code": 400,
    "message": "Logprobs is not supported for the current model.",
    "status": "INVALID_ARGUMENT"
  }
}

Does anyone have a successful request?

Cheers

1 Like