Hi
I saw that recently logprobs have been added into Gemini-1.5-flash/pro-002
Do you think it is good idea to use logprobs for calculating confidence for documents separation, classification, and extraction?
Are there other any parameters which could help with that?
Is there available any documentations about that in context of confidence?
Logprobs really are optimal in a situation where you have prompted the AI as a classifier of its own, and instructed it with the exact outputs it can produce, with no possibility of it deciding to chat.
For example, a schema provided for a JSON, a list of options that are acceptable, instructions that the output is being sent to an API, can make a particular parsed position in output return logprobs that are not just the produced value, but scoring of the alternates, which can be used as more weighting.
"The only acceptable response output to the API starts with:
{"relevance_to_forum_topic": "
“Confidence” is not really something that can be evaluated with quality. It has been proposed that one could evaluate the perplexity over a sequence, but perplexity itself is certainty of an output, and “As an AI language model” can be very certain once it enters that pattern, for example of why it can’t work for evaluating truthfulness or correctness.