Model cannot focus on most recent user request when function calling

[ibgib]

OK, so I’ve implemented the plumbing that always provides parity between the FunctionCallPart and FunctionResponsePart in the Content. Now the model is behaving as expected. So it appears that when there is a FunctionCallPart that is not matched with a FunctionResponsePart, the model seems to believe that the function needs to be called again sometimes.

FunctionResponsePart quirk

There is also an issue regarding the response value and I’m including this here to help others with another quirk to get this workflow executing properly. In code (in @google/generative-ai node package) the FunctionResponsePart.functionResponse is typed as follows:

export declare interface FunctionResponse {
    name: string;
    response: object;
}

The response will error out if this is an array, even though in JavaScript an array is an object (typeof [] === 'object'). So this FunctionResponse is expecting a POJO, I imagine stemming from the actual API not being JS-specific. So I simply wrapped my raw function call result in a wrapper object, e.g., response : { value: rawResponse } and the model AFAICT understands this. This also wraps raw response values that may be undefined or null.

summary

So here is a short summary for multi-turn and multi-function calling, and this applies to the contents array when sending a message to the model:

• a content entry part of type FunctionCallPart must correspond to a FunctionResponsePart.
• the response content entry must…
  1. immediately follow the function call entry in the contents sent to the model.
  2. match in parts array length and function name in the parts array with the function call entry.
  3. have role set to "function" and NOT "user" as is given in the official example documentation.
  4. functionCall.response must be a POJO, not just be typeof === 'object'.

So it looks like a documentation tweak could have helped a lot here: include a multi-function-call multi-turn example, correct the role for the FunctionResponsePart, include info on the response “object” issue.