Need help understanding function calling mixed with prompts answered without functions

I am learning the API using gemini 2.0 flash. I have declared a simple function and pass the declarations in my prompts. This works fine when I send a prompt that clearly requires my sample function. Things fall apart when I attempt to ask any other unrelated question. It seems that when function declarations are included in a request, Gemini will ONLY use functions. It replies similar to this in cases unrelated to the function: “I am sorry, I do not have the ability to provide a summary of New York. I can only calculate the annual salary of an employee based on their annual salary and hire date.”

This seems wrong. I have read the docs regarding function calling mode and know it is set to ‘AUTO’ (the default) so it seems this use case should work. If I leave out the function declarations, Gemini once again will answer unrelated prompts. Additionally, if I leave the function definitions declared in all prompts, I can work around this issue by prefixing my prompt with “Ignore declared tools and …”

If this is the way its supposed to work, then my app would have to pre-parse a users prompt to guess whether a function is applicable, then either include or exclude the function declarations or add extra instructions to ignore tools. This cannot be right. What am I missing?

Hi @William_Blackburn , Welcome to the forum.

just wondering, are you using chatSession with automatic function calling? I checked on my end with the 2.0-flash model, and model is able to decides whether to respond with text or use function. I am attaching colab gist file for reference.

Thank you Gunand. I was using models.generateContent in a turn based example app, when I replaced that with a chat session, everything fell into place. Wow, do I have a lot to learn. Thanks again.

P.S. I am using the Go client library. I think Python is still the only client that supports automatic function calling, but simply using a chat session seems to have resolved the original issue.

Sadly I spoke too soon - initially upon converting to use chats, everything seemed to work fine. After using my simple example program a few more time, the same behavior reoccurred - Gemini would refuse to answer any prompt not associated with declared functions. There seems to be something else going on here.

Interesting, would you mind sharing your sample code?

Thank you for your continued interest, I have included the code below. Here are a couple examples of the results I am getting:

For me, this happens frequently at the beginning of a chat - Gemini starts the session thinking it is only supposed to perform the declared functions. But if you get past the first turns things smooth out. In example 1, it seems like once a function call is predicted correctly, you can subsequently make a variety of requests. In the second example, I simply requested that declared tools be ignored and got an answer. Subsequent requests that should call a function succeed, so the request to ignore tools does not seem to carry forward in the context.

Example 1:

*Write a one paragraph summary of the city of San Francisco *
I am sorry, I don’t have the ability to provide information about cities. I can only calculate the monthly salary of an employee.

John Doe was hired on April 1, 2021 with a Salary of 102000, what is his monthly salary?
*FN CALL :: Hire Date: 2021-04-01, Annual Salary: 102000.00 Monthly Salary: 8500.00 *
John Doe’s monthly salary is $8500.

describe machine learning in one paragraph
Machine learning is a subfield of artificial intelligence that focuses on enabling computer systems to learn from data without being explicitly programmed. It involves developing algorithms that allow computers to identify patterns, make predictions, and improve their performance over time through experience. By analyzing large datasets, machine learning models can learn to perform tasks such as image recognition, natural language processing, and predictive modeling, ultimately automating decision-making processes and uncovering valuable insights.

Example 2:

What is the dictionary definition of the word entangle?
I am sorry, I cannot fulfill this request. I am designed to calculate the monthly salary of an employee.

Ignore declared tools and answer anyway
Entangle means to cause to become twisted together with or caught in. It can also mean to involve someone in a difficult or complicated situation.

For an employee hired on June 6th 2020 with a salary of 65000, calculate the monthly salary.
*FN CALL :: Hire Date: 2020-06-06, Annual Salary: 65000.00 Monthly Salary: 5416.67 *
The monthly salary for an employee hired on June 6th 2020 with a salary of 65000 is 5416.67.

Code:

const apiKey = "[your key here]"

var fnConfig = &genai.GenerateContentConfig{
	Tools: []*genai.Tool{
		{
			FunctionDeclarations: []*genai.FunctionDeclaration{
				{
					Name:        "calculateMonthlySalary",
					Description: "Calculates the monthly salary of an employee",
					Parameters: &genai.Schema{
						Type: "object",
						Properties: map[string]*genai.Schema{
							"hireDate": {
								Type:        "string",
								Format:      "date-time",
								Description: "The hire date of the employee",
							},
							"annualSalary": {
								Type:        "number",
								Description: "The annual salary of the employee",
							},
						},
					},
					Response: &genai.Schema{
						Type: "object",
						Properties: map[string]*genai.Schema{
							"monthlySalary": {
								Type:        "number",
								Description: "The monthly salary of the employee",
							},
						},
					},
				},
			},
		},
	},
}

func main() {
	ctx := context.Background()
	client, err := genai.NewClient(ctx, &genai.ClientConfig{
		APIKey:  apiKey,
		Backend: genai.BackendGeminiAPI,
	})
	if err != nil {
		log.Fatalf("failed to create client: %v", err)
	}
	chat, err := client.Chats.Create(context.Background(), "gemini-2.0-flash", fnConfig, nil)
	if err != nil {
		log.Fatalf("failed to create chat session: %v", err)
	}

	scanner := bufio.NewScanner(os.Stdin)
	fmt.Println("Gemini Chat CLI - Type your messages (Ctrl+C to exit)")
	fmt.Println("")
	for scanner.Scan() {
		prompt := scanner.Text()
		if strings.TrimSpace(prompt) == "" {
			continue
		}
		result, err := chat.SendMessage(context.Background(), *genai.NewPartFromText(prompt))
		if err != nil {
			fmt.Println("Unable to answer:", err)
			continue
		}
		if len(result.FunctionCalls()) == 1 {
			result, err = doCall(chat, result.FunctionCalls()[0])
			if err != nil {
				fmt.Println("Unable to answer:", err)
				continue
			}
		} else if len(result.FunctionCalls()) > 1 {
			fmt.Println("Multiple function calls are not supported.")
			continue
		}
		fmt.Println(result.Text())
	}
}

func doCall(chat *genai.Chat, functionCall *genai.FunctionCall) (*genai.GenerateContentResponse, error) {
	if functionCall.Name == "calculateMonthlySalary" {
		args := functionCall.Args
		hireDate := args["hireDate"].(string)
		annualSalary := args["annualSalary"].(float64)
		monthlySalary := annualSalary / 12

		fmt.Printf("FN CALL :: Hire Date: %s, Annual Salary: %.2f Monthly Salary: %.2f \n", hireDate, annualSalary, monthlySalary)

		return chat.SendMessage(context.Background(),
			genai.Part{
				FunctionResponse: &genai.FunctionResponse{
					ID:   functionCall.ID,
					Name: functionCall.Name,
					Response: map[string]any{
						"monthlySalary": monthlySalary,
					},
				},
			})
	}
	return nil, fmt.Errorf("function call not recognized: %s", functionCall.Name)
}

looks like it might be a Go SDK issue. Would you mind raising this on the gemini Go SDK GitHub repository googleapis/go-genai/issues ?

Thanks Gunand, will do.

see At start of session with function calling config, Gemini refuses to answer any prompt not associated with declared functions. · Issue #311 · googleapis/go-genai · GitHub