Functions and their usage

Lets talk about functions. How should functions work? IMO currently they do not they are broken. Either that or i completely miss understand how its supposed to work.

Lets look at this example:

Here we have a function called 'get_product_price_description_and_information with a description of ‘A tool used to get information about an items, description or price based upon its name’

get_info = glm.Tool(
            function_declarations=[
                glm.FunctionDeclaration(
                    name='get_product_price_description_and_information',
                    description="A tool used to get information about an items, description or price based upon its name",
                    parameters=glm.Schema(
                        type=glm.Type.OBJECT,
                        properties={
                            'product': glm.Schema(type=glm.Type.STRING),
                            'query': glm.Schema(type=glm.Type.STRING)
                        },
                        required=['product', 'query']
                    )
                )
            ]
        )

Great makes sense to me. However it does not make sense to the AI. As you can see by the name and the description I have done everything I can to get the AI to understand what the function is for. I cant tell is it the name of the function it uses to decide if its a match, is it the description or is it the name properties? is it some combination of all of them or some of them.

Example:

User: “How much does a Google Pixel seven cost?”
answer: I cannot fulfill this request. I do not have the functionality to determine the price of a product.

User: "What do you know about Google Pixel seven?
model: get_product_price_description_and_information

User: “Tell me about Google Pixel seven?”
answer: I cannot fulfill this request. The available tools lack the desired functionality.

This actually gets worse due to the fact that you will sometimes get different responses even if you run it again. The same request will occasionally hit the function and other times not.

Generic Questions

Then there is the issue of it answering generic questions it should be able to answer in general. If it has functions it will only attempt to answer questions related to functions it has ignoring all other requests.

User “Who is James T. Kirk”
model: I cannot fulfill this request because I lack the corresponding tools.

This has forced me to have a fall back on all my systems with Functions that catch this No tool and sends it to another model object that does not have tools enabled. Causing me to make two calls to the AI when i could have just used one.

So i guess the question there is then. If you add functions should it only respond to the configured functions and ignore all other requests?

Can we get some clear documentation on how we are to configure functions for them to be detected by the AI reliably the key her being reliability because if they are not reliably detected then I fail to see the point.

Thanks for the post!
Quick question - which docs were you referring to? This one? Tutorial: chamada de funções com a API Gemini  |  Gemini API  |  Google for Developers

Taking a closer look.

Another follow up -
would you mind sharing the rest of the code so we can better understand and try to reproduce the issue on our end?

Think my main point is there is no guidance in how to design functions. The code itself works. Its how the ai responds that’s the issue.

Name + description to best enable the ai to detect it reliably

I will post you an example in the morning if I have time I am in a meeting most of the day

Hi Linda,

To answer one of your specific questions, when deciding whether to use a function the model uses the function name, function description, parameter names, and parameter types.

I’ve tried to reproduce your results but am not seeing some of the behavior you’re mentioning on gemini-1.0-pro. Which model are you using?

I used the get_info from your example exactly, then

model = genai.GenerativeModel(model_name='gemini-1.0-pro', tools=[get_info])
response = model.generate_content('How much does a Google Pixel seven cost?')
response.candidates[0].content.parts

I ran it several times and got variations of

[function_call {
  name: "get_product_price_description_and_information"
  args {
    fields {
      key: "product"
      value {
        string_value: "Google Pixel seven"
      }
    }
    fields {
      key: "query"
      value {
        string_value: "How much does it cost?"
      }
    }
  }
}
]

I did reproduce the “Who is James T. Kirk” case and am checking what the expected behavior is.

As an aside, the Python SDK can handle creating Tools for you. This is roughly equivalent to your example:

def get_product_price_description_and_information(product: str, query: str):
    """
    A tool used to get information about an items, description or price based upon its name
    """
    pass

model = genai.GenerativeModel(model_name='gemini-1.0-pro',tools=[get_product_price_description_and_information])

We can confirm by making a FunctionLibrary manually:

get_info = genai.types.FunctionLibrary(get_product_price_description_and_information)
get_info.to_proto()

Outputs:

[function_declarations {
   name: "get_product_price_description_and_information"
   description: "\n    A tool used to get information about an items, description or price based upon its name\n    "
   parameters {
     type_: OBJECT
     properties {
       key: "product"
       value {
         type_: STRING
       }
     }
     properties {
       key: "query"
       value {
         type_: STRING
       }
     }
     required: "product"
     required: "query"
   }
 }]
1 Like

Just a note I am just playing I want to do a video on functions, but I also have a project im working on that I would really like to use them for but it will require that i figure out how to get them to reliably pick up.

Let me know what you think maybe i’m doing something wrong. Could we maybe get a description on the properties so that we can describe what they are so the ai could pick it up if a user uses different terms. Or should we maybe have more then one proptery so if the user says

product, item, shoe

My code

App.py


from dotenv import load_dotenv
import google.generativeai as genai
import google.ai.generativelanguage as glm
import os

from data import get_products_data

load_dotenv()

# name of the AI model used in this call.
CHAT_MODEL_NAME = os.getenv("CHAT_MODEL_NAME")


def get_prompt(message: str, file: str) -> str:
    # Open the file in read mode
    with open(file, 'r') as file:
        # Read the entire contents of the file
        file_contents = file.read()
        return file_contents.replace("{User's Question}", message)


class GeminiService:
    generation_config: str = {
        'temperature': 0.9,
        'top_p': 1,
        'top_k': 40,
        'max_output_tokens': 2048,
        'stop_sequences': [],
    }

    safety_settings: list[str] = [{"category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_NONE"},
                                  {"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_NONE"},
                                  {"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "threshold": "BLOCK_NONE"},
                                  {"category": "HARM_CATEGORY_DANGEROUS_CONTENT", "threshold": "BLOCK_NONE"}]

    def __init__(self):
        genai.configure(api_key=os.getenv("API_KEY"))
        check_inventory = glm.Tool(
            function_declarations=[
                glm.FunctionDeclaration(
                    name='check_inventory',
                    description="Function for checking inventory",
                    parameters=glm.Schema(
                        type=glm.Type.OBJECT,
                        properties={
                            'product': glm.Schema(type=glm.Type.STRING),
                            'query': glm.Schema(type=glm.Type.STRING)
                        },
                        required=['product', 'query']
                    )
                )
            ]
        )
        get_info = glm.Tool(
            function_declarations=[
                glm.FunctionDeclaration(
                    name='get_product_price_description_and_information',
                    description="A tool used to get information about an items, description or price based upon its name",
                    parameters=glm.Schema(
                        type=glm.Type.OBJECT,
                        properties={
                            'product': glm.Schema(type=glm.Type.STRING),
                            'query': glm.Schema(type=glm.Type.STRING)
                        },
                        required=['product', 'query']
                    )
                )
            ]
        )
        self.model = genai.GenerativeModel(model_name=CHAT_MODEL_NAME,
                                           tools=[get_info, check_inventory],
                                           generation_config=self.generation_config,
                                           safety_settings=self.safety_settings
                                           )

        self.model2 = genai.GenerativeModel(model_name=CHAT_MODEL_NAME,
                                            generation_config=self.generation_config,
                                            safety_settings=self.safety_settings)

    def single_completion(self,
                          message: str) -> str:
        response = self.model.generate_content(message)

        # if the response is a detected function call.
        if response.candidates[0].content.parts[0].function_call:
            response_function_call_content = response.candidates[0].content
            fc = response.candidates[0].content.parts[0].function_call
            if fc.name == 'check_inventory':
                query = response.candidates[0].content.parts[0].function_call.args["query"]
                product_name = response.candidates[0].content.parts[0].function_call.args["product"]
                product_id = get_products_data("name", product_name, "supplier_id")
                product_description = get_products_data("name", product_name, "description")
                product_price = get_products_data("name", product_name, "price")

                prompt = get_prompt(query, "prompt_formula.txt")
                formula = self.model2.generate_content(prompt)
                return formula.text
            elif fc.name == 'get_product_price_description_and_information':
                print("get_product_price_description_and_information")
                query = response.candidates[0].content.parts[0].function_call.args["query"]
                product_name = response.candidates[0].content.parts[0].function_call.args["product"]
                product_id = get_products_data("name", product_name, "supplier_id")
                product_description = get_products_data("name", product_name, "description")
                product_price = get_products_data("name", product_name, "price")

                prompt = get_prompt(query, "prompt_formula.txt")
                formula = self.model2.generate_content(prompt)
                return formula.text
            elif fc.name == 'fallback':
                response = self.model2.generate_content(message)
                return response.text
            else:

                return f"Unsupported function: {fc.name}"

        # probably a text response.
        return response.text


class GeminiUI:
    def __init__(
            self
    ) -> None:
        self.service = GeminiService()

    def single_answer(self, message: str) -> str:
        prompt = get_prompt(message, "prompt.txt")
        return self.service.single_completion(message=prompt)


gemini = GeminiUI()
answer = gemini.single_answer("Who is James T. Kirk")
print(f"answer: {answer}")

Data.py


suppliers_data = [
    {"name": "Samsung Electronics", "address": "Seoul, South Korea", "contact": "800-726-7864"},
    {"name": "Apple Inc.", "address": "Cupertino, California, USA", "contact": "800–692–7753"},
    {"name": "OnePlus Technology", "address": "Shenzhen, Guangdong, China", "contact": "400-888-1111"},
    {"name": "Google LLC", "address": "Mountain View, California, USA", "contact": "855-836-3987"},
    {"name": "Xiaomi Corporation", "address": "Beijing, China", "contact": "1800-103-6286"},
]

products_data = [
    {"name": "Samsung Galaxy S21", "description": "Samsung flagship smartphone", "price": 799.99, "supplier_id": 1},
    {"name": "Samsung Galaxy Note 20", "description": "Samsung premium smartphone with stylus", "price": 999.99,
     "supplier_id": 1},
    {"name": "iPhone 13 Pro", "description": "Apple flagship smartphone", "price": 999.99, "supplier_id": 2},
    {"name": "iPhone SE", "description": "Apple budget smartphone", "price": 399.99, "supplier_id": 2},
    {"name": "OnePlus 9", "description": "High performance smartphone", "price": 729.00, "supplier_id": 3},
    {"name": "OnePlus Nord", "description": "Mid-range smartphone", "price": 499.00, "supplier_id": 3},
    {"name": "Google Pixel 6", "description": "Google's latest smartphone", "price": 599.00, "supplier_id": 4},
    {"name": "Google Pixel 5a", "description": "Affordable Google smartphone", "price": 449.00, "supplier_id": 4},
    {"name": "Xiaomi Mi 11", "description": "Xiaomi high-end smartphone", "price": 749.99, "supplier_id": 5},
    {"name": "Xiaomi Redmi Note 10", "description": "Xiaomi budget smartphone", "price": 199.99, "supplier_id": 5},
]


def get_products_data(name: str, equals: str, respond: str):
    for product in products_data:
        if product[name] == equals:
            return product[respond]
    return None  # Return None if the product name is not found


inventory_data = [
    {"product_id": 1, "quantity": 150, "min_required": 30},
    {"product_id": 2, "quantity": 100, "min_required": 20},
    {"product_id": 3, "quantity": 120, "min_required": 30},
    {"product_id": 4, "quantity": 80, "min_required": 15},
    {"product_id": 5, "quantity": 200, "min_required": 40},
    {"product_id": 6, "quantity": 150, "min_required": 25},
    {"product_id": 7, "quantity": 100, "min_required": 20},
    {"product_id": 8, "quantity": 90, "min_required": 18},
    {"product_id": 9, "quantity": 170, "min_required": 35},
    {"product_id": 10, "quantity": 220, "min_required": 45}]

.env

API_KEY=[redacted]
CHAT_MODEL_NAME=gemini-pro

Hi Linda, follow-up to an earlier point; models always trying to use a tool when provided tools is not the intended behavior, but is the current behavior and a known issue. It’s being worked on!

Would love to take a look. Although I can probably infer from the code, can you also describe what you’re trying to accomplish/demo at a high level to make sure I’m understanding correctly? Also, how would you describe your Tools as actual Python function signatures and docstring?

One thing I’m wondering about is whether a ‘query’ parameter is too vague. Also, you could imagine e.g. get_product_price_description_and_information having only product_name as an input, returning the whole dict with all the product info, and then letting the LLM craft a response to the original user query with that context.

Just have time for a skim for now, but should have more time tomorrow or next week to look more carefully.

I would have to question why the AI cares what the Python function signatures and docstring are? I could call the function read_crap ( one, two, three). I dont understand why this would effect the AI tool unless you expect us to be sending the python function to the ai can it run python? if so that would be cool. maybe not for say java developers though.

No argument there but i couldn’t figure out what else to call it basically why i am coming here trying to understand if we are going to have to become better at naming things just to get this to work?

This was more to understand your use case, but is also relevant with the Python SDK. If you pass a Python function to GenerativeModel(tools=) it will get auto-converted to a FunctionDeclaration for you as shown in my previous example. Also if you’re using a ChatSession the code CAN automatically call the functions for you with model.start_chat(enable_automatic_function_calling=True).

Possibly. :slight_smile: Having well designed and named functions helps both models and humans understand what’s going on. Happy to provide more specific feedback if you explain what you’re end goal is/what you’re trying to accomplish. As is, it’s not clear to me why get_product_price_description_and_information needs a query parameter at all. The typical use would be to just return a FunctionResponse in the context of a multi-turn conversation and let the model figure out the next response. See Manual function calling in cookbook/quickstarts/Function_calling.ipynb at main · google-gemini/cookbook · GitHub for a more complete example.

Nice move regarding the flag enable_automatic_function_calling which I also discovered earlier today in the Python sources. It seems that this is undocumented or are there any references already?

I’m asking because I was working on the implementation in C#, and I’m a bit confused how the function_responses are extracted from the tools object. It is also interesting to see how the afc is added to the history collection. However, to my understanding afc is not handled in the rewind method.

Or am I barking up the wrong tree?

The Python SDK generated docs haven’t been updated due to a tooling issue, but it was recently worked on so hopefully the site is refreshed soon or with the next release.

1 Like

Cross posting this item

It has a bearing on documentation: the documentation for FunctionDeclaration (https://ai.google.dev/api/rest/v1beta/Tool#FunctionDeclaration) mentions adherence to Open API 3.03 specification repeatedly, but fails to explicitly point out that the period, not listed in the set of available characters when composing name, is excluded in a departure from the Open API specification for name. According to Open API specs, implementations may diverge, if they specifically point out where they diverge.

Of course, allowing periods in names and eliminating this departure from Open API 3.03 would be preferable, in my opinion.

1 Like