I have come so damn close, but alas, I have come at a crossroads, and I can’t continue without some help from the cracked devs I know and love here.
Part of the reason I’ve avoided help on the topic is because, well, most people are not using C# here. I’m aware it’s not officially supported (yet), but it is the language Unity uses, for better or for worse.
That being said, this might be a fun puzzle, and if I can get this working, I can dump it into a repo/library for others to use if they want.
So, What’s the situation, and what’s the issue here?
One word: Serialization.
To pass a JSON over to Gemini (or any LLM API for that matter), the correct JSON request must be sent in the correct format, or it will interpret it as a bad request.
This is very simple and easy for most languages and platforms; it is notably a pain in the ass with Unity and C# I am discovering.
I cannot use JsonUtility (their built-in serializer), because it does not support serializing dictionaries. This already sucks, because as many who work with these kinds of APIs know, that’s pretty much what the JSON objects are: dictionaries of dictionaries, so on and so forth.
So, instead I have to use the (unlisted but still valid thank god) Newtonsoft library for advanced serialization. If you program in plain .NET C#, this is actually the same library you are likely using to serialize your stuff in. So, this is ultimately a C# question, not a Unity one.
The Problem
Function call definitions are very finicky, and it’s challenging to identify the culprit of a bad request because you don’t get nice debug statements telling you any specifics about what you’ve done wrong. You also can’t just throw debug logs in the code execution to identify the culprit, because the problem is that the serialized object might look fine up to completion, but makes the API barf anyway. I can’t identify why the serialized object is being rejected by the request.
The Goal
It was, at first, easy and perfect. Construct the entire request as an object / class, serialize it, and send it over. This means that I create class definitions for each component, and can easily construct/deconstruct requests at will with very few lines of code.
Just so we are clear, this technique worked before I implemented function calling. You can see the basic structure below:
The Class Definitions that work
[Serializable]
public class Request {
[JsonProperty(Order = 1)] public SystemInstruction system_instruction;
[JsonProperty(Order = 2)] public Contents contents;
//[JsonProperty(Order = 3)] public Tool[] tools; //The array to hold tool objects
public Request(SystemInstruction Sys, string ContentText, Tool[] tools = null) {
this.system_instruction = Sys;
this.contents = new Contents(ContentText);
// this.tools = tools; <- we'll get to the tools here in a minute
}
}
[Serializable]
public class SystemInstruction {
[JsonProperty(Order = 1)] public Parts parts;
public SystemInstruction(string text) { parts = new Parts(text); }
}
[Serializable]
public class Contents {
[JsonProperty(Order = 1)] public string role = "user";
[JsonProperty(Order = 2)] public Parts parts;
public Contents(string text) { parts = new Parts(text); }
}
[Serializable]
public class Parts {
[JsonProperty(Order = 1)] public string text;
public Parts(string text) { this.text = text; }
}
This was fine for simple requests that included a system instruction.
When I added tools though, this is where things began to break down.
To test and ensure that I could even send the kind of function I wanted in the first place, I made a test string using JObjects to ensure that I could successfully make a function call in some manner. Lo and behold, it does work (not focusing on hallucinations here, just valid requests):
The Testing JSON (prepare your eyes)
new JProperty("system_instruction",
new JObject(
new JProperty("parts",
new JObject(
new JProperty("text", "You do stuff, so do stuff (not actual instruction).")
)
)
)
),
new JProperty("contents",
new JObject(
new JProperty("role", "user"),
new JProperty("parts",
new JObject(
new JProperty("text", "This is a test to determine if the function call JSON request is readable to you. Please provide a valid response.")
)
)
)
),
new JProperty("tools",
new JArray(
new JObject(
new JProperty("function_declarations",
new JArray(
new JObject(
new JProperty("name", "cast_function"),
new JProperty("description", "[actual descriptions in OG code]."),
new JProperty("parameters",
new JObject(
new JProperty("type", "object"),
new JProperty("properties",
new JObject(
new JProperty("param 1",
new JObject(
new JProperty("type", "object"),
new JProperty("properties",
new JObject(
new JProperty("name",
new JObject(
new JProperty("type", "string"),
new JProperty("enum", new JArray("Foo", "Bar")),
new JProperty("description", "[actual descriptions in OG code].")
)
),
new JProperty("special_number",
new JObject(
new JProperty("type", "string"),
new JProperty("enum", new JArray("-1", "0", "1", "2")),
new JProperty("description", "[actual descriptions in OG code].")
)
)
)
),
new JProperty("description", "[actual descriptions in OG code].")
)
),
new JProperty("param 2",
new JObject(
new JProperty("type", "array"),
new JProperty("items",
new JObject(
new JProperty("type", "string"),
new JProperty("enum", new JArray("Item 0", "Item 1", "Item 2", "Item 3")),
new JProperty("description", "[actual descriptions in OG code].")
)
),
new JProperty("description", "[actual descriptions in OG code].")
)
),
new JProperty("param 3",
new JObject(
new JProperty("type", "string"),
new JProperty("description", "[actual descriptions in OG code].")
)
),
new JProperty("param 4",
new JObject(
new JProperty("type", "string"),
new JProperty("description", "[actual descriptions in OG code].")
)
)
)
),
new JProperty("required", new JArray("param 1", "param 2", "param 3", "param 4"))
)
)
)
)
)
)
)
)
);
So, this means that something is wrong when serializing the request directly from my class definitions.
I can tell you right now what the issue is with a high probability, the problem is how exactly I’m supposed to work around it. This is where I need advice.
Let’s observe the class definitions defined below:
The Problem Children
public class Tool {
[JsonProperty(Order = 1)] public FunctionDecleration[] function_declerations; //these are the declerations of the functions that can be called
public Tool(FunctionDecleration[] functions) {
this.function_declerations = functions;
}
}
[Serializable]
public class FunctionDecleration {
[JsonProperty(Order = 1)] public string name;
[JsonProperty(Order = 2)] public string description;
[JsonProperty(Order = 3)] public FunctionParameters parameters;
public FunctionDecleration(string name, string desc, FunctionParameters args) {
this.name = name;
this.description = desc;
this.parameters = args;
}
}
[Serializable]
public class FunctionParameters {
[JsonProperty(Order = 1)] public string type; //defines the overall data type, ex. object
[JsonProperty(Order = 2)] public Dictionary<string, Property> properties; //this is a dictionary of the parameters and their data types
[JsonProperty(Order = 3)] public List<string> required; //the required parameter names mandatory for the function to operate
public FunctionParameters(string type, Dictionary<string, Property> props, List<string> req) {
this.type = type;
this.properties = props;
this.required = req;
}
}
[Serializable]
public class Property {
[JsonProperty(Order = 1)] public string type; //this represents the data type of the parameter
[JsonProperty(Order = 2)]public string description; //this describes the parameter's purpose and expected format
public Property(string type, string desc)
{ this.type = type; this.description = desc; }
}
[Serializable]
public class EnumProperty : Property {
[JsonProperty(Order = 3, PropertyName = "enum")] public List<string> values; //this is a list of possible values for the parameter, represented as an enum to the LLM
public EnumProperty(string type, string desc, List<string> vals) : base(type, desc)
{ this.type = type; this.description = desc; this.values = vals; }
}
[Serializable]
public class ObjectProperty : Property {
[JsonProperty(Order = 3)] public Dictionary<string, Property> properties; //this is a dictionary of the parameters and their data types
public ObjectProperty(string type, string desc, Dictionary<string, Property> props) : base(type, desc) { this.properties = props; }
}
[Serializable]
public class ArrayProperty : Property {
[JsonProperty(Order = 3)] public Property items; //this is the data type of the array's elements
public ArrayProperty(string type, string desc, Property items) : base(type, desc) { this.items = items; }
}
I tried to be sly and use object inheritance to get by and create different “types” of properties, so that I can make a dictionary of properties, but this completely mucks up the serialization process (I think). It’s very easy to end up accidentally making a recursive loop because of this.
That being said, I don’t know how to construct these classes in a way that can be serialized the way Gemini wants it to be without getting a Bad Request back. My intended structure works fine as we see from the testing JSON. It’s when I try to represent that with objects is where things go awry. I’m stuck, because in JSON formatting, properties can include other properties inside of itself, but in C#, you cannot make a definition of an object that refers to an object of itself without using inheritance. Then again, that may not even be the issue here, I don’t know. It might be due to arrays of tools[] and function declerations[] perhaps, it’s quite hard to tell.
I could try flattening the ArrayProperty, or work around this using JObjects in my object definitions instead of their base data structures. I don’t know which direction to go in.
. I might as well give this to the world so no one else has to suffer like I do with this lol. Better to have it out for good use than let it collect dust. I have plenty more code the judges can assess lol.