How do you manage the workflow of your AI agent?

Gemini API
,
1

I find it hard to work with the workflow of when programming for the Live API. Here is an example

I have a site where users can do an exam with the AI. The workflow is something like this:

  1. Start exam, greet the user
  2. Run the exam asking questions
  3. After the time is up i tell it (with .sendRealtimeInput text) that the exam is up.
    1. Now i wait for the AI to finish what is was saying when i told it that the exam is up.
    2. Then it hopefully starts saying that the exam is finish (but you never know though)
    3. Then i have to wait for that sentence to finish before i put the user outside the door
  4. Now the AI has to give a grade and feedback via tool call. I then have to wait for that to respond (but maybe the tool never gets called)
  5. Then finally i can show the grade and feedback if i get this far in the process

It makes for a brittle and slightly unstable workflow. Most of the times it works just fine. Other times it fails quite bad

How do you manage these kind of workflows?