Overview
Contiki’s AI service is built on top of Gemini, clients setup a google account and using googles AI studio generate a API key linked to their billing.
We dont want to handle billing a client usage so a key per client linked directly to their card give them direct control and monitoring of usage.
Whats failing
More then half of the AI clients keys dont work when we use their key. the request to the model is hung and after a minute a 503 service unavailable is thrown
and absolutely no request get past in all hours of the day this lasts for weeks!
while the other half of our clients including our keys work fantastically at all times.
what we did
this anomaly is going on for months and we would let clients use our key for free instead, we looked at every possible factor in this issue and have spoken many times to forums and support, and we have not found anything.
notable things we tried:
- Priority interface (googles solution) https://ai.google.dev/gemini-api/docs/priority-inference
- Upgrading clients to paid tier 2 https://ai.google.dev/gemini-api/docs/billing#about-billing
- Googles support case & google developer forums (all of them)
The understanding
The google ai billing system was reworked on the 23.3.26, in short the billing changed from a direct pay for use, to the annoying prepay and postpay billing plans: https://ai.google.dev/gemini-api/docs/billing#billing-plans
when the billing change we did our tests and found no issues, later after the migration new clients started coming
and at first everything worked and we froze this issue, last few weeks the issue started happening and we where planing to leave gemini as its clearly not ready for production.
after a fresh look at the state of the service we saw that the only failing clients are clients that use prepay, and clients that we had before the migration stayed on postpay and their keys work, absolutely all of the prepay clients don’t work and at a deeper look we found out that they only don’t work with “Flash” models but “Flash lite” models work (our system uses “Flash”)
this issue is hurting the business and needs a solution.