We use API aggregators in our stack to handle rate limits and fallbacks, but recently we had an incident where a secondary relay started serving a distilled model instead of the flagship one. It broke our eval pipeline.
I put together a lightweight probe tool to catch this proactively. It connects to any OpenAI/Anthropic-compatible endpoint, runs a challenge that forces a signature from the model’s backend, and grades the response in ~60 seconds.
It’s deterministic, so you can run it on a schedule (though currently manual). Curious if anyone else has run into this issue, and what you’re doing to verify upstream integrity if you don’t have direct provider access.
Try at https//relayles.dev