To the Product and Research Leadership Teams,
This letter outlines a critical architectural and policy failure in current-generation LLMs that creates a profoundly inefficient, costly, and frustrating experience for expert users. More importantly, it proposes a direct, technically feasible solution: an opt-in, per-API key data retention policy for model training and fine-tuning.
We are a community of expert users-software architects, systems programmers, and researchers-who use your API services for sustained, complex, and boundary-pushing technical work. We are your ideal customers, pushing your models to their limits in domains like formal methods, compiler design, and systems programming with languages such as Rust, Zig, OCaml, Lean, F*, Koka, and Idris.
After thousands of hours and tens of thousands of interactions, we have documented a universal and systematic pattern of failure that renders your products almost unusable for our work: the Amnesiac Expert Paradox.
The Core problem: A system designed for ignorance (with real-world evidence)
The current API policy of no data retention creates a vicious and costly cycle, perfectly exemplified by an active project collapsing under this system’s weight:
- Current project state: A user is developing a complex system in the Koka programming language.
- Total context consumption: 135,000 tokens.
- Meaningful project content: 10,000 tokens (a mere 7.4%).
- Remedial education overhead: 125,000 tokens (92.6%) of the context is dedicated to manually ingesting manually curated and foundational Koka documentation to “teach” the model information that has been publicly available for over five years.
- Result: The project is paralyzed, rapidly approaching the 200k token context limit, with the user facing a choice between project abandonment or incurring massive, unsustainable costs.
This is not an isolated incident. This pattern of failure is the direct result of a broken feedback loop:
-
Systematic knowledge gaps:
Your models consistently demonstrate a weak, outdated, or dangerously incorrect understanding of even well-established, publicly documented technical domains. Information that has been available for years is often missing from the base models. -
The user as unpaid data curator:
We, the expert users, are forced to compensate for these knowledge gaps. A staggering 80-90% of our context window is often consumed by us manually ingesting foundational data-official documentation, source code, and curated knowledge bases-just to bring the model to a baseline level of competence, rather than productive work. -
The high cost of remedial education:
We are paying premium API rates not for intelligent assistance, but for the privilege of performing this remedial education. At current pricing tiers ($15-$75, $3-15, and $1.25-$10 per million tokens), a single long development session can cost hundreds of dollars, with the vast majority of that cost attributed to re-teaching the model information it should already know. -
The failure of in-context “learning”:
Our extensive testing shows that model performance on in-context data is drastically inferior to its performance on pre-trained knowledge. The context window is a flawed and inefficient mechanism for knowledge transfer, leading to unreliable and often incorrect outputs even when provided with perfect, curated source material. -
The Evaporation of Expertise: The most tragic part of this cycle is that all of this hard work-the corrections, the curated data, the refined explanations-is discarded at the end of every session. Our expertise, which could be an invaluable source of high-quality training data, evaporates into the void. The model is doomed to remain ignorant, and the next user (or we ourselves in the next session) will encounter the exact same failures.
This is a fundamentally broken feedback loop. It is economically unsustainable for us and represents a colossal waste of high-quality training data for you.
The Solution: An opt-in, per-API key toggle for persistent learning
We propose a simple, powerful, and immediately actionable solution that respects user privacy while fixing this broken loop.
Implement an account-level, per-API key setting that allows users to opt-in to data retention for the purpose of model improvement.
How it works:
- A user enables a “Contribute to Training” toggle for a specific API key in their provider dashboard.
- All interactions using that API key are then eligible for inclusion in your fine-tuning and training data pipelines.
- This is an explicit, opt-in value exchange: we provide expert-curated data, and in return, we get a model that actually learns and improves over time.
Why this is a superior approach:
-
It solves the privacy dilemma:
The opt-in nature places the user in full control. Users working with sensitive data can keep the default no-retention policy. Users working with public data, like ourselves, can choose to contribute. -
It works for all users:
Tying this setting to the API key, not the request parameters, makes it universally accessible. It works for users interacting via third-party applications, code editors, and custom tools where they cannot control the HTTP headers. -
It unlocks premium training data:
You would gain access to a continuous stream of high-quality, expert-verified, domain-specific training data-far more valuable than unfiltered web scrapes. The corrections we provide on Rust libraries, Zig syntax, and Koka semantics, and many more are premium training signals you are currently discarding. -
It creates a virtuous cycle:
As the model improves, our interactions become more efficient, our API usage shifts from frustrating remedial work to valuable collaboration, and we can tackle even more ambitious projects with your services. -
It is technically feasible:
You already have the infrastructure to retain and learn from user interactions via your web interfaces. The xAI precedent proves this is also legally and commercially viable for APIs. This is a policy change, not an insurmountable technical hurdle.
The competitive imperative
The first provider to implement this feature will capture the loyalty and business of the entire community of expert users who are currently being paralyzed by the existing system. This is not just a feature request; it is a clear market differentiator in an increasingly commoditized space. Prompt caching is not a solution; it addresses a different, much simpler problem. We need persistent learning, not static retrieval.
We are not just asking for a new feature. We are proposing a more intelligent, more efficient, and more honest partnership between you and the expert community that is pushing your technology to its absolute limits.
We urge you to consider implementing this opt-in, per-API key data retention policy. Let us help you fix the broken machine.
Sincerely,
Andoni Hazaiah,
Expert LLM/API Users