Feature Request: A Standardized Capability and Limitation Card for AI Models
As a local AI user, I often download models based on their size, name, and general description, but I have no clear way to know which tasks they actually perform well.
For example, a model may support dozens of languages but hallucinate basic facts about well-known movies. Users have to discover these weaknesses through trial and error.
I propose that every model release include a simple, standardized capability card showing:
- Supported languages and tested proficiency in each language.
- Performance by domain, such as accounting, programming, history, translation, and general knowledge.
- Factual accuracy and hallucination rates on clearly defined tests.
- Known limitations and tasks for which the model has not been evaluated.
- Model size, parameter count, quantization, and hardware requirements.
- Links to reproducible evaluation results, not just marketing claims.
The goal is to help users choose the right model for their needs before downloading it, especially users running small models locally on limited hardware.
A model does not need to be good at everything. Users simply need an honest, standardized description of what it can and cannot do.