NeonLib: On-device Android document retrieval with EmbeddingGemma and LiteRT
Hi Gemma community!
I'd like to share NeonLib, an Android knowledge library I've been building with EmbeddingGemma 300M and LiteRT for on-device document retrieval.
**Try NeonLib on Google Play:**
https://play.google.com/store/apps/details?id=com.neonbreak.neonlib&hl=en
The idea is to bring notes, documents and recognised text into one library, ask questions, and inspect relevant source passages.
### What NeonLib does
- Imports notes, text files, Markdown, HTML, CSV and text-based PDFs.
- Recognises text from photos and scanned PDFs using ML Kit OCR.
- Uses EmbeddingGemma to retrieve relevant passages locally.
- Lets users select which collections to search and open the original passages.
- Supports optional answers from downloaded local language models or selected online AI providers.
- Supports private collection sharing, including compatible prepared embeddings.
EmbeddingGemma handles retrieval; a separate language model generates optional answers. Local retrieval does not require a cloud embedding API. Downloads, online answers and other connected features require internet access.
### Technical implementation
- Kotlin and Jetpack Compose
- EmbeddingGemma 300M mixed-precision TFLite export
- LiteRT CPU inference with a 1024-token input
- 768-dimensional ObjectBox HNSW index using cosine distance
- Retrieval-specific query and document prefixes
- Token-aware chunking targeting 450 tokens, with up to 60 tokens of overlap
- Serialized embedding inference, prioritising waiting queries over document indexing
Collection packages can include prepared vectors. Before reusing them, the app checks model and tokenizer hashes, preprocessing settings, and exact chunk text. Incompatible vectors fall back to local indexing; query embeddings are computed on the phone.
### What I'm evaluating
Croatian and English are my starting points. Initial functional checks are encouraging, but I am still working towards a broader labelled retrieval evaluation and reproducible Android latency and memory measurements.
One interesting issue is cross-runtime consistency: PC-generated and Android-generated vectors can differ slightly even with matching model assets. A small retrieval pilot passed, although a stricter numerical comparison did not meet its chosen tolerance.
I'd welcome feedback from the Gemma and Google AI Edge communities:
1. What checks would you recommend for validating this embedding export and its preprocessing?
2. How would you evaluate retrieval consistency between prepared desktop vectors and on-device query vectors?
3. Which measurements would make this a useful mobile, multilingual retrieval case study?
Happy to share more implementation details and discuss the tradeoffs.
Thanks for building EmbeddingGemma and making on-device retrieval accessible!