Research: Offline AI assistant architecture, Gemma 4 E2B-it, memory, GPS and performance

Google AI Edge
1

Hello Google AI Edge team,

I am researching the architecture of a privacy-first, offline-capable Android assistant called FAM.

As part of the research, I have been testing Google AI Edge Gallery with Gemma-4-E2B-it on an Android phone.

I would like to share several observations and ask for guidance on the recommended architecture.

  1. Offline operation

With Wi-Fi and mobile data disabled, Gemma-4-E2B-it was still able to process prompts locally.

This is important for our intended use case, because the assistant should remain functional when there is no Internet connection.

  1. Performance

The model works locally, but response generation is noticeably slow on the test device.

This raised a question about whether an LLM should be treated as an always-running core component of an Android assistant, or instead as an optional reasoning layer that is invoked only when necessary.

  1. Context and memory experiment

I provided the model with information about a person named Elisa:

  • Elisa has a strong personality.
  • She communicates very directly.
  • She prefers clear explanations without unnecessary wording.

The model was able to retain these facts within the conversation, distinguish them from its own inferences, and use them to generate contextual advice.

However, after leaving the conversation and attempting to ask about Elisa again, the model encountered:

“Status Code: 13”
“Failed to invoke the compiled model”

The application then displayed:

“Session re-initialized”

Therefore, I could not determine whether the information would have persisted across sessions.

This raises an architectural question:

Should persistent user memory be implemented outside the LLM, with the application retrieving relevant memories and injecting them into the model context when needed?

  1. GPS and device sensors

Another part of the research involves an offline safety scenario.

A phone can potentially determine its location using GNSS/GPS without Wi-Fi or mobile data, but the local LLM itself does not necessarily have access to those sensors.

What is the recommended Google AI Edge architecture for an Android application where:

  • the application accesses GPS/GNSS and other device sensors;
  • the LLM runs completely locally;
  • the application can provide selected sensor information to the LLM;
  • the LLM does not directly control critical device functions;
  • the system continues working if the LLM fails or is unavailable?
  1. Proposed architecture

Our current thinking is:

Android application
|
±- Core assistant / deterministic commands
|
±- Persistent local memory
|
±- GPS / sensors / device APIs
|
±- Local Gemma model
|
±- reasoning only when required

In this architecture, Gemma would not be the entire assistant. It would be a local reasoning component.

We would appreciate guidance on whether this is consistent with recommended AI Edge architecture, and which Google AI Edge / LiteRT components would be appropriate for implementing it.

We are particularly interested in:

  • persistent local memory / RAG;
  • session management;
  • model initialization and re-initialization;
  • performance optimization on Android;
  • integration with Android sensors;
  • tool/function calling;
  • keeping critical functions independent of the LLM;
  • and best practices for running Gemma locally without unnecessarily affecting device performance.

Thank you for any technical guidance or references you can provide.