Hello everyone,
I am currently developing a local AI orchestration system for a smart mapping application named “Nashmi” within Android Studio, utilizing the llama.cpp library and offline GGUF models.
I have successfully integrated a small, 0.5-billion-parameter model (approximately 322 MB) that runs entirely offline. However, I am encountering resource constraints and slight latency during both compilation and runtime within the Android environment; this is likely due to how the device handles large data arrays (tensors) in RAM.
I would appreciate insights on engineering best practices to ensure memory efficiency in this setup, specifically:
- What is the optimal build configuration (using CMake/NDK) to fully and smoothly leverage hardware capabilities?
- Should I strictly rely on
mmapconfiguration (i.e., settinguse_mmap = true) in the C++ code to prevent the Android system from terminating the background process via the “Low Memory Killer” (LMK) mechanism? - How can I pin specific versions of dependencies in the
build.gradle.ktsfile to avoid unexpected conflicts with third-party libraries (such as automatic JDK version resolution tools)?
I look forward to hearing your technical opinions and experiences in this area!
Best regards,
Sami