Does tflite utilize unified memory architecture of SoC?

TensorFlow General Discussion
,
1

I am currently investigating the memory management of the tensorflow lite.

There are various SoC in mobile devices with unified memory, which is used to share the main memory to CPU, GPU, NPU, etc.

However, it seems that tflite does not utilize the shared memory architecture.

I think it is crucial because some operators fallback to the CPU when they are not supported by the target device, which will lead to excessive memory use.

Does tflite allocate different memory spaces for each device? Or, is there any management policy that exploit unified memory, and where can I find it?

3

Hi @widiba03304,
As of right now, TFLite does not make full use of unified memory across all operators and devices. Even if it might make use of particular characteristics based on the hardware and backend, a comprehensive use of the unified memory architecture is still under development. Generally, TFLite allots memory for tensors independently for every device (CPU, GPU, and NPU) that is utilized during inference. Under some circumstances, some backends, such as NNAPI on Android, may use shared memory for certain tasks. For the updates please check link1 and link2.

Thank You