Hi,
When executing a model with my plugin, I noticed that TF calls device_memory_usage at the start but doesn’t call it anymore. If I choose a model big enough then I eventually run out of memory on my device, so it seems that tensorflow is not using that information.
I was expecting tensorflow to use that information to avoid running out of memory on the device but that doesn’t seem to be the case, isn’t it?
What am I missing in order to avoid getting out of memory?
Does anyone know why?
Up please, I keep getting my device out of memory and I would like to know why tensorflow is not making use of the device memory information
Up anyone?
This post is getting old
You can get the current memory usage using this API: tf.config.experimental.get_memory_info()['current']
Please turn on memory growth by calling tf.config.experimental.set_memory_growth, it allocates more memory as the process increases and demands extra memory. Thank you.
tf.config.experimental.set_memory_growth doesn’t solve the issue, the device still runs out of memory.
My question is whether tensorflow can/should avoid this by using the device_memory_usage of the stream executor implementation or if that’s not to be expected.