This is my first post. Please let me know if you have any issues.
We have several systems with two GPUs. On one of the systems, the behavior of tf.Variables is different.
The values set in tf.Variables are not shared between the two GPUs. tf.Tensor values are shared.
Any causes, solutions, or tips?
Are there any BIOS settings or other things I should check?
The code I ran is as follows
import tensorflow as tf
#tf.debugging.set_log_device_placement(True)
my_tensor = tf.constant([[1.0, 2.0], [3.0, 4.0]])
my_variable = tf.Variable(my_tensor)
print("on any my_variable: ", my_variable)
print("on any my_tensor: ", my_tensor)
with tf.device('/gpu:1'):
print("on GPU1_my_variable: ", my_variable)
print("on GPU1_my_tensor: ", my_tensor)
with tf.device('/gpu:0'):
print("on GPU0_my_variable: ", my_variable)
print("on GPU0_my_tensor: ", my_tensor)
Expected result
on any my_variable: <tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=
array([[1., 2.],
[3., 4.]], dtype=float32)>
on any my_tensor: tf.Tensor(
[[1. 2.]
[3. 4.]], shape=(2, 2), dtype=float32)
on GPU1_my_variable: <tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=
array([[1., 2.],
[3., 4.]], dtype=float32)>
on GPU1_my_tensor: tf.Tensor(
[[1. 2.]
[3. 4.]], shape=(2, 2), dtype=float32)
on GPU0_my_variable: <tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=
array([[1., 2.],
[3., 4.]], dtype=float32)>
on GPU0_my_tensor: tf.Tensor(
[[1. 2.]
[3. 4.]], shape=(2, 2), dtype=float32)
Actual result
on any my_variable: <tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=
array([[1., 2.],
[3., 4.]], dtype=float32)>
on any my_tensor: tf.Tensor(
[[1. 2.]
[3. 4.]], shape=(2, 2), dtype=float32)
on GPU1_my_variable: <tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=
array([[0., 0.],
[0., 0.]], dtype=float32)>
on GPU1_my_tensor: tf.Tensor(
[[1. 2.]
[3. 4.]], shape=(2, 2), dtype=float32)
on GPU0_my_variable: <tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=
array([[1., 2.],
[3., 4.]], dtype=float32)>
on GPU0_my_tensor: tf.Tensor(
[[1. 2.]
[3. 4.]], shape=(2, 2), dtype=float32)
Execution environment:.
Singularity container (NGC:tensorflow_22.03-tf2-py3.sif)
Best regards