You must feed a value for placeholder tensor 'gradients/.../split_dim' with dtype int32

dimanne · March 27, 2023, 12:43am

Hi,

Does anyone know why the following code:

import tensorflow as tf
from tensorflow import keras

print("TensorFlow version:", tf.__version__)
print("Eager execution:", tf.executing_eagerly())

n_a = 64
n_values = 90
Tx = 30

reshaper = keras.layers.Reshape((1, n_values))
X = keras.layers.Input(shape=(Tx, n_values))
x = reshaper(X[:, 1, :])
a0 = keras.layers.Input(shape=(n_a,), name='a0')
c0 = keras.layers.Input(shape=(n_a,), name='c0')

LSTM_cell = keras.layers.LSTM(n_a, return_state = True)
LSTM_cell(inputs=x, initial_state=[a0, c0])

says: You must feed a value for placeholder tensor 'gradients/split_grad/concat/split/split_dim' with dtype int32?

Full output:

2023-03-26 18:18:23.694775: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-26 18:18:24.337919: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
TensorFlow version: 2.12.0
Eager execution: True
2023-03-26 18:18:25.187142: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.187375: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.204655: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.204848: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.205007: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.205156: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.374631: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.374832: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.374998: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.375147: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.375294: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:25.375440: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:27.198365: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:27.198556: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:27.198704: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:27.198839: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:27.198971: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:27.199106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2965 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3080, pci bus id: 0000:08:00.0, compute capability: 8.6
2023-03-26 18:18:27.199355: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-03-26 18:18:27.199504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 2101 MB memory:  -> device: 1, name: NVIDIA GeForce RTX 3060 Ti, pci bus id: 0000:09:00.0, compute capability: 8.6
2023-03-26 18:18:27.469631: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'gradients/split_2_grad/concat/split_2/split_dim' with dtype int32
	 [[{{node gradients/split_2_grad/concat/split_2/split_dim}}]]
2023-03-26 18:18:27.470367: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'gradients/split_grad/concat/split/split_dim' with dtype int32
	 [[{{node gradients/split_grad/concat/split/split_dim}}]]
2023-03-26 18:18:27.471034: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'gradients/split_1_grad/concat/split_1/split_dim' with dtype int32
	 [[{{node gradients/split_1_grad/concat/split_1/split_dim}}]]

Process finished with exit code 0

nvidia-smi:

NVIDIA-SMI 525.85.05    Driver Version: 525.85.05    CUDA Version: 12.0

CUDA 11.8 installed:

apt list --installed | rg cuda                                                                                                            654ms  Sun Mar 26 17:43:51 2023

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

cuda-cccl-11-8/unknown,now 11.8.89-1 amd64 [installed]
cuda-command-line-tools-11-8/unknown,now 11.8.0-1 amd64 [installed]
cuda-compat-11-8/unknown,now 520.61.05-1 amd64 [installed]
cuda-compiler-11-8/unknown,now 11.8.0-1 amd64 [installed]
cuda-cudart-11-8/unknown,now 11.8.89-1 amd64 [installed]
cuda-cudart-dev-11-8/unknown,now 11.8.89-1 amd64 [installed]
cuda-cuobjdump-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-cupti-11-8/unknown,now 11.8.87-1 amd64 [installed]
cuda-cupti-dev-11-8/unknown,now 11.8.87-1 amd64 [installed]
cuda-cuxxfilt-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-driver-dev-11-8/unknown,now 11.8.89-1 amd64 [installed]
cuda-gdb-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-libraries-11-8/unknown,now 11.8.0-1 amd64 [installed]
cuda-libraries-dev-11-8/unknown,now 11.8.0-1 amd64 [installed]
cuda-memcheck-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-minimal-build-11-8/unknown,now 11.8.0-1 amd64 [installed]
cuda-nvcc-11-8/unknown,now 11.8.89-1 amd64 [installed]
cuda-nvdisasm-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-nvml-dev-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-nvprof-11-8/unknown,now 11.8.87-1 amd64 [installed]
cuda-nvprune-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-nvrtc-11-8/unknown,now 11.8.89-1 amd64 [installed]
cuda-nvrtc-dev-11-8/unknown,now 11.8.89-1 amd64 [installed]
cuda-nvtx-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-profiler-api-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-sanitizer-11-8/unknown,now 11.8.86-1 amd64 [installed]
cuda-toolkit-11-8-config-common/unknown,now 11.8.89-1 all [installed]
cuda-toolkit-11-config-common/unknown,now 11.8.89-1 all [installed]
cuda-toolkit-config-common/unknown,now 12.1.55-1 all [installed]
libcudnn8-dev/unknown,now 8.7.0.84-1+cuda11.8 amd64 [installed,upgradable to: 8.8.1.3-1+cuda12.0]
libcudnn8/unknown,now 8.7.0.84-1+cuda11.8 amd64 [installed,upgradable to: 8.8.1.3-1+cuda12.0]
libnccl-dev/unknown,now 2.15.5-1+cuda11.8 amd64 [installed,upgradable to: 2.17.1-1+cuda12.1]
libnccl2/unknown,now 2.15.5-1+cuda11.8 amd64 [installed,upgradable to: 2.17.1-1+cuda12.1]

chunduriv · March 27, 2023, 4:26am

@dimanne,

Welcome to the Tensorflow Forum!

I attempted to execute code using the latest version of TF 2.12, and it is working as expected. Please refer to the gist.

Could you please tell us what is the version of Tensorflow installed? also share more details regarding OS?

Thank you!

Andrej_Kubanda · March 31, 2023, 10:09am

Hello @dimanne,
have you managed to fix this issue or find out what the problem is?
All I know is that it happens when I use LSTM layers.
For me, the problem does not occur in collab. However, it happens on my machine and also on the team’s server (Python3.10 and TF2.12).

Juan_J_Sanchez · April 4, 2023, 11:26am

Hi, @Andrej_Kubanda.

Same situation here with TF 2.12.0 and Python 3.10.6.
Funny thing is that those warnings appear after installing “tensorflow_hub” and “tensorflow_text”. Uninstalling everything and reinstalling didn’t solve the problem in my case.

Please comment if you manage to solve this. It’s not a stopper but kind of annoying.

Juan_J_Sanchez · April 5, 2023, 9:44am

Hi, @dimanne and @Andrej_Kubanda,

I managed to get rid of those warning messages by downgrading to TF 2.11.
You can do that with pip install install tensorflow==2.11.

TF 2.12 was released two weeks ago Releases · tensorflow/tensorflow · GitHub
and, in my case, it seems I made the upgrade while installing other libraries.
Hope this helps.

Juan

Hugo_Ferreira · April 6, 2023, 10:36am

Would just like to point out that this is not directly related to Keras. I am using TF’s level level io module. If I do:

    list_ds = tf.data.Dataset.from_tensor_slices( list(files) )
    image_count = len(list_ds)

    list_ds = list_ds.shuffle(image_count, reshuffle_each_iteration=False)
    list(list_ds)

then the last line is enough to trigger the warning.

HTHs

UpBoi · May 23, 2023, 5:04am

Continuing the discussion from You must feed a value for placeholder tensor 'gradients/…/split_dim' with dtype int32:

Yes, I have the same problem while training a simple denoisy encoder using TF 2.12, I found my result is not repeatable despite of adding:

tf.random.set_seed(seed)
np.random.seed(seed)
# tf.keras.utils.set_random_seed(seed)
random.seed(seed)

but when I add

os.environ[‘TF_DETERMINISTIC_OPS’] = ‘1’,

the results are repeatable, but a new warning jumps out of nowhere, it appears triggerd in “model.fit” process, and it looks like an internal bug not fixed yet
“2023-05-23 12:38:39.234478: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor ‘Placeholder/_10’ with dtype uint8 and shape [60000,784]”

Funny thing is that this is TF 2.x, so there should be no “placeholder” anywhere since the official page says this rountine is no longer supported and certainly not defined in my code. Second is that the input shape of dtype is the same as required, so it definitely has something to do with underlying code. For now, I do not have solution, what I will try next is to downgrade like above answer suggested, I will get back with update results later.

UpBoi · May 24, 2023, 6:11am

My warning:

2023-05-24 13:46:21.798953: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor ‘Placeholder/_10’ with dtype uint8 and shape [60000,784]
[[{{node Placeholder/_10}}]]
2023-05-24 13:46:21.799142: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor ‘Placeholder/_10’ with dtype uint8 and shape [60000,784]

I have tried downgrade the TF from tensorflow==2.12 to tensorflow==2.11, and yes I can confirm that this warning no longer exists, and the results are the same with tensorflow==2.12 but without the warning.

I would suggest create a new virtual envrionment mimic the current envrionment except using different version of TF.

Plus, I also tried to downgrade tenssort in TF = 2.12 to see whether is it a problem of “tenssort”, it results the same warning though. My warning was triggerd by "os.environ[‘TF_DETERMINISTIC_OPS’] = ‘1’ to make the code repeatable, so it might be a good start point to look at what happened out there (remove this the warning is gone but the code is not repeatable).

Besides of this walkaround, I would hope this could be fixed in following TF version.