I am working with machine learning project using TensorFlow and Keras to process audio data. The project is designed to handle inputs with multiple channels, but there’s an issue during the training phase where the input shape does not match the expected model input shape.
In the github repo you can find model.py, I think that is supposed to reduce the channel dimension of audio data from 3 to 1 using a TimeDistributed layer wrapped around a Conv2D layer. However, during training, I’m encountering a shape mismatch error:
TypeError: generator yielded an element of shape (32, 2, 15, 80, 3) where an element of shape (None, 2, 15, 80, 1) was expected.
The github repo is here GitHub - cpuguy96/StepCOVNet: Deep Learning to Create StepMania SM FIles
I used that repo to prep the training data from the raw data and went to create the model which gives the mismatch.
The prints before seem to suggest some shapes some of the time are correct?
I’m just not sure where to go from here. My goal is to create the model and use it for inferencing a novel song into stepmania step data.
I’m not sure if the repo is just in a bad unworking state or if its user error on my part.
I tend to learn by doing and thought this would be a good project to start in.
Exact command and output.
(StepNet) nonlin@Nonlin:/mnt/c/Users/gerfe/StepCOVNet$ python train.py -i /home/nonlin/miniconda3/envs/StepNet/TrainingData -o /home/nonlin/miniconda3/envs/StepNet/Model -d 1 --name "MyModel"
2024-05-11 10:00:27.499059: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-05-11 10:00:27.499132: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-05-11 10:00:27.516496: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-05-11 10:00:27.564170: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-11 10:00:28.449546: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
2024-05-11 10:00:30.166859: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:0b:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-05-11 10:00:30.508089: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2256] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
WARNING:root:Failed to set memory growth for GPU.
Traceback (most recent call last):
File "/mnt/c/Users/gerfe/StepCOVNet/stepcovnet/tf_config.py", line 10, in <module>
tf.config.list_physical_devices("GPU")[0], enable=True
IndexError: list index out of range
WARNING:tensorflow:Mixed precision compatibility check (mixed_float16): WARNING
The dtype policy mixed_float16 may run slowly because this machine does not have a GPU. Only Nvidia GPUs with compute capability of at least 7.0 run quickly with mixed_float16.
If you will use compatible GPU(s) not attached to this host, e.g. by running a multi-worker model, you can ignore this warning. This message will only be logged once
WARNING:tensorflow:Mixed precision compatibility check (mixed_float16): WARNING
The dtype policy mixed_float16 may run slowly because this machine does not have a GPU. Only Nvidia GPUs with compute capability of at least 7.0 run quickly with mixed_float16.
If you will use compatible GPU(s) not attached to this host, e.g. by running a multi-worker model, you can ignore this warning. This message will only be logged once
/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
2024-05-11 10:02:19.167084: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 154389504 exceeds 10% of free system memory.
2024-05-11 10:02:19.202167: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 154389504 exceeds 10% of free system memory.
2024-05-11 10:02:19.232825: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 154389504 exceeds 10% of free system memory.
2024-05-11 10:02:20.980241: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 154389504 exceeds 10% of free system memory.
All PyTorch model weights were used when initializing TFGPT2Model.
All the weights of TFGPT2Model were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFGPT2Model for predictions without further training.
Model: "StepCOVNet"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
audio_input (InputLayer) [(None, 2, 15, 80, 1)] 0 []
time_distributed (TimeDist (None, 2, 15, 80, 1) 2 ['audio_input[0][0]']
ributed)
arrow_input (InputLayer) [(None, None)] 0 []
arrow_mask (InputLayer) [(None, None)] 0 []
batch_normalization (Batch (None, 2, 15, 80, 1) 4 ['time_distributed[0][0]']
Normalization)
tfgpt2_model (TFGPT2Model) TFBaseModelOutputWithPastA 1244398 ['arrow_input[0][0]',
ndCrossAttentions(last_hid 08 'arrow_mask[0][0]']
den_state=(None, None, 768
),
past_key_values=((2, None
, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64),
(2, None, 12, None, 64)),
hidden_states=None, atten
tions=None, cross_attentio
ns=None)
VGGish (Functional) (None, 2, 512) 4499712 ['batch_normalization[0][0]']
global_max_pooling1d (Glob (None, 768) 0 ['tfgpt2_model[0][0]']
alMaxPooling1D)
bidirectional (Bidirection (None, 256) 656384 ['VGGish[0][0]']
al)
concatenate (Concatenate) (None, 1024) 0 ['global_max_pooling1d[0][0]',
'bidirectional[0][0]']
dense (Dense) (None, 256) 262400 ['concatenate[0][0]']
batch_normalization_1 (Bat (None, 256) 1024 ['dense[0][0]']
chNormalization)
activation (Activation) (None, 256) 0 ['batch_normalization_1[0][0]'
]
dropout_37 (Dropout) (None, 256) 0 ['activation[0][0]']
onehot_encoded_arrows (Den (None, 256) 65792 ['dropout_37[0][0]']
se)
==================================================================================================
Total params: 129925126 (495.63 MB)
Trainable params: 40370436 (154.00 MB)
Non-trainable params: 89554690 (341.62 MB)
__________________________________________________________________________________________________
Saving model metadata at /home/nonlin/miniconda3/envs/StepNet/Model/MyModel
Training on 1975957 samples (156 songs) and validating on 231660 samples (18 songs)
Starting training...
Epoch 1/15
2024-05-11 10:02:28.230065: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 154389504 exceeds 10% of free system memory.
2024-05-11 10:02:36.035559: W tensorflow/core/framework/op_kernel.cc:1827] INVALID_ARGUMENT: TypeError: `generator` yielded an element of shape (32, 2, 15, 80, 3) where an element of shape (None, 2, 15, 80, 1) was expected.
Traceback (most recent call last):
File "/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/tensorflow/python/ops/script_ops.py", line 270, in __call__
ret = func(*args)
File "/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 643, in wrapper
return func(*args, **kwargs)
File "/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/tensorflow/python/data/ops/from_generator_op.py", line 235, in generator_py_func
raise TypeError(
TypeError: `generator` yielded an element of shape (32, 2, 15, 80, 3) where an element of shape (None, 2, 15, 80, 1) was expected.
Traceback (most recent call last):
File "/mnt/c/Users/gerfe/StepCOVNet/train.py", line 166, in <module>
train(
File "/mnt/c/Users/gerfe/StepCOVNet/train.py", line 115, in train
run_training(
File "/mnt/c/Users/gerfe/StepCOVNet/train.py", line 61, in run_training
executor.TrainingExecutor(stepcovnet_model=stepcovnet_model).execute(
File "/mnt/c/Users/gerfe/StepCOVNet/stepcovnet/executor.py", line 111, in execute
history = self.train(input_data, self.get_training_callbacks(hyperparameters))
File "/mnt/c/Users/gerfe/StepCOVNet/stepcovnet/executor.py", line 191, in train
history = self.stepcovnet_model.model.fit(
File "/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:
Detected at node PyFunc defined at (most recent call last):
<stack traces unavailable>
TypeError: `generator` yielded an element of shape (32, 2, 15, 80, 3) where an element of shape (None, 2, 15, 80, 1) was expected.
Traceback (most recent call last):
File "/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/tensorflow/python/ops/script_ops.py", line 270, in __call__
ret = func(*args)
File "/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 643, in wrapper
return func(*args, **kwargs)
File "/home/nonlin/miniconda3/envs/StepNet/lib/python3.10/site-packages/tensorflow/python/data/ops/from_generator_op.py", line 235, in generator_py_func
raise TypeError(
TypeError: `generator` yielded an element of shape (32, 2, 15, 80, 3) where an element of shape (None, 2, 15, 80, 1) was expected.
[[{{node PyFunc}}]]
[[IteratorGetNext]] [Op:__inference_train_function_33834]