Graph execution error at model.fit() function call

Hey, I’m trying to fine-tune VGG16 model for object detection. I’ve added a few dense layers and freezed the convolutional layers. There are 2 outputs of the model (bounding boxes and class labels) and the input is 512*512 images.

I have checked the model output shape and the training data’s ‘y’ shape.
The label and annotations have the shape: (6, 4) (6, 3)
The model outputs have the same shape:
<KerasTensor shape=(None, 6, 4), dtype=float32, sparse=False, name=keras_tensor_24>,
<KerasTensor shape=(None, 6, 3), dtype=float32, sparse=False, name=keras_tensor_30>

The error I see is (the file path is edited):

Traceback (most recent call last):
File “train.py”, line 163, in
history = model.fit(
File “\lib\site-packages\keras\src\utils\traceback_utils.py”, line 122, in error_handler
raise e.with_traceback(filtered_tb) from None
File “\lib\site-packages\tensorflow\python\eager\execute.py”, line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node ScatterNd defined at (most recent call last):
File “train.py”, line 163, in

File “\lib\site-packages\keras\src\utils\traceback_utils.py”, line 117, in error_handler

File “\lib\site-packages\keras\src\backend\tensorflow\trainer.py”, line 318, in fit

File “lib\site-packages\keras\src\backend\tensorflow\trainer.py”, line 121, in one_step_on_iterator

File “\lib\site-packages\keras\src\backend\tensorflow\trainer.py”, line 108, in one_step_on_data

File “\lib\site-packages\keras\src\backend\tensorflow\trainer.py”, line 77, in train_step

File “lib\site-packages\keras\src\trainers\trainer.py”, line 444, in compute_metrics

File “lib\site-packages\keras\src\trainers\compile_utils.py”, line 330, in update_state

File “lib\site-packages\keras\src\trainers\compile_utils.py”, line 17, in update_state

File “lib\site-packages\keras\src\metrics\iou_metrics.py”, line 129, in update_state

File “lib\site-packages\keras\src\metrics\metrics_utils.py”, line 682, in confusion_matrix

File “lib\site-packages\keras\src\ops\core.py”, line 237, in scatter

File “lib\site-packages\keras\src\backend\tensorflow\core.py”, line 354, in scatter

indices[0] = [286, 0] does not index into shape [3,3]
[[{{node ScatterNd}}]] [Op:__inference_one_step_on_iterator_4213]