Hello everyone. Currently i was working a pre-trained object detection model. The code was working well until this week. This week the same code occured an error which i cannot find the solution. Could you please help me?
This is the train log:
2025-02-06 07:55:12.837226: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2025-02-06 07:55:12.883135: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-06 07:55:12.883187: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-06 07:55:12.883223: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-06 07:55:12.891546: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-02-06 07:55:14.228627: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2025-02-06 07:55:17.085280: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2211] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at Install TensorFlow with pip for how to download and setup the required libraries for your platform.
Skipping registering GPU devices…
INFO:tensorflow:Using MirroredStrategy with devices (‘/job:localhost/replica:0/task:0/device:CPU:0’,)
I0206 07:55:17.087865 134964590924416 mirrored_strategy.py:423] Using MirroredStrategy with devices (‘/job:localhost/replica:0/task:0/device:CPU:0’,)
INFO:tensorflow:Maybe overwriting train_steps: 7000
I0206 07:55:18.221821 134964590924416 config_util.py:552] Maybe overwriting train_steps: 7000
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0206 07:55:18.222005 134964590924416 config_util.py:552] Maybe overwriting use_bfloat16: False
WARNING:tensorflow:From /usr/local/lib/python3.11/dist-packages/object_detection/model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
W0206 07:55:18.261016 134964590924416 deprecation.py:50] From /usr/local/lib/python3.11/dist-packages/object_detection/model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
INFO:tensorflow:Reading unweighted datasets: [‘/content/gdrive/MyDrive/Tensorflow/riceLeaf/data/train.record’]
I0206 07:55:18.577897 134964590924416 dataset_builder.py:162] Reading unweighted datasets: [‘/content/gdrive/MyDrive/Tensorflow/riceLeaf/data/train.record’]
INFO:tensorflow:Reading record datasets for input file: [‘/content/gdrive/MyDrive/Tensorflow/riceLeaf/data/train.record’]
I0206 07:55:18.578376 134964590924416 dataset_builder.py:79] Reading record datasets for input file: [‘/content/gdrive/MyDrive/Tensorflow/riceLeaf/data/train.record’]
INFO:tensorflow:Number of filenames to read: 1
I0206 07:55:18.578492 134964590924416 dataset_builder.py:80] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0206 07:55:18.578563 134964590924416 dataset_builder.py:86] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /usr/local/lib/python3.11/dist-packages/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.deterministic.
W0206 07:55:18.587682 134964590924416 deprecation.py:50] From /usr/local/lib/python3.11/dist-packages/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.deterministic.
WARNING:tensorflow:From /usr/local/lib/python3.11/dist-packages/object_detection/builders/dataset_builder.py:235: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.map() W0206 07:55:18.611178 134964590924416 deprecation.py:50] From /usr/local/lib/python3.11/dist-packages/object_detection/builders/dataset_builder.py:235: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map()
WARNING:tensorflow:From /usr/local/lib/python3.11/dist-packages/tensorflow/python/util/dispatch.py:1260: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead.
W0206 07:55:24.602539 134964590924416 deprecation.py:50] From /usr/local/lib/python3.11/dist-packages/tensorflow/python/util/dispatch.py:1260: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a tf.sparse.SparseTensor and use tf.sparse.to_dense instead.
WARNING:tensorflow:From /usr/local/lib/python3.11/dist-packages/tensorflow/python/util/dispatch.py:1260: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0206 07:55:29.357388 134964590924416 deprecation.py:50] From /usr/local/lib/python3.11/dist-packages/tensorflow/python/util/dispatch.py:1260: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2025-02-06 07:55:33.884864: W tensorflow/core/framework/dataset.cc:959] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
/usr/local/lib/python3.11/dist-packages/keras/src/backend.py:452: UserWarning: tf.keras.backend.set_learning_phase is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the training argument of the __call__ method of your layer or model.
warnings.warn(
INFO:tensorflow:depth of additional conv before box predictor: 0
I0206 07:55:49.720321 134956076471872 convolutional_keras_box_predictor.py:152] depth of additional conv before box predictor: 0
WARNING:tensorflow:From /usr/local/lib/python3.11/dist-packages/tensorflow/python/autograph/impl/api.py:460: Tensor.experimental_ref (from tensorflow.python.framework.tensor) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.
W0206 07:56:02.072342 134956076471872 deprecation.py:50] From /usr/local/lib/python3.11/dist-packages/tensorflow/python/autograph/impl/api.py:460: Tensor.experimental_ref (from tensorflow.python.framework.tensor) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.
WARNING:tensorflow:From /usr/local/lib/python3.11/dist-packages/tensorflow/python/util/dispatch.py:1260: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.
See tf.nn.softmax_cross_entropy_with_logits_v2.
W0206 07:56:07.811655 134956076471872 deprecation.py:50] From /usr/local/lib/python3.11/dist-packages/tensorflow/python/util/dispatch.py:1260: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.
See tf.nn.softmax_cross_entropy_with_logits_v2.
2025-02-06 07:56:47.174910: W tensorflow/core/framework/dataset.cc:959] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
^C