I am getting the following error when training a custom model:
W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
Has anyone seen this error before? My code is a bit of a mess right now so hard to share any specifics. So I am not looking for a concrete solution, just some pointers on what could cause this so I can debug further. Right now I don’t even know what could possibly cause this error/warning.
Generally I suggest you to use tags on new thread cause specialized technical team members could be subscribed only on tag subset (e.g. in this case tfdata and help_request tags).
I am getting the same warning. For me, this warning occurs frequently, but not on every training step.
Using the code below, I fit the model and print the shape of input data and target data.
print(states.shape, targets_full.shape)
self.model.fit(states, targets_full, epochs=1, verbose=1)
The output shows that the shapes are always the same, yet sometimes I get the warning and sometimes I do not. A random snippet of the output is pasted below.
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2445
2021-06-15 13:02:19.379526: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1214
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1480
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2558
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1036
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1048
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2647
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1280
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1074
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2401
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1160
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1348
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2424
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1197
2021-06-15 13:02:20.594590: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1415
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2358
2021-06-15 13:02:20.790803: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1212
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1649
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2220
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1064
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1488
2021-06-15 13:02:21.299211: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
Any help is understanding or resolving this warning would be appreciated.
Hi , I am also facing the same issue. Any solution ?
What TF version are you using?
Have you checked:
I was using TF version 2.5 and have just upgraded to 2.6 where I am still getting the same issue. I am also using Python 3.6.
After I upgraded to TF 2.6, I am still getting the error, but the path is different.
In 2.6:
“W tensorflow/core/data/root_dataset.cc:167] Optimization loop failed: Cancelled: Operation was cancelled”
Whereas in 2.5:
“W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled”
I looked through that tread, but I wasn’t able to solve the problem. The error is that tread also has a different path:
“W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled”
Any help would be appreciated.
I was able to fix this, the reason why it was throwing an error was my batch_size was set greater than the size of the whole dataset.
This might be the case.
I encountered the same problem. As stated by @Farquhar13 , this warning information doesn’t always show up so this means the size of the batch should be fine? Please correct me if I am wrong.
Thanks!
I’m getting this same error.
2021-12-13 17:59:55.175834: W tensorflow/core/data/root_dataset.cc:163] Optimization loop failed: CANCELLED: Operation was cancelled
I know that my problem isn’t the batch size being larger than my dataset. I checked the size of the dataset each time I got the error, and it’s the same size as when I don’t get the error. Speaking of which, I don’t get this error every time I run the code. It seems almost random, but it obviously can’t be.
I can’t find a solution to this anywhere, so I’m hoping someone here can help me out.
I am running into a similar issue. I’m running TS v2.7 and python 3.7.4
I’ve checked the shapes passing and the batch size as referred to above and those are not the issues. However I get it on every iteration.
Any help would be appreciated!
I have the same problem as you, how did you solve it, I hope you can help me, thank you
TF 2.8 and python 3.8
I have the same issue when serving the model online, the error seems to show up for every model.predict() call.
My batch size and input sample is always 1, I dont think batch_size is the problem here.
Also I could not reproduce this warning in my local.
I am facing the same problem. My batch size and input sample is always 1, I don’t think batch size is the problem here.
I’m having similar problem, with this Error:
“W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled”
My batch is not larger than the whole dataset
The exact code are taken from a YouTube tutorial:
you can get the code to reproduce the error from here (Although in the Video tutorial he didn’t get that error):
So what is the problem is it because i’m using a newer version of Tensorflow ?
Note:
i’m using Python 3.9 | Tensorflow 2.10.0 | Tensorflow-gpu 2.10.0 | keras 2.10.0 | Window 10
Hello I’m getting the same error with TF 2.7 (Python 3.9) when making predictions with the trained model.
My data is grouped in cycles and interestingly I only get the error with some cycles in a - what seems to be random - pattern. Batchsize is always 1 in my usecase:
INFO Processed cycle: 3/20
INFO Processed cycle: 4/20
2022-10-04 11:05:41.750979: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:41.869399: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:41.892579: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.490775: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.540553: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.587145: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.610908: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.636638: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.660311: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:46.755783: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.348900: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.466292: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.487418: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.510366: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:50.600544: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:56.583640: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:56.611070: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:56.689325: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:05:56.715945: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.150230: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.391989: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.466320: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.532687: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.594002: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.636137: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.660478: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.785125: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:06.836864: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:13.186727: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:13.292232: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:13.497294: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:31.738377: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:34.954275: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:06:38.405368: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:07:01.726403: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:07:01.776447: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
INFO Processed cycle: 5/20
2022-10-04 11:09:01.265304: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
INFO Processed cycle: 6/20
2022-10-04 11:10:16.427091: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
2022-10-04 11:11:04.562144: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
INFO Processed cycle: 10/20
Edit: I restarted the script and got this result:
INFO Processed cycle: 3/20
INFO Processed cycle: 4/20
INFO Processed cycle: 5/20
INFO Processed cycle: 6/20
INFO Processed cycle: 10/20
2022-10-04 11:32:09.333830: W tensorflow/core/data/root_dataset.cc:266] Optimization loop failed: CANCELLED: Operation was cancelled
Also updating to TF 2.9 or 2.10 increases the amount of warnings.
I have the same issue, solved by adding
gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpu, True)
Have you tried
model.predict(input_data, training=False)
This solution worked for me!
Tried it and I have the same problem. I’m solving the Lunar Lander problem from OpenAI’s GYM using a DQN approach in Kaggle. The problem appears both with GPU (P100) and without it
Added your lines and had no more problem since. Thank you!