2024-12-01 11:27:55.010584: F tensorflow/stream_executor/cuda/cuda_driver.cc:147] Failed setting context: CUDA_ERROR_NOT_PERMITTED: operation not permitted

Excuse me, why is there this error here

D:\Anaconda3\envs\tf_st\python.exe D:\Study\tfst\QuantizationAwareDeepOptics-main\main.py --task=hyperspectral --doe_material=SK1300 --quantization_level=2 --scene_depth_m=1 --tag=000A4LevelHyperspctralTraining --sensor_distance_mm=50 
2024-12-01 11:27:51.554149: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

 
 GPU(s) to be used: 

[LogicalDevice(name='/device:GPU:0', device_type='GPU')]
Task:  tasks.hyperspectral
Start TASK tasks.hyperspectral
Extra Arguments:  {'doe_material': 'SK1300', 'quantization_level': 2, 'quantize_at_test_only': False, 'alpha_blending': False, 'adaptive_quantization': False, 'checkpoint': None, 'continue_training': False, 'tag': '000A4LevelHyperspctralTraining', 'sensor_distance_mm': 50, 'scene_depth_m': 1}
2024-12-01 11:27:51.985576: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0
2024-12-01 11:27:51.985689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5449 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4060 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.9
2024-12-01 11:27:52 [INFO] Wavelength list used: [4.0e-07 4.1e-07 4.2e-07 4.3e-07 4.4e-07 4.5e-07 4.6e-07 4.7e-07 4.8e-07
 4.9e-07 5.0e-07 5.1e-07 5.2e-07 5.3e-07 5.4e-07 5.5e-07 5.6e-07 5.7e-07
 5.8e-07 5.9e-07 6.0e-07 6.1e-07 6.2e-07 6.3e-07 6.4e-07 6.5e-07 6.6e-07
 6.7e-07 6.8e-07 6.9e-07 7.0e-07]
Network Args:
activation= elu
final_activation= sigmoid
kernel_initializer= he_uniform
final_kernel_initializer= glorot_uniform
2024-12-01 11:27:52 [INFO] 

==============>DOE Args<===============
  > General:
 {'wave_length_list': array([4.0e-07, 4.1e-07, 4.2e-07, 4.3e-07, 4.4e-07, 4.5e-07, 4.6e-07,
       4.7e-07, 4.8e-07, 4.9e-07, 5.0e-07, 5.1e-07, 5.2e-07, 5.3e-07,
       5.4e-07, 5.5e-07, 5.6e-07, 5.7e-07, 5.8e-07, 5.9e-07, 6.0e-07,
       6.1e-07, 6.2e-07, 6.3e-07, 6.4e-07, 6.5e-07, 6.6e-07, 6.7e-07,
       6.8e-07, 6.9e-07, 7.0e-07]), 'wavelength_to_refractive_index_func': <function refractive_index_glass_ohara_sk1300 at 0x00000297000A6670>, 'height_map_initializer': None, 'height_tolerance': 2e-08} 
  > Extra:
 {'quantization_level_cnt': 2, 'quantize_at_test_only': False, 'adaptive_quantization': False, 'alpha_blending': False, 'step_per_epoch': 1672, 'alpha_blending_start_epoch': 5, 'alpha_blending_end_epoch': 40} ==============<DOE Args>===============


2024-12-01 11:27:52 [INFO] [DOE] Quantization base height: 1.536909583045e-06
2024-12-01 11:27:52 [INFO] [QDO] Blending Start Step: 8360
2024-12-01 11:27:52 [INFO] [QDO] Blending End Step: 66880
2024-12-01 11:27:52 [INFO] [QDO] Blending Start Epoch: 5
2024-12-01 11:27:52 [INFO] [QDO] Blending End Epoch: 40
2024-12-01 11:27:52 [INFO] [Sensor] SRF Type= rgb
2024-12-01 11:27:52 [WARNING] [!] The PSF resize mode is enabled.
2024-12-01 11:27:52 [WARNING] [!] The Image resizing is disabled because the `wave_resolution` and `sensor_resolution` is identical.
2024-12-01 11:27:52 [INFO] DOE physical size = 4.10e-03 m.
 Wave resolution = 512.
2024-12-01 11:27:52 [INFO] Using group tag:  000A4LevelHyperspctralTraining
2024-12-01 11:27:52 [INFO] [DIR] training log directory name =  000A4LevelHyperspctralTraining/20241201-112752-SK1300-20nmNoise-2Lv-STE-Sd50-Sc1m
2024-12-01 11:27:52 [INFO] Preparing training datasets from D:/yanyi/datas/ICVL/train...
2024-12-01 11:27:52 [INFO] Dataset loaded from:  D:/yanyi/datas/ICVL/train
2024-12-01 11:27:52 [INFO] Preparing validation datasets from D:/yanyi/datas/ICVL/validation...
2024-12-01 11:27:52 [INFO] Dataset loaded from:  D:/yanyi/datas/ICVL/validation
2024-12-01 11:27:53.118917: I tensorflow/core/profiler/lib/profiler_session.cc:101] Profiler session initializing.
2024-12-01 11:27:53.118969: I tensorflow/core/profiler/lib/profiler_session.cc:116] Profiler session started.
2024-12-01 11:27:53.119025: I tensorflow/core/profiler/backends/gpu/cupti_tracer.cc:1664] Profiler found 1 GPUs
2024-12-01 11:27:53.199258: I tensorflow/core/profiler/lib/profiler_session.cc:128] Profiler session tear down.
2024-12-01 11:27:53.200246: I tensorflow/core/profiler/backends/gpu/cupti_tracer.cc:1798] CUPTI activity buffer flushed
WARNING:tensorflow:From D:\Anaconda3\envs\tf_st\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py:1332: start (from tensorflow.python.eager.profiler) is deprecated and will be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.start` instead.
2024-12-01 11:27:53.219975: I tensorflow/core/profiler/lib/profiler_session.cc:101] Profiler session initializing.
2024-12-01 11:27:53.220040: I tensorflow/core/profiler/lib/profiler_session.cc:116] Profiler session started.
WARNING:tensorflow:From D:\Anaconda3\envs\tf_st\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py:1383: stop (from tensorflow.python.eager.profiler) is deprecated and will be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.stop` instead.
2024-12-01 11:27:53.277306: I tensorflow/core/profiler/lib/profiler_session.cc:67] Profiler session collecting data.
2024-12-01 11:27:53.278280: I tensorflow/core/profiler/backends/gpu/cupti_tracer.cc:1798] CUPTI activity buffer flushed
2024-12-01 11:27:53.298034: I tensorflow/core/profiler/backends/gpu/cupti_collector.cc:521]  GpuTracer has collected 1 callback api events and 1 activity events. 
2024-12-01 11:27:53.298264: I tensorflow/core/profiler/lib/profiler_session.cc:128] Profiler session tear down.
WARNING:tensorflow:From D:\Anaconda3\envs\tf_st\lib\site-packages\tensorflow\python\ops\summary_ops_v2.py:1383: save (from tensorflow.python.eager.profiler) is deprecated and will be removed after 2020-07-01.
Instructions for updating:
`tf.python.eager.profiler` has deprecated, use `tf.profiler` instead.
WARNING:tensorflow:From D:\Anaconda3\envs\tf_st\lib\site-packages\tensorflow\python\eager\profiler.py:150: maybe_create_event_file (from tensorflow.python.eager.profiler) is deprecated and will be removed after 2020-07-01.
Instructions for updating:
`tf.python.eager.profiler` has deprecated, use `tf.profiler` instead.
2024-12-01 11:27:53 [INFO] Creating dir: ./checkpoint/000A4LevelHyperspctralTraining/20241201-112752-SK1300-20nmNoise-2Lv-STE-Sd50-Sc1m
2024-12-01 11:27:53 [INFO] [QDO] `quantization-aware` mode is enabled.
2024-12-01 11:27:53 [INFO] [QDO] using quantization-aware approach: <STE>.
2024-12-01 11:27:53 [INFO] [QDO] Quantization levels: 2
2024-12-01 11:27:54 [INFO] Simulated fabrication noise on height map: 2.00e-08
2024-12-01 11:27:55.010584: F tensorflow/stream_executor/cuda/cuda_driver.cc:147] Failed setting context: CUDA_ERROR_NOT_PERMITTED: operation not permitted

Hi @Zhou_Wu, Could you please let us know how this error occurred like executing which command/code causes this error. Also let us know the environmental details you are using. Thank You.