Tensorflow randomly stops working with GPU

TensorFlow General Discussion
, ,
1

I have created a conda environment and installed tensorflow 2.8 by following the windows native installation guide
When I then run my program using the said conda environment it works perfectly. However randomly after a while then it stops using my GPU and the following errors occur, this has happened numerous times with different conda environments and installation methods.

2023-05-17 14:39:30.489082: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2023-05-17 14:39:30.489465: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2023-05-17 14:39:33.663179: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2023-05-17 14:39:33.665053: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2023-05-17 14:39:33.667174: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2023-05-17 14:39:33.669143: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2023-05-17 14:39:33.671077: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2023-05-17 14:39:33.673018: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusolver64_11.dll'; dlerror: cusolver64_11.dll not found
2023-05-17 14:39:33.675509: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2023-05-17 14:39:33.677622: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

I then have to try and recreate the environment from scratch every time and the problem still happens after initially working fine. I have checked the environment folder and those DLL files are in there. Any help would be appreciated.

Also when running tensorflow.config.list_physical_devices(‘GPU’) in the command prompt with the environment active it shows that it is working in the GPU but when I run it in pycharm with my program the errors occur

3

@A.G_OSMAN,

I then have to try and recreate the environment from scratch every time and the problem still happens after initially working fine. I have checked the environment folder and those DLL files are in there. Any help would be appreciated.

According to tested build configurations, please try with CUDA 11.2 and cuDNN 8.1.

but when I run it in pycharm with my program the errors occur

Ensure that you have activated the same environment in PyCharm as you are using in the command prompt.

Thank you!

4

I am using those versions and also have made sure it is using the same environment. I uninstalled and reinstalled pycharm which fixed it for day but now the issue is back

6

Do you mean Tensorflow 2.12? I don’t think 2.8 is out yet.

Regarding Windows and TF 2.12 with GPU support. I tried several times without luck. The instructions say it needs WSL2, but not whether it must be installed from the wsl prompt or windows prompt. I don’t know how it gets to the windows cuda drivers, or any other of that kind of installation hell. I tried PyTorch, that installs and works with GPU support, without sweat.