While using model.fit method in keras-tensorflow i get the following error:
Could not load library libcublasLt.so.12. Error: libcublasLt.so.12: cannot open shared object file: No such file or directory
It is true that I did have a cuda 12.1 version installed for a second (tensorflow could not detect it), so I downgraded it to 11.8 version (tensorflow detects it, but I get the above error ). I do not have any clue why still tries to load the version 12.
- I reinstalled cuda multiple times, together with tensorflow, cudnn; it did not help
- These commands
import tensorflow as tf
print('TensorFlow version:',tf.__version__)
physical_devices = tf.config.list_physical_devices()
for dev in physical_devices:
print(dev)
sys_details = tf.sysconfig.get_build_info()
cuda_version = sys_details["cuda_version"]
print("CUDA version:",cuda_version)
cudnn_version = sys_details["cudnn_version"]
print("CUDNN version:",cudnn_version)
produce the following result
TensorFlow version: 2.12.0
PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')
PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
CUDA version: 11.8
CUDNN version: 8
3 Problem starts when using .fit. Otherwise print(tf.reduce_sum(tf.random.normal([1000, 1000])))
indicates that tensorflow is able to make computations.
- I have set LD_LIBRARY_PATH to include directores having the appropriate files (but version 11.8 of course), and more; just in case.
$ export | grep LD_
declare -x LD_LIBRARY_PATH="/home/jprzew/.local/lib/python3.8/site-packages/nvidia/cudnn/:/usr/lib/x86_64-linux-gnu:/usr/local/cuda-11.8/lib64/:/usr/local/cuda-11.8/targets/x86_64-linux/lib/"
$ sudo find / -name "libcublasLt*"
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcublasLt.so.11.9.2.110
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcublasLt.so
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcublasLt_static.a
/usr/local/cuda-11.6/targets/x86_64-linux/lib/stubs/libcublasLt.so
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcublasLt.so.11
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt_static.a
/usr/local/cuda-11.8/targets/x86_64-linux/lib/stubs/libcublasLt.so
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so.11.11.3.6
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so.11
I would be grateful for any clue