Which subpackage of tensorflow loads libcublasLt.so?

TensorFlow General Discussion
1

While using model.fit method in keras-tensorflow i get the following error:

Could not load library libcublasLt.so.12. Error: libcublasLt.so.12: cannot open shared object file: No such file or directory

It is true that I did have a cuda 12.1 version installed for a second (tensorflow could not detect it), so I downgraded it to 11.8 version (tensorflow detects it, but I get the above error :frowning: ). I do not have any clue why still tries to load the version 12.

  1. I reinstalled cuda multiple times, together with tensorflow, cudnn; it did not help
  2. These commands
import tensorflow as tf

print('TensorFlow version:',tf.__version__)
physical_devices = tf.config.list_physical_devices()
for dev in physical_devices:
    print(dev)

sys_details = tf.sysconfig.get_build_info()
cuda_version = sys_details["cuda_version"]
print("CUDA version:",cuda_version)
cudnn_version = sys_details["cudnn_version"]
print("CUDNN version:",cudnn_version)

produce the following result

TensorFlow version: 2.12.0
PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')
PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
CUDA version: 11.8
CUDNN version: 8

3 Problem starts when using .fit. Otherwise print(tf.reduce_sum(tf.random.normal([1000, 1000]))) indicates that tensorflow is able to make computations.

  1. I have set LD_LIBRARY_PATH to include directores having the appropriate files (but version 11.8 of course), and more; just in case.
$ export | grep LD_
declare -x LD_LIBRARY_PATH="/home/jprzew/.local/lib/python3.8/site-packages/nvidia/cudnn/:/usr/lib/x86_64-linux-gnu:/usr/local/cuda-11.8/lib64/:/usr/local/cuda-11.8/targets/x86_64-linux/lib/"



$ sudo find / -name "libcublasLt*"
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcublasLt.so.11.9.2.110
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcublasLt.so
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcublasLt_static.a
/usr/local/cuda-11.6/targets/x86_64-linux/lib/stubs/libcublasLt.so
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcublasLt.so.11
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt_static.a
/usr/local/cuda-11.8/targets/x86_64-linux/lib/stubs/libcublasLt.so
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so.11.11.3.6
/usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so.11

I would be grateful for any clue

3

Looks like sudo apt-get install libcublas-12-0 solved the issue.

1 Like
5

Thank you for this timely reply, solved my problem! :smile: