Problems loading h5 model

ahmed_lone · August 5, 2021, 8:16am

import os

import tensorflow as tf
from tensorflow import keras
from google.colab import drive
drive.mount(‘/content/gdrive/’)

reconstructed_model = keras.models.load_model(“/content/gdrive/MyDrive/TF/tf_model.h5”)

error:

Drive already mounted at /content/gdrive/; to attempt to forcibly remount, call drive.mount("/content/gdrive/", force_remount=True).

ValueError Traceback (most recent call last)
in ()
6 drive.mount(‘/content/gdrive/’)
7
----> 8 reconstructed_model = keras.models.load_model(“/content/gdrive/MyDrive/TF/tf_model.h5”)

1 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/saving/hdf5_format.py in load_model_from_hdf5(filepath, custom_objects, compile)
174 model_config = f.attrs.get(‘model_config’)
175 if model_config is None:
→ 176 raise ValueError(‘No model found in config file.’)
177 if hasattr(model_config, ‘decode’):
178 model_config = model_config.decode(‘utf-8’)

ValueError: No model found in config file.

lgusm · August 5, 2021, 9:54am

if you do
!ls /content/gdrive/MyDrive/TF/

does it show the file there?
I’d also try copying the model to the colab folder (just for testing) and load it from there. This way you isolate the issue (is it Drive’s fault or not?)

ahmed_lone · August 5, 2021, 9:57am

lgusm · August 5, 2021, 12:17pm

I’ve just tried the model and I’m getting the same issue with the config.json
I don’t know how load that outside of the HF library.
@markdaoust do you know if it can be done?

Mark_Daoust · August 5, 2021, 1:04pm

After hitting that error run a cell containing only %debug.

That will drop you into post mortem debugging and let you poke around and see what’s wrong. You’ll be on this line:

github.com

tensorflow/tensorflow/blob/master/tensorflow/python/keras/saving/hdf5_format.py#L176


      
          if opened_new_file:
            f = h5py.File(filepath, mode='r')
          else:
            f = filepath
          
          model = None
          try:
            # instantiate model
            model_config = f.attrs.get('model_config')
            if model_config is None:
              raise ValueError('No model found in config file.')
            if hasattr(model_config, 'decode'):
              model_config = model_config.decode('utf-8')
            model_config = json_utils.decode(model_config)
            model = model_config_lib.model_from_config(model_config,
                                                       custom_objects=custom_objects)
          
            # set weights
            load_weights_from_hdf5_group(f['model_weights'], model.layers)
          
            if compile:

This failed: model_config = f.attrs.get('model_config') (returned None). So inspect that f.attrs and try to figure out what’s wrong, maybe you can repair the model.

ahmed_lone · August 5, 2021, 2:12pm

i ran debg and it shows me this :

i am a bit lost

ahmed_lone · August 5, 2021, 2:18pm

Mark_Daoust · August 5, 2021, 2:33pm

i ran debg and it shows me this :

Ah, the post mortem ended up in the finally block.

You can use %%debug and set a breakpoint… but that f.close() tells us that opened_new_file is True.

So isinstance(filepath, h5py.File) is returning False. That doesn’t seem to be an H5py file. and that code creates a new empty h5py file?:

github.com

tensorflow/tensorflow/blob/master/tensorflow/python/keras/saving/hdf5_format.py#L165


      
          Raises:
              ImportError: if h5py is not available.
              ValueError: In case of an invalid savefile.
          """
          if h5py is None:
            raise ImportError('`load_model` requires h5py.')
          
          if not custom_objects:
            custom_objects = {}
          
          opened_new_file = not isinstance(filepath, h5py.File)
          if opened_new_file:
            f = h5py.File(filepath, mode='r')
          else:
            f = filepath
          
          model = None
          try:
            # instantiate model
            model_config = f.attrs.get('model_config')
            if model_config is None:

ahmed_lone · August 5, 2021, 2:39pm

okay soo from what u said i understand i don’t have a right h5 file? i am sorry if that wasn’t what u said but i don’t have any experince with h5 files or keras etc. My question is if I could load this model in the hugging face tensorflow version waht should i do right now to load this model in tensorflow

Mark_Daoust · August 5, 2021, 3:43pm

I thought it meant the file was broken, but it looks like I was wrong about that.

%debug opens to that f.close() line.

But if you want figure out what’s wrong with that file try:

f = h5py.File(filepath, mode='r')

and then inspect that f and f.attrs. it’s supposed to have a model_config attribute.

model_config.json

Have you tried: keras.models.model_from_json?

ahmed_lone · August 5, 2021, 3:59pm

that doesn’t seem to be working

as for the attributes

looks like it doesn’t have the proper attributes tho i do have that seperate config file .

Mark_Daoust · August 5, 2021, 4:17pm

I think we’re making progress. The json is the model config, and maybe the h5 is just the weights.

for keras.models.model_from_json I think you need to read the json from the file before so something like model = keras.models.model_from_json(pathlib.Path(filename).read_text())
Then try model.load_weights on the h5 file.

ahmed_lone · August 5, 2021, 4:27pm

I think there is something wrong with the format that hugging face used . I will ask them about it on the form thank you for all your help I will get back to you after I find some more information from there cause I believe this isn’t a problem with tensorflow .

Sergii_Kavun · August 6, 2021, 7:19am

Let’s try to do the following:

just upload h5 model to your local drive - for checking file integrity
if step 1 is OK, then go to step 3, if not - your problem is this saved model as h5 file
I propose to try to use this code for a load of your model just a simple file:

!pip3 install googledrivedownloader # custom library for reading from gdrive
from google_drive_downloader import GoogleDriveDownloader as gdd

gdd.download_file_from_google_drive(file_id=‘file_ID’,
dest_path=‘path_to_your_model’,
unzip=False)

#* file_ID - it’s an identification of a file when you share it by the link, such as “1-CffbOeasBB6s5JkDFuD7zgGkli1aR_f”
#* path_to_your_model - path for your file, such as ./content/gdrive/MyDrive/TF/tf_model.h5 …also will try to a bit change this path in different variants

ahmed_lone · August 6, 2021, 10:25am

hello thanks for the advice! The model is uploaded to my google drive so I didn’t need to do that again as for the saving script