There is an issue when my model is being saved using tf.saved_model.save in tf 2.16.1. The error is that, there is a type missmatch for a certain feature (my_text_feature) is expecting string and getting float).
Input ‘input_bytes’ of ‘DecodePaddedRaw’ Op has type float32 that does not match expected type of string.
Arguments received by BytesSequenceToEncodingBiLSTM.call():
• inputs=tf.Tensor(shape=(32, 1), dtype=float32)
• training=None
I made sure my data types are correct, plus the model training and evaluation completed successfully. This error happens only when trying to save the model.
I did some tracing for the error and I found the following. The call to tf.saved_model.save would invoke my model twice. The first call to it the feature value for my_text_feature is correct (string) which is then transformed to floats (after applying BiLSTM embedding), which is correct.
however the second call passes float to my_text_feature instead of string which crashed in the embedding phase.
I did some tracing and found that this function _build_meta_graph_impl in tensorflow.python.saved_model.save.py gets called when tf.saved_model.save is called. this function _build_meta_graph_impl make the call twice.
#first call
signatures, wrapped_functions, defaults = (signature_serialization.canonicalize_signatures(signatures))
#second call
signature_serialization.validate_augmented_graph_view(augmented_graph_view)
Can someone help why the second call is problematic and why this issue did not rise in older tf version (specifically 2.9)