Hi, to debug the origin of NaN values in gradients I’d like to plot the graph of the gradient computation.
Is this possible in tensorflow? I looked around a lot and did not find a working solution.
The question is obsolete - I was able to solve my NaN problem. For the sake of future readers: I used tf.norm somewhere, where I had forgotten it is used (Vector Quantizer). tf.norm is numerically unstable for small values. You have to modify it according to:
def safe_norm(x, epsilon=1e-12, axis=None, keepdims=False):
return tf.sqrt(tf.reduce_sum(x ** 2, axis=axis, keepdims=keepdims) + epsilon)
I found this variant on github. Approach the issue similarly for anything using tf.sqrt.