What does the run_eagerly parameter in model.compile do?

[Lu_Bin_Liu]

From the documentation:

Bool. Defaults to False . If True , this Model 's logic will not be wrapped in a tf.function. Recommended to leave this as None unless your Model cannot be run inside a tf.function. run_eagerly=True is not supported when using tf.distribute.experimental.ParameterServerStrategy.

What is the significance of being wrapped in tf.function and what are the practical advantage/disadvantages of setting run_eagerly = True?

[Sayak_Paul]

When something is wrapped inside tf.function it has the advantage of being run in graph mode. All the backend compilation engineering is handled by TensorFlow itself in this case. The advantage is when all the operations are available as a graph we know much resources to allocate and how to best optimize the graph with the available resources. For more details refer to the following:

Better performance with tf.function  |  TensorFlow Core

run_eagerly=True lets figure out what exactly is going inside your model training loop. Let’s say you have implemented a custom loop and put that inside the train_step() method of a subclasses model. Setting run_eagerly to True will help you debug that loop if anything goes wrong. For practical applications of this, refer to the following guide:

Keras documentation: Keras debugging tips

[Lu_Bin_Liu]

Thanks very much! From the second link,

Thankfully, there’s an easy way to run your code in “debug mode”, fully eagerly: pass run_eagerly=True to compile(). Your call to fit() will now get executed line by line, without any optimization. It’s slower, but it makes it possible to print the value of intermediate tensors, or to use a Python debugger. Great for debugging.

does this mean that when run_eagerly = False, values of intermediate tensors in the train_step() cannot be saved? If I save some intermediate tensor(s) as an instance variable in a subclassed Model object, can I extract the values of these tensors after .fit() is complete as say a numpy array?

[Sayak_Paul]

Could you provide an example use-case?

[Lu_Bin_Liu]

I slightly modified the first example from this tutorial by saving the y_pred from each epoch

class CustomModel(keras.Model):

def __init__(self, **kwargs):
    super().__init__(**kwargs)
    self.saved_pred = []

def train_step(self, data):
    x, y = data
    
    with tf.GradientTape() as tape:
        y_pred = self(x, training=True)  # Forward pass
        loss = self.compiled_loss(y, y_pred, regularization_losses=self.losses)

    # Compute gradients
    trainable_vars = self.trainable_variables
    gradients = tape.gradient(loss, trainable_vars)
    # Update weights
    self.optimizer.apply_gradients(zip(gradients, trainable_vars))
    # Update metrics (includes the metric that tracks the loss)
    self.compiled_metrics.update_state(y, y_pred)
    # Return a dict mapping metric names to current value

    self.saved_pred.append(y_pred)

    return {m.name: m.result() for m in self.metrics}

import numpy as np

inputs = keras.Input(shape=(32,))
outputs = keras.layers.Dense(1)(inputs)
model = CustomModel(inputs = inputs, outputs = outputs)
model.compile(optimizer=“adam”, loss=“mse”, metrics=[“mae”])

x = np.random.random((1000, 32))
y = np.random.random((1000, 1))
model.fit(x, y, epochs=3)

Here’s the output of model.saved_preds:

ListWrapper([<tf.Tensor ‘custom_model_1/dense_5/BiasAdd:0’ shape=(None, 1) dtype=float32>, <tf.Tensor ‘custom_model_1/dense_5/BiasAdd:0’ shape=(None, 1) dtype=float32>])

There’s no numpy attribute for the Tensors in this list, so I’m wondering if it’s possible to extract their values.

[Sayak_Paul]

Looks like a list. Can you do something like model.saved_preds[0].numpy()? The model is supposed to be returning predictions i.e. some distribution or a set of values that must not contain any operations like custom_model_1/dense_5/BiasAdd:0.

[Lu_Bin_Liu]

When trying .numpy() and tf.keras.backend.get_values(), I got an error saying that Tensor object has no attribute numpy. I also tried .eval which gave this error:

ValueError: Cannot use the given session to evaluate tensor: the tensor’s graph is different from the session’s graph.

[Sayak_Paul]

Since you are running with run_eagerly=True now, you can print self.saved_preds inside your train_step() function directly for a better debugging experience. Could you do while calling .fit() which I suppose you are doing currently.

[Lu_Bin_Liu]

But if I want to set run_eagerly = False (for deployment), then there’s no way to extract the value of the Tensors from self.saved_preds?

[Sayak_Paul]

Why would you wanna do that when you can simply call .predict() or even the model(x) directly? Just trying to better understand the situation here.

[Lu_Bin_Liu]

Sorry I gave that as a simplified example. I would like to see if it’s possible to save intermediate Tensor results in the train_step(), with y_pred being a stand in for any arbitrary Tensor that was computed inside train_step() in a subclassed Model class.

[Sayak_Paul]

Yeah, it should be possible. I have been able to save entire models with train_step(). Here’s one such example:

Keras documentation: Self-supervised contrastive learning with SimSiam