# Help Debugging Mirrored Strategy with Loss going to NAN

**URL:** <https://discuss.ai.google.dev/t/help-debugging-mirrored-strategy-with-loss-going-to-nan/32176>\
**Category:** General Discussion\
**Tags:** gpu, keras, distributed-training\
**Created:** [January 16, 2023, 2:03am UTC](https://discuss.ai.google.dev/t/help-debugging-mirrored-strategy-with-loss-going-to-nan/32176 "2023-01-16T02:03:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![arosado](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@arosado](https://discuss.ai.google.dev/u/arosado)\
**Post date:** [January 16, 2023, 2:03am UTC](https://discuss.ai.google.dev/t/help-debugging-mirrored-strategy-with-loss-going-to-nan/32176/1 "2023-01-16T02:03:39Z")

</div>

I’ve been having a lot of difficulty getting distributed training to work as I would expect using mirrored strategy. This is the first tensorflow project I’ve been working on distributed training with. My project involves input data that is effectively an image into a general CNN (defined below). To accomplish this I’m using a general keras training loop (also defined below). The model trains without issues using a single GPU, but **when I attempt the mirrored strategy my loss goes to nan**. I’ve attempted using different optimizer and optimization settings (adjusting learning rate, epsilon), changing the activation function into the model’s softmax, removing regularization from my model, but all of my attempts have failed to result in progress. **I’m considering writing my own training loop, but I would like to learn if there are any other options I can try as well to debug my issue.**

```auto
model = keras.Sequential([
    keras.layers.Input(shape=input_shape),
    keras.layers.Conv2D(32, kernel_size=3, activation='relu',data_format='channels_last'),
    BatchNormalization(),
    keras.layers.MaxPool2D((2,2), data_format='channels_last'),
    keras.layers.Conv2D(64, kernel_size=3, activation='relu',data_format='channels_last'),
    BatchNormalization(),
    keras.layers.MaxPool2D((2,2), data_format='channels_last'),
    keras.layers.Conv2D(128, kernel_size=3, activation='relu',data_format='channels_last'),
    BatchNormalization(),
    keras.layers.MaxPool2D((2,2), data_format='channels_last'),
    keras.layers.Conv2D(128, kernel_size=3, activation='relu',data_format='channels_last'),
    BatchNormalization(),
    keras.layers.MaxPool2D((2,2), data_format='channels_last'),
    keras.layers.Conv2D(128, kernel_size=3, activation='relu',data_format='channels_last'),
    BatchNormalization(),
    keras.layers.MaxPool2D((2,2), data_format='channels_last'),
    keras.layers.Dropout(0.5),
    keras.layers.Flatten(),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dense(32, activation='relu'),
    keras.layers.Dense(2, activation='softmax')
])

```

```auto
with strategy.scope():
        from models.cnn.cnn import model

        model.summary()
        optimizer = optimization_helper.return_optimizer()
        loss_object = tf.keras.losses.CategoricalCrossentropy()
        metrics_list = [
            metrics.Precision(),
            metrics.CategoricalAccuracy(),
        ]

        earlystopping = callbacks.EarlyStopping(monitor = "val_loss", mode= "min",
                                                patience= optimization['early_stopping_patience'], restore_best_weights = True)

        scheduler = callbacks.ReduceLROnPlateau(monitor='val_loss', factor=optimization['scheduler_factor'],
                                                patience=optimization['scheduler_patience'], min_lr=optimization['scheduler_min_lr'], verbose=1)

        checkpointer = callbacks.ModelCheckpoint(filepath=os.path.join(results_dir, 'best_model_state.hdf5'),
                                                 verbose=1, save_best_only=True)

        callback_list = [
            earlystopping,
            scheduler,
            checkpointer
        ]

    model.compile(
        run_eagerly=False,
        optimizer=optimizer,
        loss=loss_object,
        metrics=metrics_list
    )

    steps_per_epoch = math.ceil(train_len/train_batch_size)
    steps_per_val = math.ceil(val_len/val_batch_size)

    history = model.fit(x=train_datasets,
                        validation_data=val_datasets,
                        steps_per_epoch=steps_per_epoch,
                        validation_steps=steps_per_val,
                        epochs=optimization['total_target_epochs'],
                        verbose=1,
                        callbacks = callback_list,
                        use_multiprocessing=False,
                        shuffle=True)

```

---

<div class="post-metadata">

**Author:** ![jasmine\_dhantule](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/jasmine_dhantule/32/32711_2.png) [@jasmine\_dhantule](https://discuss.ai.google.dev/u/jasmine_dhantule)\
**Post date:** [October 29, 2025, 5:32pm UTC](https://discuss.ai.google.dev/t/help-debugging-mirrored-strategy-with-loss-going-to-nan/32176/3 "2025-10-29T17:32:12Z")

</div>

Hi @arosado,

Please upgrade to latest Keras version and check if you’re still facing this issue, if yes provide standalone code to investigate this further. Thanks!