# Train accuracy is decreasing while loss is decreasing

**URL:** <https://discuss.ai.google.dev/t/train-accuracy-is-decreasing-while-loss-is-decreasing/30906>\
**Category:** General Discussion\
**Tags:** tf-probability, models, help\_request\
**Created:** [February 8, 2022, 9:40am UTC](https://discuss.ai.google.dev/t/train-accuracy-is-decreasing-while-loss-is-decreasing/30906 "2022-02-08T09:40:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Constantin\_Werner](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/constantin_werner/32/10524_2.png) [@Constantin\_Werner](https://discuss.ai.google.dev/u/Constantin_Werner)\
**Post date:** [February 8, 2022, 9:40am UTC](https://discuss.ai.google.dev/t/train-accuracy-is-decreasing-while-loss-is-decreasing/30906/1 "2022-02-08T09:40:04Z")

</div>

Hello

I built a model for sequence classification with the hugging face model:

```auto
bert_config = BertConfig.from_pretrained(MODEL_NAME)
        bert_config.output_hidden_states = True
        backbone = TFAutoModelForSequenceClassification.from_pretrained(MODEL_NAME,config=bert_config)

        input_ids = tf.keras.layers.Input(shape=(MAX_LENGTH,), name='input_ids', dtype='int32')
        features = backbone(input_ids)[1][-1]
        pooling = tf.keras.layers.GlobalAveragePooling1D()(features)
        dense1 = tfp.layers.DenseFlipout(512, activation=tf.nn.relu)(pooling)
        final = tfp.layers.DenseFlipout(len(label2id), name='output' ,activation=tf.nn.softmax)(dense1)

        model = tf.keras.Model(inputs=[input_ids], outputs = [final])
        optimizer = tf.keras.optimizers.Adam(learning_rate=6e-6)
        #loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
        loss = elbo_loss
        model.compile(optimizer=optimizer,loss=loss, metrics='accuracy')

```

where elbo\_loss is

```auto
@tf.function
def elbo_loss(labels, logits):
    loss_en = tf.nn.softmax_cross_entropy_with_logits(labels, logits)
    loss_kl = tf.keras.losses.KLD(labels, logits)
    loss = tf.reduce_mean(tf.add(loss_en, loss_kl))
    return loss

```

After several epochs train accuracy is decreasing, but the loss is decreasing. The same thing with CategoricalCrossentropy. I do not see such problem with classical approach (without Bayssien layers)

```auto
Epoch 16/100
1119/1119 [==============================] - 455s 407ms/step - loss: 1489082.6484 - accuracy: 0.8298
Epoch 17/100
1119/1119 [==============================] - 455s 407ms/step - loss: 1485046.3760 - accuracy: 0.8398
Epoch 18/100
1119/1119 [==============================] - 455s 407ms/step - loss: 1481018.1098 - accuracy: 0.8446
Epoch 19/100
1119/1119 [==============================] - 455s 407ms/step - loss: 1476998.0167 - accuracy: 0.8469
Epoch 20/100
1119/1119 [==============================] - 455s 407ms/step - loss: 1472984.8798 - accuracy: 0.8375
Epoch 21/100
1119/1119 [==============================] - 455s 407ms/step - loss: 1468978.1942 - accuracy: 0.8240
Epoch 22/100
1119/1119 [==============================] - 455s 407ms/step - loss: 1464977.2963 - accuracy: 0.7886
Epoch 23/100
1119/1119 [==============================] - 455s 407ms/step - loss: 1460981.7481 - accuracy: 0.7421

```

… up to zero

What can be reason for such behavior?

---

<div class="post-metadata">

**Author:** ![Bhack](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/bhack/32/6388_2.png) [@Bhack](https://discuss.ai.google.dev/u/Bhack)\
**Post date:** [February 8, 2022, 12:18pm UTC](https://discuss.ai.google.dev/t/train-accuracy-is-decreasing-while-loss-is-decreasing/30906/3 "2022-02-08T12:18:19Z")

</div>

I suppose that you want to maximize ELBO:  
[https://github.com/tensorflow/probability/blob/main/tensorflow\_probability/examples/vae.py#L28-L31](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/examples/vae.py#L28-L31)

[https://github.com/tensorflow/probability/blob/main/tensorflow\_probability/examples/vae.py#L384](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/examples/vae.py#L384)

---

<div class="post-metadata">

**Author:** ![Constantin\_Werner](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/constantin_werner/32/10524_2.png) [@Constantin\_Werner](https://discuss.ai.google.dev/u/Constantin_Werner)\
**Post date:** [February 8, 2022, 6:56pm UTC](https://discuss.ai.google.dev/t/train-accuracy-is-decreasing-while-loss-is-decreasing/30906/5 "2022-02-08T18:56:37Z")

</div>

Unfortunately, it does not solve my problem. Moreover, it fails to train a model at all.

---

<div class="post-metadata">

**Author:** ![8bitmp3](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/8bitmp3/32/6397_2.png) [@8bitmp3](https://discuss.ai.google.dev/u/8bitmp3)\
**Post date:** [February 15, 2022, 2:02am UTC](https://discuss.ai.google.dev/t/train-accuracy-is-decreasing-while-loss-is-decreasing/30906/7 "2022-02-15T02:02:30Z")

</div>

Hi @Constantin_Werner 👋 Would you be able to share the entire code or a Colab, so we can try to debug together? Assuming that you’re working with publicly available data, etc.

---

<div class="post-metadata">

**Author:** ![Bhack](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/bhack/32/6388_2.png) [@Bhack](https://discuss.ai.google.dev/u/Bhack)\
**Post date:** [February 15, 2022, 2:58am UTC](https://discuss.ai.google.dev/t/train-accuracy-is-decreasing-while-loss-is-decreasing/30906/8 "2022-02-15T02:58:33Z")

</div>

I still think that you need to use a negative version of your loss:

> **[卷积变分自编码器  |  TensorFlow Core](https://www.tensorflow.org/tutorials/generative/cvae?hl=zh-cn)**

---

<div class="post-metadata">

**Author:** ![Constantin\_Werner](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/constantin_werner/32/10524_2.png) [@Constantin\_Werner](https://discuss.ai.google.dev/u/Constantin_Werner)\
**Post date:** [February 15, 2022, 11:01am UTC](https://discuss.ai.google.dev/t/train-accuracy-is-decreasing-while-loss-is-decreasing/30906/10 "2022-02-15T11:01:06Z")

</div>

Hi @8bitmp3

Here is a link to kaggle notebook with the simplest version of my code. It can reproduce the problem.

> **[bnn\_linear](https://www.kaggle.com/code/icantlie/bnn-linear/notebook)**
>
> Explore and run machine learning code with Kaggle Notebooks | Using data from multiple data sources

It can be seen that on GPU after 0.94 accuracy (\>70 epochs) it starts to decrease while loss is still falling.

Thank you in advance!