Unable to train MobileNet V3 Small. Model is no training at all

TensorFlow
1

Hello Everyone

I am new to Tensorflow (versrion 2.19.0) and face problem which i can’t solve and understand. I am trying to train MobileNetV3 Small for image classification of my custom image dataset.

First i have selected MobileNetV3 small, because i need small model which can run on edge device. Input and ouput shape were modifed. No initial weights.

model = keras.applications.MobileNetV3Small(
    input_shape=(64, 256, 1),
    weights=None,
    classes=4,
    alpha=0.50,
    include_top=True,
    include_preprocessing=False,
)

Than for Tensorflow learning purpose i have selected Cifar10 database and modify shape and label so they fit my final model purpose. I have also shrink database only to 3000 image (2600 train, 400 validation), because this i how large is my current custom image database.

(train_ds, test_ds), ds_info = tfds.load(
    'cifar10',
    split=['train', 'test'],
    as_supervised=True
)
def preprocess(image, label):
    image = tf.image.resize(image, [64, 256])
    image = tf.image.rgb_to_grayscale(image) 
    label = tf.one_hot(label % 4, 4)
    return image, label
train_ds = train_ds.map(preprocess).take(2600)
test_ds = test_ds.map(preprocess).take(400)

My compilation setup:

model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-4),
    loss=tf.keras.losses.CategoricalCrossentropy(from_logits=False),
    metrics=[tf.keras.metrics.CategoricalAccuracy()]
)

Than i start training:

history = model.fit(
    train_ds,
    validation_data=test_ds,
    epochs=50,
)

During training i am facing fallowing problem:

  • model is very strong overfitting for traning data, accuracy is 100%
  • accuracy for validation is not changing at all, looks model is not training at all
Epoch 1/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 18s 84ms/step - categorical_accuracy: 0.3531 - loss: 1.3429 - val_categorical_accuracy: 0.1860 - val_loss: 1.3841
Epoch 2/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 79ms/step - categorical_accuracy: 0.4496 - loss: 1.2161 - val_categorical_accuracy: 0.1860 - val_loss: 1.3847
Epoch 3/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 78ms/step - categorical_accuracy: 0.5381 - loss: 1.0560 - val_categorical_accuracy: 0.1860 - val_loss: 1.3850
Epoch 4/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 78ms/step - categorical_accuracy: 0.6314 - loss: 0.8830 - val_categorical_accuracy: 0.1860 - val_loss: 1.3854
Epoch 5/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 78ms/step - categorical_accuracy: 0.7281 - loss: 0.6966 - val_categorical_accuracy: 0.1860 - val_loss: 1.3870
Epoch 6/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 78ms/step - categorical_accuracy: 0.8022 - loss: 0.5202 - val_categorical_accuracy: 0.1860 - val_loss: 1.3900
Epoch 7/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 8s 80ms/step - categorical_accuracy: 0.8784 - loss: 0.3808 - val_categorical_accuracy: 0.1860 - val_loss: 1.3940
Epoch 8/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 78ms/step - categorical_accuracy: 0.9171 - loss: 0.2701 - val_categorical_accuracy: 0.1860 - val_loss: 1.4014
Epoch 9/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 79ms/step - categorical_accuracy: 0.9382 - loss: 0.1935 - val_categorical_accuracy: 0.1860 - val_loss: 1.4121
Epoch 10/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 8s 81ms/step - categorical_accuracy: 0.9537 - loss: 0.1513 - val_categorical_accuracy: 0.1860 - val_loss: 1.4218
Epoch 11/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 79ms/step - categorical_accuracy: 0.9578 - loss: 0.1396 - val_categorical_accuracy: 0.1860 - val_loss: 1.4291
Epoch 12/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 8s 80ms/step - categorical_accuracy: 0.9724 - loss: 0.1169 - val_categorical_accuracy: 0.1860 - val_loss: 1.4509
Epoch 13/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 79ms/step - categorical_accuracy: 0.9730 - loss: 0.0942 - val_categorical_accuracy: 0.1860 - val_loss: 1.4747
Epoch 14/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 8s 80ms/step - categorical_accuracy: 0.9915 - loss: 0.0414 - val_categorical_accuracy: 0.1860 - val_loss: 1.4959
Epoch 15/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 78ms/step - categorical_accuracy: 0.9958 - loss: 0.0234 - val_categorical_accuracy: 0.1860 - val_loss: 1.5192
Epoch 16/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 79ms/step - categorical_accuracy: 0.9949 - loss: 0.0197 - val_categorical_accuracy: 0.1860 - val_loss: 1.5578
Epoch 17/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 8s 80ms/step - categorical_accuracy: 0.9956 - loss: 0.0148 - val_categorical_accuracy: 0.1860 - val_loss: 1.6030
Epoch 18/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 7s 79ms/step - categorical_accuracy: 0.9989 - loss: 0.0082 - val_categorical_accuracy: 0.1860 - val_loss: 1.6624
Epoch 19/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 8s 80ms/step - categorical_accuracy: 0.9998 - loss: 0.0042 - val_categorical_accuracy: 0.1860 - val_loss: 1.7080
Epoch 20/50
94/94 ━━━━━━━━━━━━━━━━━━━━ 8s 82ms/step - categorical_accuracy: 1.0000 - loss: 0.0015 - val_categorical_accuracy: 0.1860 - val_loss: 1.7699

The most strange part is after training, when i evaluate model for both training and validation model model perform very badly, like it didn’t train at all

train_loss, train_acc = model.evaluate(train_ds, verbose=0)
print(f"Train accuracy: {train_acc}")
print(f"Train loss: {train_loss}")
val_loss, val_acc = model.evaluate(test_ds, verbose=0)
print(f"Validation accuracy: {val_acc}")
print(f"Validation loss: {val_loss}")

Train accuracy 0.20267
Train loss: 5.95243
Validation accuracy: 0.18600
Validation loss: 6.137097

Please explain me:

  • why validation accuracy don’t change at all, and loss even increase. It looks like the model not training at all
  • why for training dataset in model evaluation accuracy is so low when in training model get 100% for the same dataset
2

I have even done fallowing test, where training and validation dataset is the same (I know this isn’t correct approach, but let’s ignore it)

history = model.fit(
    train_ds,
    validation_data=train_ds,
    epochs=50,
    verbose=1,
    callbacks=[checkpoint_cb]
)

In theory for such a setup model should strongly over-fitting and accuracy for both training and validation should be very high (almost the same)

But again model:

  • very strongly overfitting for training dataset (as expected), accuracy reach 100%
  • no learning at all for validation dataset, even if this is exactly the same database as training (we should expect here also almost 100%)
Epoch 1/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 21s 110ms/step - categorical_accuracy: 0.3517 - loss: 1.3471 - val_categorical_accuracy: 0.1969 - val_loss: 1.3844
Epoch 2/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 99ms/step - categorical_accuracy: 0.4447 - loss: 1.2225 - val_categorical_accuracy: 0.1969 - val_loss: 1.3855
Epoch 3/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 101ms/step - categorical_accuracy: 0.5396 - loss: 1.0788 - val_categorical_accuracy: 0.1969 - val_loss: 1.3868
Epoch 4/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 102ms/step - categorical_accuracy: 0.6181 - loss: 0.8995 - val_categorical_accuracy: 0.1969 - val_loss: 1.3887
Epoch 5/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 101ms/step - categorical_accuracy: 0.6952 - loss: 0.7301 - val_categorical_accuracy: 0.1969 - val_loss: 1.3923
Epoch 6/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 101ms/step - categorical_accuracy: 0.7625 - loss: 0.5682 - val_categorical_accuracy: 0.1969 - val_loss: 1.3984
Epoch 7/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 103ms/step - categorical_accuracy: 0.8235 - loss: 0.4365 - val_categorical_accuracy: 0.1969 - val_loss: 1.4073
Epoch 8/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 9s 104ms/step - categorical_accuracy: 0.8668 - loss: 0.3546 - val_categorical_accuracy: 0.1969 - val_loss: 1.4185
Epoch 9/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 9s 104ms/step - categorical_accuracy: 0.9177 - loss: 0.2464 - val_categorical_accuracy: 0.1969 - val_loss: 1.4339
Epoch 10/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 9s 106ms/step - categorical_accuracy: 0.9509 - loss: 0.1630 - val_categorical_accuracy: 0.1969 - val_loss: 1.4557
Epoch 11/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 102ms/step - categorical_accuracy: 0.9703 - loss: 0.1207 - val_categorical_accuracy: 0.1969 - val_loss: 1.4815
Epoch 12/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 102ms/step - categorical_accuracy: 0.9783 - loss: 0.0877 - val_categorical_accuracy: 0.1969 - val_loss: 1.5078
Epoch 13/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 102ms/step - categorical_accuracy: 0.9713 - loss: 0.0985 - val_categorical_accuracy: 0.1969 - val_loss: 1.5390
Epoch 14/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 103ms/step - categorical_accuracy: 0.9835 - loss: 0.0602 - val_categorical_accuracy: 0.1969 - val_loss: 1.5737
Epoch 15/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 103ms/step - categorical_accuracy: 0.9907 - loss: 0.0352 - val_categorical_accuracy: 0.1969 - val_loss: 1.6057
Epoch 16/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 101ms/step - categorical_accuracy: 0.9953 - loss: 0.0272 - val_categorical_accuracy: 0.1969 - val_loss: 1.6318
Epoch 17/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 102ms/step - categorical_accuracy: 0.9989 - loss: 0.0153 - val_categorical_accuracy: 0.1969 - val_loss: 1.6752
Epoch 18/50
82/82 ━━━━━━━━━━━━━━━━━━━━ 8s 103ms/step - categorical_accuracy: 1.0000 - loss: 0.0063 - val_categorical_accuracy: 0.1969 - val_loss: 1.7243

After training again evaluation for training is very bad, like model didn’t learn anything.

train_loss, train_acc = model.evaluate(train_ds, verbose=0)
print(f"Train accuracy: {train_acc}")
print(f"Train loss: {train_loss}")

Train accuracy: 0.19692
Train loss: 4.62061

Please explain why it is possible that in model fit, for training dataset it got 100%, but later in evaluation of the exactly the same dataset it get only 19%. Model didn’t learn anything.

4

you have used tf.keras.losses.CategoricalCrossentropy(from_logits=False) but you have not applied softmax in the final layer due to which the ouputs are raw logits due to which wring gradients come during training and your accuracy comes 100% but when in evaluation it drops drastically.