How to classify new images using fine-tuned pre-trained model without an input image for reference

Hi, I’ve been trying to make a model for image classification for a long time, failing every attempt, and the last thing I found searching on google was using a pre-trained model and fine tuning it.

I got the code working and can generate models, but I don’t know how to use it to classify a new image as “x” or “non x”. Right now the code I have to do it requires the input of an image (class “x”) to pre-process it and check if the image I want to analyse is going to be from that class or not, but I would like to know how could I do this without the need of that input.

import matplotlib.pyplot as plt
import numpy as np
import os
import tensorflow as tf

tf.keras.backend.clear_session()
PATH = 'database'

train_dir = os.path.join(PATH, 'train')
validation_dir = os.path.join(PATH, 'validation')

BATCH_SIZE = 8
IMG_SIZE = (224, 224)

train_dataset = tf.keras.utils.image_dataset_from_directory(train_dir,
                                                            shuffle=True,
                                                            batch_size=BATCH_SIZE,
                                                            image_size=IMG_SIZE)

validation_dataset = tf.keras.utils.image_dataset_from_directory(validation_dir,
                                                            shuffle=True,  
                                                            batch_size=BATCH_SIZE,
                                                            image_size=IMG_SIZE)

class_names = train_dataset.class_names

val_batches = tf.data.experimental.cardinality(validation_dataset)
test_dataset = validation_dataset.take(val_batches // 5)
validation_dataset = validation_dataset.skip(val_batches // 5)

AUTOTUNE = tf.data.AUTOTUNE

train_dataset = train_dataset.prefetch(buffer_size=AUTOTUNE)
validation_dataset = validation_dataset.prefetch(buffer_size=AUTOTUNE)
test_dataset = test_dataset.prefetch(buffer_size=AUTOTUNE)

data_augmentation = tf.keras.Sequential([
  tf.keras.layers.RandomFlip('horizontal'),
  tf.keras.layers.RandomRotation(0.2),
])

preprocess_input = tf.keras.applications.vgg16.preprocess_input

IMG_SHAPE = IMG_SIZE + (3,)
base_model = tf.keras.applications.VGG16(input_shape=IMG_SHAPE,
                                               include_top=False,
                                               weights='imagenet')

image_batch, label_batch = next(iter(train_dataset))
feature_batch = base_model(image_batch)

base_model.trainable = False

global_average_layer = tf.keras.layers.GlobalAveragePooling2D()
feature_batch_average = global_average_layer(feature_batch)

prediction_layer = tf.keras.layers.Dense(1)
prediction_batch = prediction_layer(feature_batch_average)

inputs = tf.keras.Input(shape=(224, 224, 3))
x = data_augmentation(inputs)
x = preprocess_input(x)
x = base_model(x, training=False)
x = global_average_layer(x)
x = tf.keras.layers.Dropout(0.2)(x)
outputs = prediction_layer(x)
model = tf.keras.Model(inputs, outputs)

base_learning_rate = 0.0001
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=base_learning_rate),
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=[tf.keras.metrics.BinaryAccuracy(threshold=0, name='accuracy')])

initial_epochs = 20

loss0, accuracy0 = model.evaluate(validation_dataset)

print("initial loss: {:.2f}".format(loss0))
print("initial accuracy: {:.2f}".format(accuracy0))

history = model.fit(train_dataset,
                    epochs=initial_epochs,
                    validation_data=validation_dataset)

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

# Fine-tune from this layer onwards
fine_tune_at = 16

# Freeze all the layers before the `fine_tune_at` layer
for layer in base_model.layers[:fine_tune_at]:
  layer.trainable = False

model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer = tf.keras.optimizers.RMSprop(learning_rate=base_learning_rate/10),
              metrics=[tf.keras.metrics.BinaryAccuracy(threshold=0, name='accuracy')])

fine_tune_epochs = 20
total_epochs =  initial_epochs + fine_tune_epochs

history_fine = model.fit(train_dataset,
                         epochs=total_epochs,
                         initial_epoch=history.epoch[-1],
                         validation_data=validation_dataset)

acc += history_fine.history['accuracy']
val_acc += history_fine.history['val_accuracy']

loss += history_fine.history['loss']
val_loss += history_fine.history['val_loss']

loss, accuracy = model.evaluate(test_dataset)
print('Test accuracy :', accuracy)

model.save('finetuned_vgg16_20epoch_16layers.h5')

And this is the code I have now for checking if a new image is “x” or not, which needs an input image of class “x”.

import tensorflow as tf
from tensorflow.keras.applications.vgg16 import VGG16
import tensorflow.keras.utils as image
from tensorflow.keras.applications.vgg16 import preprocess_input
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
import os

model = tf.keras.models.load_model('finetuned_vgg16.h5')

query_image_path = 'input_image.jpg'
img = image.load_img(query_image_path, target_size=(224, 224))

x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

query_features = model.predict(x)

database_image_dir = 'db_images/'
similarities = []
for database_image_name in os.listdir(database_image_dir):
	img = image.load_img(database_image_dir+database_image_name, target_size=(224, 224))
	x = image.img_to_array(img)
	x = np.expand_dims(x, axis=0)
	x = preprocess_input(x)
	database_image_features = model.predict(x)
	similarity = cosine_similarity(query_features.reshape(1, -1), database_image_features.reshape(1, -1))
	print(query_features.reshape(1, -1))
	print(database_image_features.reshape(1, -1))

	if similarity >= 0.5: similarities.append(database_image_dir+database_image_name)
	print(similarity)

print(len(similarities))
print(similarities)

How could I do to check if an image if from class “x” or not without having to give it an “x” class image to preprocess?

Hi @Sergio, To make predictions on new images first you have to resize the image to the input shape of the model. For example, if your input shape was (224,224) the image should also be (224,224).

Then you have to convert those image into an array. If you have a normalization layer defined inside you don’t need to normalize the array or else you have to normalize the array.

Now you need to add a dimension to the input array which will be the batch size. And you can make predictions using model.predict( ).

Example code

model = ResNet50(weights='imagenet')

img_path = '/content/elephant.jpeg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

preds = model.predict(x)

Thank you.

Hi @Kiran_Sai_Ramineni,
Thanks for answering. I tried that previously but got some results I don’t know how to interpret.
I get results varying between -20 to 30 (maybe the difference is higher but I don’t have more images to check) and the negative ones are the correct ones (the images that belong to class “x”).

I have no clue why the VGG16 model gives results from 0.0 to 1.0 but my fine-tuned model messes the result values. Any idea what can be happening or how could I make the results go from 0 to 1 again?

As your classification task is binary, for the last dense layer the activation should be sigmoid

prediction_layer=tf.keras.layers.Dense(1, activation='sigmoid')

The default activation will be none for the dense layer. So could you please add the sigmoid activation and then train the model. And see the prediction results. Thank You.

Hi, I tried training a model again adding activation=‘sigmoid’ to the dense layer, but now the accuracy and the validation accuracy stay the same through the whole process. Constant 0.7914 and 0.7807 after 100 epoch.

Hi @Sergio, If the accuracy and loss become constant after a certain number of epochs which implies the model has reached the saturation state and stops learning. In order to increase the accuracy try adding more data to training, perform hyper parameter tuning like using different learning rates and batch size. Thank You.

I don’t think that’s my case. I’ve tried all that but the accuracy stays the same it is from first epoch.
When I do it without sigmoid, the model trains properly (apparently), but the predict numbers get all messed up.
Something must not be right with the code and I don’t know what can it be.