Binary Image Classification Same precision and recall with sixteen decimal digits

Keras
,
1

I there. I am attempting to implement a new neuron initializer (licensed under my university), that had prove that so far works well with multiclass identification type of problem. Corrently i am facing a strange behavior because in a binary classification kind of problem not only the licensed algorithm fails (precision == recall == accuracy) as well as other know algorithms (for instance glorot normal). My neuron layers come from a JSON in a HTTP API way to the APP so i will not publish the neuron layers in code instead this is what i got:
,025 Model: “sequential”
,025 _________________________________________________________________
,025 Layer (type) Output Shape Param #
,025 =================================================================
,026 flatten (Flatten) (None, 12000) 0
,026
,026 rescaling (Rescaling) (None, 12000) 0
,026
,026 dense (Dense) (None, 16) 192016
,026
,026 batch_normalization (Batch (None, 16) 64
,027 Normalization)
,027
,027 dropout (Dropout) (None, 16) 0
,027
,027 dense_1 (Dense) (None, 16) 272
,027
,027 batch_normalization_1 (Bat (None, 16) 64
,027 chNormalization)
,027
,028 dense_2 (Dense) (None, 16) 272
,028
,028 dropout_1 (Dropout) (None, 16) 0
,028
,028 batch_normalization_2 (Bat (None, 16) 64
,028 chNormalization)
,028
,028 dense_3 (Dense) (None, 2) 34
,029
,029 =================================================================
,030 Total params: 192786 (753.07 KB)
,030 Trainable params: 192690 (752.70 KB)
,030 Non-trainable params: 96 (384.00 Byte)
,030 _________________________________________________________________

Only the 2 closing neurons are ‘softmax’ as activation and the rest from above are ‘relu’.
So far this the code i have been working on:

def ml_modeling(last_process_ml_process_response, user_serialized, ml_process, media):
    split_images_only_path = ml_process.split_output_path(media)

    train_path = split_images_only_path + '/train'
    train_path_0 = train_path + '/0'
    train_path_1 = train_path + '/1'
    val_path = split_images_only_path + '/val'
    val_path_0 = val_path + '/0'
    val_path_1 = val_path + '/1'
    model_history_path = split_images_only_path + '/model_history'
    FILE_HELPER.create_directory(model_history_path)

    num_classes = ml_process.dl_ann_img_number_classes
    number_epochs = ml_process.dl_ann_img_number_epochs
    neurons_configuration = ml_process.dl_ann_img_neurons_configurations
    batch_size = ml_process.dl_ann_img_batch_size

    img_height = ml_process.dataset_serialized["image_height"]
    img_width = ml_process.dataset_serialized["image_length"]

    files_in_train_path_0 = False
    if any(os.path.isfile(os.path.join(train_path_0, item)) for item in os.listdir(train_path_0)):
        files_in_train_path_0 = True

    files_in_train_path_1 = False
    if any(os.path.isfile(os.path.join(train_path_1, item)) for item in os.listdir(train_path_1)):
        files_in_train_path_1 = True

    files_in_val_path_0 = False
    if any(os.path.isfile(os.path.join(val_path_0, item)) for item in os.listdir(val_path_0)):
        files_in_val_path_0 = True

    files_in_val_path_1 = False
    if any(os.path.isfile(os.path.join(val_path_1, item)) for item in os.listdir(val_path_1)):
        files_in_val_path_1 = True

    if not (files_in_train_path_0 and files_in_train_path_1 and files_in_val_path_0 and files_in_val_path_1):
        if last_process_ml_process_response == '':
            saved_model_path = ''
        else:
            saved_model_path = last_process_ml_process_response
        return {"error": "No files found in train and val paths.", "saved_model_path": saved_model_path }

    raised_exception = False
    try:

        train_images, train_labels = MlProcessModel.load_images_from_directory(train_path, (img_height, img_width))

        val_images, val_labels = MlProcessModel.load_images_from_directory(val_path, (img_height, img_width))

    except ValueError as e:
        raised_exception = True

    if raised_exception:
        if last_process_ml_process_response == '':
            saved_model_path = ''
        else:
            saved_model_path = last_process_ml_process_response
        return {"error": "An error occurred while creating the image dataset from directory.", "saved_model_path": saved_model_path }

    if (last_process_ml_process_response == ''):
        model = ml_process.model_initializer(train_images, train_labels, val_images, val_labels)

    else:
        model = load_model(last_process_ml_process_response)
        print(model.summary())

    model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0055),
                  loss='binary_crossentropy',  # binary_crossentropy # categorical_crossentropy # sparse_categorical_crossentropy
                  metrics=['accuracy',
                           tf.keras.metrics.Precision(),
                           # keras.metrics.Accuracy(),
                           # keras.metrics.Precision(),
                           tf.keras.metrics.Recall(),
                           # keras.metrics.F1Score(),
                           # keras.metrics.AUC(),
                           ])

    tensorflow_callback_early_stopping = EarlyStopping(monitor='val_loss', mode='min', verbose=0, patience=25)

    # checkpoint_file_path = model_history_path + '/' + "weights-improvement-{epoch:02d}-{val_accuracy:.2f}.hdf5"
    # tensorflow_callback_checkpoint = ModelCheckpoint(checkpoint_file_path, monitor='val_accuracy', verbose=1, save_best_only=True, mode='max')

    lr_scheduler = LearningRateScheduler(MlProcessModel.scheduler)

    callbacks = [lr_scheduler, tensorflow_callback_early_stopping]

    datagen = ImageDataGenerator(
        rotation_range=8,
        width_shift_range=0.08,
        height_shift_range=0.08,
        shear_range=0.3,
        zoom_range=0.08,
    )

    num_train_images = train_images.shape[0]
    num_val_images = val_images.shape[0]

    train_images = train_images[..., 0]
    val_images = val_images[..., 0]

    train_images = train_images / 255

    img_height = ml_process.dataset_serialized['image_height']
    img_width = ml_process.dataset_serialized['image_length']

    x_train = train_images[:num_train_images, :img_height, :img_width]
    y_train = train_labels[:num_train_images]

    val_images = val_images / 255

    x_test = val_images[:num_val_images, :img_height, :img_width]
    y_test = val_labels[:num_val_images]

    # Checkpoint
    x_train_fold = x_train.reshape(-1, img_height, img_width, 1)
    y_train_fold = y_train
    x_val_fold = x_test #.reshape(-1, img_height, img_width, 1)
    y_val_fold = y_test

    y_train_one_hot = to_categorical(y_train_fold, num_classes=2)
    y_val_one_hot = to_categorical(y_val_fold, num_classes=2)

    print(model.predict(x_test)[:10])

    history = model.fit(datagen.flow(x_train_fold, y_train_one_hot, batch_size=batch_size),
                        validation_data=(x_val_fold, y_val_one_hot), epochs=number_epochs, callbacks=lr_scheduler)

I have tried different loss function in model.compile so far and the results are as follow:

loss,accuracy,precision,recall,val_loss,val_accuracy,val_precision,val_recall,lr
0.9258964657783508,0.7076612710952759,0.7858796119689941,0.6844757795333862,0.6897426843643188,0.78951096534729,0.78951096534729,0.78951096534729,0.0055
0.7247691750526428,0.7893145084381104,0.7893145084381104,0.7893145084381104,0.6698439717292786,0.78951096534729,0.78951096534729,0.78951096534729,0.0055
0.6848592162132263,0.7893145084381104,0.7893145084381104,0.7893145084381104,0.6629940271377563,0.78951096534729,0.78951096534729,0.78951096534729,0.0033
0.7155748009681702,0.7893145084381104,0.7893145084381104,0.7893145084381104,0.6562079191207886,0.78951096534729,0.78951096534729,0.78951096534729,0.0033
0.6694061160087585,0.7893145084381104,0.7893145084381104,0.7893145084381104,0.6524480581283569,0.78951096534729,0.78951096534729,0.78951096534729,0.00198
0.6675164103507996,0.7893145084381104,0.7893145084381104,0.7893145084381104,0.6485042572021484,0.78951096534729,0.78951096534729,0.78951096534729,0.002
0.6571675539016724,0.7893145084381104,0.7893145084381104,0.7893145084381104,0.646232008934021,0.78951096534729,0.78951096534729,0.78951096534729,0.0012
0.659740149974823,0.7893145084381104,0.7893145084381104,0.7893145084381104,0.642375648021698,0.78951096534729,0.78951096534729,0.78951096534729,0.002
0.6671141982078552,0.7893145084381104,0.7893145084381104,0.7893145084381104,0.6400602459907532,0.78951096534729,0.78951096534729,0.78951096534729,0.0012
0.6524556279182434,0.7893145084381104,0.7893145084381104,0.7893145084381104,0.6363130807876587,0.78951096534729,0.78951096534729,0.78951096534729,0.002

P.S:

    def load_images_from_directory(directory_path, target_size):
        """
        Load images from a directory, where each subdirectory represents a class,
        and transform them into numpy arrays.

        Args:
            directory_path (str): Path to the directory containing image folders.
            target_size (tuple): Desired image size as a tuple (height, width).

        Returns:
            X (np.ndarray): Array of image data.
            y (np.ndarray): Array of corresponding class labels.
        """
        X = []
        y = []
        class_names = sorted(os.listdir(directory_path))
        class_indices = {class_name: index for index, class_name in enumerate(class_names)}

        for class_name in class_names:
            class_dir = os.path.join(directory_path, class_name)
            if os.path.isdir(class_dir):
                for img_name in os.listdir(class_dir):
                    img_path = os.path.join(class_dir, img_name)
                    try:
                        img = ImageReader.load_img(img_path, target_size=target_size)
                        img_array = tf.keras.utils.img_to_array(img)
                        X.append(img_array)
                        #print("class_indices[class_name]", int(class_indices[class_name]))
                        y.append(class_indices[class_name]) 
                    except Exception as e:
                        print(f"Error loading image {img_path}: {e}")

        X = np.array(X, dtype='float32')
        y = np.array(y, dtype='int32')

        return X, y

I can provide more sort of open parts of the project at request. Thanks in advance!

3

Hi @KerasWizardBeginner, Could you please confirm if my understanding about the issue is correct or not, you are getting the same precision and recall for the binary classification you are performing. Thank You.

4

You are exacly right precision == recall == accuracy that is the case. @Kiran_Sai_Ramineni

6

Hi @KerasWizardBeginner, There are few reasons for getting the same values for precision and recall.

The precision and recall are calculated using, precision= TP/TP+FP, recall=TP/TP+FN.

The accuracy is calculated using, accuracy= TN+TP/Total no,of predictions made.

If we observed the formula in the denominator only change is FP and FN

In your model prediction contains the same number of FalsePositives(FP) and FalseNegative(FN) then in that case we get the same values for precision and recall.

Please print the confusion matrix and see how your model is making the predictions.

Also try by using different thresholds. Thank You.

8

Thanks @Kiran_Sai_Ramineni .
For 5179 Images the confusion matrix of the last iteration of 2 iterations is the following:
[[ 0 877]
[ 0 3320]]

    y_pred = np.argmax(model.predict(x_test), axis=-1)
    cm = sklearn_confusion_matrix(y_test, y_pred)

    print("Confusion Matrix: ", cm)

What would you recommend to debug the lack of False Negatives or True Positives?

10

Hi @KerasWizardBeginner, Generally confusion matrix will be in the form of
[[TN, FP]
[FN, TP]]

you have got TN,FN as 0, and FP=877 and TP=3320. From this we can conclude that whatever the input to the model is given it is always predicting the positive only making the model more biased to the positive class.

This can be due to the sample in the training data containing more samples for positive class than negative class making the training data imbalance or the model architecture is not suitable for this dataset.

Also you can try by creating a greater threshold instead of taking max probability from the predictions made. Thank You.

11

I assume i might be getting a bit confused now.

What have been attempted so far:
Frist of all this is a CNN approach to identify anomalous comunnication in datasets such as KDD-CUP-99.

Using a balanced proportion of attacks to normal communications the problem mantains it self being precision == recall == accuracy. The confusion matrix produced is this one:
[[ 0 1768]
[ 0 1800]] for 4403 images.

Correct me if i am wrong here, there have been attempts to alter the threshold in model.predict(), but is not that used only to predictions after the model.fit() indicating that the metrics are already produced?

Is it possible to use softmax activation function with 2 neurons for a binary classification problem?

Thanks

13

Hi @KerasWizardBeginner, You can use softmax with 2 neurons. Thank You.