Discrepancy in KL Divergence Calculation Using `tf.keras.metrics.KLDivergence'

Keras
1

Hi,

I’ve encountered a discrepancy when calculating KL Divergence using TensorFlow’s tf.keras.metrics.KLDivergence metric compared to a manual calculation.

Here’s a summary of what I’ve observed:

  1. Manual Calculation:

    import tensorflow as tf
import numpy as np

true_probs = np.array([0.7])
predicted_probs = np.array([0.5])

true_probs_tf = tf.convert_to_tensor(true_probs, dtype=tf.float32)
predicted_probs_tf = tf.convert_to_tensor(predicted_probs, dtype=tf.float32)

epsilon = 1e-10
p = tf.clip_by_value(true_probs_tf, epsilon, 1.0 - epsilon)
q = tf.clip_by_value(predicted_probs_tf, epsilon, 1.0 - epsilon)

kl_divergence_manual = tf.reduce_sum(p * tf.math.log(p / q) + (1 - p) * tf.math.log((1 - p) / (1 - q)))
print("KL Divergence (Manual TensorFlow calculation with stability):", kl_divergence_manual.numpy())

    Results in: 0.082282856

  2. TensorFlow Metric:

    kl_divergence_metric = tf.keras.metrics.KLDivergence()
kl_divergence_metric.update_state(true_probs_tf, predicted_probs_tf)
kl_divergence = kl_divergence_metric.result().numpy()
print("KL Divergence (TensorFlow metric):", kl_divergence)

    Results in: 0.23553054

The manual calculation differs from TensorFlow’s metric result. Could you help me understand why this discrepancy occurs? Is there a special handling or implementation detail in TensorFlow’s KLDivergence metric?

Thank you!

3

Hi @Chashi_Xianke,

Welcome to the Tensorflow forum!
It appears that the discrepancy is due to difference between the formula used in your manual calculation and the one implemented in TensorFlow’s KL Divergence metric.

The formula in the TensorFlow metric is:

metric = y_true * log(y_true / y_pred)

For more information, I kindly suggest referring to the TensorFlow documentation on KL Divergence metrics. Additionally, I’ve included a sample gist implementation for your reference.
Thank you.