Discrepancy in KL Divergence Calculation Using `tf.keras.metrics.KLDivergence'

Keras
1

Hi,

I’ve encountered a discrepancy when calculating KL Divergence using TensorFlow’s tf.keras.metrics.KLDivergence metric compared to a manual calculation.

Here’s a summary of what I’ve observed:

  1. Manual Calculation:

    import tensorflow as tf
import numpy as np

true_probs = np.array([0.7])
predicted_probs = np.array([0.5])

true_probs_tf = tf.convert_to_tensor(true_probs, dtype=tf.float32)
predicted_probs_tf = tf.convert_to_tensor(predicted_probs, dtype=tf.float32)

epsilon = 1e-10
p = tf.clip_by_value(true_probs_tf, epsilon, 1.0 - epsilon)
q = tf.clip_by_value(predicted_probs_tf, epsilon, 1.0 - epsilon)

kl_divergence_manual = tf.reduce_sum(p * tf.math.log(p / q) + (1 - p) * tf.math.log((1 - p) / (1 - q)))
print("KL Divergence (Manual TensorFlow calculation with stability):", kl_divergence_manual.numpy())

    Results in: 0.082282856

  2. TensorFlow Metric:

    kl_divergence_metric = tf.keras.metrics.KLDivergence()
kl_divergence_metric.update_state(true_probs_tf, predicted_probs_tf)
kl_divergence = kl_divergence_metric.result().numpy()
print("KL Divergence (TensorFlow metric):", kl_divergence)

    Results in: 0.23553054

The manual calculation differs from TensorFlow’s metric result. Could you help me understand why this discrepancy occurs? Is there a special handling or implementation detail in TensorFlow’s KLDivergence metric?

Thank you!