TFlite converter：why the performance get worse after calibration

TensorFlow General Discussion
1

Here is my converter code:

def representative_dataset_gen():
  for i in range(len(test_list)):
    if(i>300):
      break
    print(i)
    wavpath=test_path + test_list[i]
    x, sr = librosa.load(wavpath, sr=16000, mono=False)
    noisy = x[1, :]
    Complex_spec=WOLA_calculate(noisy).astype(np.float32)
    Complex_spec=Complex_spec.transpose(0,3,1,2)
    real=Complex_spec[:,0]
    imag=Complex_spec[:,1]
    spec_mag=np.sqrt(real ** 2 + imag ** 2)
    spec_mag=spec_mag[:,:,:5000]
    #print(X)
    yield [spec_mag]
converter = tf.lite.TFLiteConverter.from_keras_model(lstm_tf) # path to the SavedModel directory
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset =representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS,tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
tflite_quant_model = converter.convert()
# Save the model.
with open('model_5000_represent.tflite', 'wb') as f:
  f.write(tflite_quant_model)

I calcute the mask error between float model and converted int8 model,the mean error is 0.77,but i use dynamic converter,the mask error is 0.02,why calibration result get worse？

3

Hi @YinDi,

Upon observing the mask error between the two cases obviously the error(0.02) is low in case of dynamic quantization compared to int8 quantization. But in your case the int8 model has the mean error 0.77 which is relatively higher. The possible reasons for this case is either due to representative dataset or the model complexity. Can you try to resolve your issue by using AI-Edge-Torch, you can find more information here: googleblog. Please let us know your response.

Thank You