# Pruning pre-trained model with model-optimization

**URL:** <https://discuss.ai.google.dev/t/pruning-pre-trained-model-with-model-optimization/25925>\
**Category:** General Discussion\
**Tags:** model\_optimization\
**Created:** [August 9, 2023, 3:38am UTC](https://discuss.ai.google.dev/t/pruning-pre-trained-model-with-model-optimization/25925 "2023-08-09T03:38:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![stepan\_zalis](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/stepan_zalis/32/7354_2.png) [@stepan\_zalis](https://discuss.ai.google.dev/u/stepan_zalis)\
**Post date:** [August 9, 2023, 3:38am UTC](https://discuss.ai.google.dev/t/pruning-pre-trained-model-with-model-optimization/25925/1 "2023-08-09T03:38:19Z")

</div>

Hey! I have a model based on MobileNetV2 with added classification head. I followed a tutorial how to prune model, but obviously, when I pruned model, only classification head is pruned and which is not what I need. Is it possible to prune base\_model and then add it to my model?

Base model:

```auto
base_model = tf.keras.applications.MobileNetV2(
  input_shape=INPUT_IMG_SHAPE,
  include_top=False,
  weights='imagenet',
  pooling='avg'
)

```

Actual model:

```auto
model = tf.keras.models.Sequential()
model.add(base_model)
model.add(tf.keras.layers.Dropout(0.5))
model.add(tf.keras.layers.Dense(
    units=NUM_CLASSES,
    activation=tf.keras.activations.softmax,
    kernel_regularizer=tf.keras.regularizers.l2(l=0.01)
))

```

---

<div class="post-metadata">

**Author:** ![Kiran\_Sai\_Ramineni](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/kiran_sai_ramineni/32/8859_2.png) [@Kiran\_Sai\_Ramineni](https://discuss.ai.google.dev/u/Kiran_Sai_Ramineni)\
**Post date:** [August 9, 2023, 9:40am UTC](https://discuss.ai.google.dev/t/pruning-pre-trained-model-with-model-optimization/25925/3 "2023-08-09T09:40:49Z")

</div>

Hi @stepan_zalis, once you have defined your base model

```auto
base_model = tf.keras.applications.MobileNetV2(
  input_shape=(220,220,3),
  include_top=False,
  weights='imagenet',
  pooling='avg'
)

```

you can prune these model using

```auto
import tensorflow_model_optimization as tfmot
prune_low_magnitude = tfmot.sparsity.keras.prune_low_magnitude
prune_base_model=prune_low_magnitude(base_model)

```

now can pass these prune model to your model

```auto
model = tf.keras.models.Sequential()
model.add(prune_base_model)
model.add(tf.keras.layers.Dropout(0.5))
model.add(tf.keras.layers.Dense(
    units=10,
    activation=tf.keras.activations.softmax,
    kernel_regularizer=tf.keras.regularizers.l2(l=0.01)
))

```

Thank You!

---

<div class="post-metadata">

**Author:** ![stepan\_zalis](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/stepan_zalis/32/7354_2.png) [@stepan\_zalis](https://discuss.ai.google.dev/u/stepan_zalis)\
**Post date:** [August 9, 2023, 11:43am UTC](https://discuss.ai.google.dev/t/pruning-pre-trained-model-with-model-optimization/25925/4 "2023-08-09T11:43:41Z")

</div>

Thank you, it was exactly what I tried without any success. I have this now:

```auto
pruned_base_model = create_pruning_model(base_model)
compile_model(pruned_base_model)
model = create_model(pruned_base_model)

```

where `create_model` is creating the model as second snippet showing and `create_pruning_model(_)` is this:

```auto
def create_pruning_model(base_model):
    prune_low_magnitude = tfmot.sparsity.keras.prune_low_magnitude
    pruning_params = {
             'pruning_schedule': tfmot.sparsity.keras.ConstantSparsity(0.5, 0),
             'block_size': (1, 1),
             'block_pooling_type': 'AVG'           
    }

    model_for_pruning = prune_low_magnitude(base_model, **pruning_params)
    return model_for_pruning

```

However, when I compile and train the model, gzipped file is almost the same as the non pruned model around 8 MB. Not sure what I’m doing wrong.

---

<div class="post-metadata">

**Author:** ![Kiran\_Sai\_Ramineni](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/kiran_sai_ramineni/32/8859_2.png) [@Kiran\_Sai\_Ramineni](https://discuss.ai.google.dev/u/Kiran_Sai_Ramineni)\
**Post date:** [August 18, 2023, 5:53am UTC](https://discuss.ai.google.dev/t/pruning-pre-trained-model-with-model-optimization/25925/6 "2023-08-18T05:53:52Z")

</div>

Hi @stepan_zalis, The pruning function wraps a tf.keras model or layer with pruning functionality which sparsifies the layer’s weights during training. so the model size will reduce after the training the prune model. For example, I have a model

```auto
# Load MNIST dataset
mnist = keras.datasets.mnist
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

# Normalize the input image so that each pixel value is between 0 and 1.
train_images = train_images / 255.0
test_images = test_images / 255.0

# Define the model architecture.
model = keras.Sequential([
  keras.layers.InputLayer(input_shape=(28, 28)),
  keras.layers.Reshape(target_shape=(28, 28, 1)),
  keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation='relu'),
  keras.layers.MaxPooling2D(pool_size=(2, 2)),
  keras.layers.Flatten(),
  keras.layers.Dense(10)
])

# Train the digit classification model
model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

model.fit(train_images, train_labels, epochs=4, validation_split=0.1,)

```

and save the model

```auto
_, keras_file = tempfile.mkstemp('.h5')
tf.keras.models.save_model(model, keras_file, include_optimizer=False)
print('Saved baseline model to:', keras_file)

```

After training and saving i have pruned the model

```auto
import tensorflow_model_optimization as tfmot

prune_low_magnitude = tfmot.sparsity.keras.prune_low_magnitude
batch_size = 128
epochs = 2
validation_split = 0.1
num_images = train_images.shape[0] * (1 - validation_split)
end_step = np.ceil(num_images / batch_size).astype(np.int32) * epochs

pruning_params = {
      'pruning_schedule': tfmot.sparsity.keras.PolynomialDecay(initial_sparsity=0.50,
                                                               final_sparsity=0.80,
                                                               begin_step=0,
                                                               end_step=end_step)
}

model_for_pruning = prune_low_magnitude(model, **pruning_params)

# `prune_low_magnitude` requires a recompile.
model_for_pruning.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

```

and save the prune model

```auto
model_for_export = tfmot.sparsity.keras.strip_pruning(model_for_pruning)

_, pruned_keras_file = tempfile.mkstemp('.h5')
tf.keras.models.save_model(model_for_export, pruned_keras_file, include_optimizer=False)
print('Saved pruned Keras model to:', pruned_keras_file)

```

Now lets see the size of trained model and prune non trained model

```auto
Size of gzipped baseline Keras model: 78146.00 bytes
Size of gzipped pruned Keras model: 78146.00 bytes

```

You can see that the size of the trained model and prune non trained model are of some size.

Now lets train the pruned model

```auto
logdir = tempfile.mkdtemp()

callbacks = [
  tfmot.sparsity.keras.UpdatePruningStep(),
  tfmot.sparsity.keras.PruningSummaries(log_dir=logdir),
]
  
model_for_pruning.fit(train_images, train_labels,
                  batch_size=batch_size, epochs=epochs, validation_split=validation_split,
                  callbacks=callbacks)

```

and save the pruned trained model

```auto
model_for_export = tfmot.sparsity.keras.strip_pruning(model_for_pruning)

_, pruned_keras_file = tempfile.mkstemp('.h5')
tf.keras.models.save_model(model_for_export, pruned_keras_file, include_optimizer=False)
print('Saved pruned Keras model to:', pruned_keras_file)

```

Now lets see the size of trained model and prune trained model

```auto
Size of gzipped baseline Keras model: 78146.00 bytes
Size of gzipped pruned Keras model: 25820.00 bytes

```

You can see the size difference between the trained model and prune trained model.

Please refer to this [gist](https://gist.github.com/kiransair/8ee8253ba447e111f041552135673518) for working code example. Thank You!
