Customize structural pruning rate in each layer

TensorFlow General Discussion
, ,
1

Hi, is it possible to specify the pruning rate in each layer and ensure there is really no value instead of setting to 0.
for example, in the following easy model, the shape of weight of the first Conv2D is (3,3,1,32), I want to prune to (3,3,1,10) according to the L1 magnitude. Is it possible?

input_shape = (28, 28, 1)
num_classes = 10
model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Flatten(),
        layers.Dropout(0.5),
        layers.Dense(num_classes, activation="softmax"),
    ]
)

model.summary()

I also try to manually run layer.set_weights, but I was informed that

Layer conv2d weight shape (3, 3, 1, 32) is not compatible with provided weight shape (3, 3, 1, 10)

Thanks for any advice.

3

Are you looking for something like:

https://github.com/tensorflow/model-optimization/issues/732

5

yes. So it is still an to-be-done feature? Sad.

6

You could try to upvote and add a comment there for an update or in the case you want to contribute this feature to the ecosystem with a PR.

1 Like
8

Structural pruning for Conv2D is available now for the input-channel dimension. Sparse weights using structural pruning  |  TensorFlow Model Optimization

However, I think what you want is actual reduce of dimension in Conv2D weight, which is not a scope of sparsity toolkit of TF for now.