# Issue with training a MLP with DenseFlipout layers

**URL:** <https://discuss.ai.google.dev/t/issue-with-training-a-mlp-with-denseflipout-layers/28730>\
**Category:** General Discussion\
**Tags:** tf-probability, help\_request\
**Created:** [November 18, 2022, 12:00am UTC](https://discuss.ai.google.dev/t/issue-with-training-a-mlp-with-denseflipout-layers/28730 "2022-11-18T00:00:35Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![P11](https://d1dlmcr85iqnpo.cloudfront.net/user_avatar/discuss.ai.google.dev/p11/32/10396_2.png) [@P11](https://discuss.ai.google.dev/u/P11)\
**Post date:** [November 18, 2022, 12:00am UTC](https://discuss.ai.google.dev/t/issue-with-training-a-mlp-with-denseflipout-layers/28730/1 "2022-11-18T00:00:35Z")

</div>

Hi, i would like for an advice regarding a problem that i am having with a mlp with DenseFlipout layers.  
I have been trying to train the neural network for a regression task and i am using denseFlipout layers with mostly the default settings

```auto
def create_flipout_bnn_model(train_size):
  def normal_sp(params): 
      return tfd.Normal(loc=params[:,0:1], scale=1e-3 + tf.math.softplus(0.05 * params[:,1:2]))

  kernel_divergence_fn=lambda q, p, _: tfp.distributions.kl_divergence(q, p) / (train_size)
  bias_divergence_fn=lambda q, p, _: tfp.distributions.kl_divergence(q, p) / (train_size)

  inputs = Input(shape=(1,),name="input layer")

  hidden = tfp.layers.DenseFlipout(50,
                           kernel_divergence_fn=kernel_divergence_fn,
                           activation="relu",name="DenseFlipout_layer_1")(inputs)
  hidden = tfp.layers.DenseFlipout(50,
                           kernel_divergence_fn=kernel_divergence_fn,
                           activation="relu",name="DenseFlipout_layer_2")(hidden)
  hidden = tfp.layers.DenseFlipout(50,
                           kernel_divergence_fn=kernel_divergence_fn,
                           activation="relu",name="DenseFlipout_layer_3")(hidden)                    
  params = tfp.layers.DenseFlipout(2,
                           kernel_divergence_fn=kernel_divergence_fn,
                           name="DenseFlipout_layer_5")(hidden)
  dist = tfp.layers.DistributionLambda(normal_sp,name = 'normal_sp')(params) 

  model = Model(inputs=inputs, outputs=dist)

 
  return model

 
flipout_BNN = create_flipout_bnn_model(train_size=train_size)
flipout_BNN.compile(optimizer=Adam(learning_rate=0.002 ),
                  loss=NLL,metrics= [tf.keras.metrics.RootMeanSquaredError()]
                 ) 
 
flipout_BNN.summary()

history_flipout_BNN = flipout_BNN.fit(X_train, y_train, epochs=50000, verbose=0, batch_size=batch_size,validation_data=(X_val,y_val) )

```

but the plot of the loss function keep showing spikes no matter the number of epochs. what can i do to avoid this issue? i think it’s related to the fact that the weights are sampled from a distribution but still…shouldnt those spikes disappear?

 ![image](https://d3qe71uytubmmx.cloudfront.net/original/2X/5/5fd031c9acb275559d54d0f402f2a5db6242068f.png)