LSTM model only predict 1 constant value

TensorFlow General Discussion
1

Hi everyone
I am facing an issue with a model. Seem that the model itself have poor performances and need tune but it only predict single values.

Warning: Test predictions are constant. Model might not be learning effectively.
Warning: Validation predictions are constant. Model might not be learning effectively.

Sample Test Predictions: [0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502]
Sample Validation Predictions: [0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502 0.05704502 0.05704502 0.05704502 0.05704502
0.05704502 0.05704502]

def create_model(params):

#--------------------------GPU Setup--------------------------------------
# Calculate the global batch size and adjust the learning rate accordingly
#-------------------------------------------------------------------------
strategy = tf.distribute.MirroredStrategy()
num_gpus = strategy.num_replicas_in_sync  # Number of GPUs available
global_batch_size = params['batch_size'] * num_gpus
# params['learning_rate'] *= num_gpus  # Scale the learning rate

input_shape = (params['timesteps'], params['n_features'])
print('Number of devices: {}'.format(strategy.num_replicas_in_sync))

with strategy.scope():
#--------------------------GPU Setup--------------------------------------
# Calculate the global batch size and adjust the learning rate accordingly
#-------------------------------------------------------------------------
    
    # Regularization parameters
    l1_reg = 0.001  # L1 regularization factor 0.005 0.001 0.01 0.02
    l2_reg = 0.001  # L2 regularization factor

    model = Sequential()

    # First LSTM layer with more units
    model.add(LSTM(128, return_sequences=True, input_shape=input_shape,
                   kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg)))
    model.add(Dropout(0.2))

    # Second LSTM layer
    model.add(LSTM(64, return_sequences=True,
                   kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg)))
    model.add(Dropout(0.2))

    # Bidirectional LSTM layer
    model.add(Bidirectional(LSTM(32, return_sequences=True,
                                 kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg))))
    model.add(Dropout(0.2))

    # Dense layer
    model.add(Dense(1, kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg)))

==========================================================================

Model Structure Definition

==========================================================================

==========================================================================

Model Compile

==========================================================================

    optimizer = tf.keras.optimizers.Adam(learning_rate=params["learning_rate"], clipvalue=0.5)

    model.compile(optimizer=optimizer, 
                loss='mean_squared_error',
                metrics=[tf.keras.metrics.MeanAbsoluteError(), 
                        tf.keras.metrics.RootMeanSquaredError(), 
                        tf.keras.metrics.MeanAbsolutePercentageError(), 
                        r_squared, 'accuracy'])
                        # Add your custom metrics as needed

return model

==========================================================================

Model Compile

==========================================================================

Creating the model using hyperparameters

model = create_model(params)
model.summary()

U see something wrong in the model definition ?

Tnks !

3

Hi @Igor_Lessio,

As mentioned in the warning, the constant validation and testing predictions mainly arises from
the data you are using. Please check the target variables are correctly labelled or not. Make sure all the preprocessing done properly. Second thing, in the dense layer model.add(Dense(1, kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg))) , no activation function is specified. If your use case is for regression task use activation = 'linear': in dense layer as model.add(Dense(1, activation='kernel', kernel_regularizer=l1_l2(l1=l1_reg, l2=l2_reg))). Please try to do these changes. For further assistance please share your reproducible code .

Thank you