Problem with stateful LSTM and static batch size

TensorFlow General Discussion
,
1

Hi,
I want to use a simple LSTM model with stateful=True. Therefore I use

import numpy as np
import tensorflow as tf

window_length = 3
batch_size = 1
number_time_series = 1

timeseries_1 = np.arange(150).reshape(-1, 1)
# timeseries_2 = np.arange(150).reshape(-1, 1)

X = tf.keras.utils.timeseries_dataset_from_array(
    timeseries_1,
    # np.concatenate([timeseries_1, timeseries_2], axis=1),
    targets=timeseries_1[window_length:],
    sequence_stride=window_length,
    sequence_length=window_length,
    batch_size=batch_size
)

# list(X)

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.InputLayer(batch_input_shape=(batch_size, window_length, number_time_series)))
model.add(tf.keras.layers.LSTM(units=8, stateful=True))
model.add(tf.keras.layers.Dense(units=1))
model.compile(loss=tf.keras.losses.Huber(), optimizer="adam", metrics=["mae"])


class ResetStatesCallback(tf.keras.callbacks.Callback):
    def on_epoch_begin(self, epoch, logs):
        for layer in self.model.layers:
            if hasattr(layer, "reset_states"):
                layer.reset_states()


model.fit(X, epochs=3, batch_size=batch_size, callbacks=[ResetStatesCallback()], shuffle=False)

But when I run this I get

Input tensor `sequential_1/lstm_1/ReadVariableOp:0` enters the loop with shape (1, 8), but has shape (None, 8) after one iteration. To allow the shape to vary across iterations, use the `shape_invariants` argument of tf.while_loop to specify a less-specific shape.

Arguments received by LSTM.call():
  • sequences=tf.Tensor(shape=(None, None, 1), dtype=float32)
  • initial_state=None
  • mask=None
  • training=True

In some tutorials, e.g.
https://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/
I can find something like

model.add(tf.keras.layers.LSTM(units=8, batch_input_shape=(batch_size, window_length, number_time_series), stateful=True))

I also think that it worked one year ago with this solution, but it seems that there is no argument batch_inpupt_shape in LSTM

ValueError: Unrecognized keyword arguments passed to LSTM: {'batch_input_shape': (1, 3, 1)}

Can you please give me a hint? Thank you very much!
Arno

1 Like
2

I think that the problem is that X has the shape element_spec=(TensorSpec(shape=(None, None, 1).
I tried

def enforce_shape(x, y):
    x = tf.ensure_shape(x, [batch_size, window_length, number_time_series])  # Set the shape for inputs
    y = tf.ensure_shape(y, [batch_size, 1])  # Set the shape for targets
    return x, y

# Apply the shape enforcement using map
X = X.map(enforce_shape)

and it the model is fitted without errors. Is there a “nicer” way to solve the issue?
Thanks, Arno

1 Like
4

Hi @Arno_Kimeswenger, The other way is adding preprocessing layers in the model architecture. But the purpose of both will be the same one is preprocessing the data before passing to the model and the other will be preprocessing the data after passing to the model. Thank You.

1 Like
6

Thank you very much!

1 Like