Lack of explanation of "Working with RNNs" might lead to misunderstading

TensorFlow General Discussion
1

Hi, i’m working with the RNNs with nested inputs, and found what I was looking for in here.

And thankfully, someone already made an example of that cell, NestedCell which shows how to build a custom RNN cell. However, I don’t think this cell precisely describes nested input version of GRU or LSTM.

If you see the call method part of this cell, it just simply updates the states by adding the output of the current inputs. And it actually confused me how the original version of GRU or LSTM works. (And as it is widely known, states should be updated with the reset gates/forget gates/input gates and so on)

So I am writing this post to get sure I am following along that this NestedCell is not clearly the one I can use for my nested input case. This example is a great one of blueprint of the nested case though.

Thanks for any comment in advance. Please let me know if there is any other ready-made layer for nested inputs case.

3

Hi @HyeonPhil_Youn, The code you mentioned serves as example for creating custom code for nested input/output. You have to make changes in the call methods to include gates as per your requirement. As far as I know I don’t think there is a ready-made layer for nested inputs. you have to write your custom code as per your requirement. Thank You.

5

TensorFlow 2.16 + Python 3.12 – JARaaS Hybrid RAG - 6/16/2024

You are correct in noting that the specific implementation of a custom RNN cell, like the NestedCell, might not provide a fully accurate depiction of how standard GRU or LSTM cells operate, especially in the context of nested inputs.

GRU and LSTM cells utilize sophisticated mechanisms involving gates (such as reset gates, forget gates, and input gates) to manage and update their internal states. These mechanisms help capture long-term dependencies and control the flow of information, which is pivotal in time series data and sequential learning tasks.

To work with nested inputs or build custom cells that more closely mimic the behavior of GRU or LSTM cells, you can refer to the following details extracted from your source documents:

  1. LSTM Layer Configuration:

    • Time Steps Processing: LSTM layers in Keras process sequences step-by-step and maintain an internal state through the return_sequences argument, which can be set to either True or False to return outputs for each input time step or just the final output, respectively.
    • Example:
      import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.LSTM(32, return_sequences=True, input_shape=(None, 1)),
    tf.keras.layers.LSTM(32)
])

    Reference: time_series.ipynb (internal document)

  2. Building Custom RNN Cells:

    • Feedback Model Implementation: You can construct customized RNN cells and models by defining their structure and updating mechanisms. This example demonstrates how a custom feedback model uses an LSTM cell wrapped within an RNN layer.
    • Example:
      class Feedback(tf.keras.Model):
    def __init__(self, units, out_steps):
        super().__init__()
        self.out_steps = out_steps
        self.units = units
        self.lstm_cell = tf.keras.layers.LSTMCell(units)
        self.lstm_rnn = tf.keras.layers.RNN(self.lstm_cell, return_state=True)
        self.dense = tf.keras.layers.Dense(num_features)

    def warmup(self, inputs):
        # Custom warmup method
        pass

    Reference: time_series.ipynb (internal document)

  3. Complex Model Topologies:

    • Functional API: For creating more intricate models including those with nested structure, the Keras Functional API allows the building of models with shared layers or non-sequential data flows. This approach might be beneficial when dealing with nested inputs.
    • Example:
      from tensorflow.keras import layers

inputs = tf.keras.Input(shape=(32,))
x = layers.Dense(64, activation='relu')(inputs)
x = layers.Dense(64, activation='relu')(x)
predictions = layers.Dense(10, activation='softmax')(x)
model = tf.keras.Model(inputs=inputs, outputs=predictions)

    Reference: keras.ipynb (internal document)

To directly address your specific use case of nested inputs and provide an accurate GRU or LSTM implementation, it is essential to deeply understand and potentially modify the cell mechanisms, ensuring correct updates through the gate mechanisms.

If you need a ready-made layer tailored for nested inputs or additional guidance on custom implementations, consider exploring the Keras documentation and tutorials on building custom RNN cells.

Sources:

  • Time Series Analysis: time_series.ipynb (internal document)
  • Keras Functional API Guide: keras.ipynb (internal document)

For further detailed inquiries or sample implementations, please review these internal documents.