How to initialize the weights

TensorFlow
1

Hello experts
I want to know the better learning rate for optimizer. So I repeat ‘restart kernel’ and ‘run all cells’, and see the output loss or accuracy, but this is a boring work because data loading and the 1epoch takes a lot of times

How can I initialize the program without kernel restart?
My program style is something like

2

Structure your code into separate functions for data loading, model creation, and training. Create a main function that accepts a list of learning rates, loads the data once, and then loops through the rates. In each iteration, reinitialize the model and optimizer with the current learning rate, train for one epoch, and record the results. This approach allows you to test multiple learning rates in a single run, reusing the loaded data and only reinitializing the model and optimizer for each test, and it reduces the time spent on repetitive data loading and kernel restarts.
Here’s an example:

import tensorflow as tf
import numpy as np

def load_data():
    # Simulating data loading
    x = np.random.random((1000, 10))
    y = np.random.randint(2, size=(1000, 1))
    return x, y

def create_model():
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(64, activation='relu', input_shape=(10,)),
        tf.keras.layers.Dense(32, activation='relu'),
        tf.keras.layers.Dense(1, activation='sigmoid')
    ])
    return model

def train_model(model, data, learning_rate, epochs=1):
    x, y = data
    model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),
                  loss='binary_crossentropy',
                  metrics=['accuracy'])
    history = model.fit(x, y, epochs=epochs, validation_split=0.2, verbose=0)
    return history.history['val_accuracy'][-1]

def main(learning_rates):
    print("Loading data...")
    data = load_data()
    
    for lr in learning_rates:
        print(f"\nTesting learning rate: {lr}")
        model = create_model()
        accuracy = train_model(model, data, lr)
        print(f"Validation accuracy: {accuracy:.4f}")

# Test different learning rates
learning_rates = [1e-2, 1e-3, 1e-4]
main(learning_rates)
4

Thank you for your advice, ehsa_293.
Now I have a summer vacation so reply to you in the next week.
Yuichiro