Learning Rate Tuning techniques

TensorFlow General Discussion
,
1

Hi there
I have some issues with setting the learning rate for my model to predict time series data.

Problem (A): with the learning rate=1e-6, loss is higher and I have to reduce the prediction loss. If I tune the rate, loss result also changes again and get the desired result. But I am not sure how to find the best optimal learning rate for the SGD from the plot of the training result.

Problem (B): I got the prediction loss with learning rate 5e-6, it is a randomly chosen number and it worked. Nevertheless, it is not so clear how I can get 5e-6 from the curve . Please help me to clear my doubts.

This is the code how the training result curve was plotted

lrs = 1e-8 * (10 ** (np.arange(100) / 20))
plt.figure(figsize=(10, 6))

plt.grid(True)
plt.semilogx(lrs, history.history[...])
plt.tick_params(....')
plt.axis([1e-8, 1e-3, 0, 300])
3

Hi @SAHJAD_HOSSAIN,

I suggest taking this approach to find the optimal learning rate, start by trying different rates ranging from low to high. Track and plot the training loss for each rate. Identify where the loss decreases drastically. The optimal learning rate is generally just before the decreased sharp loss begins to rise again after this drastic decrease.

Here’s a sample gist for plotting and finding the optimal learning rate.

Could you please take a look at the plot to understand why 5e-6 worked and to confirm whether it might be the best choice,not a random number?

Thank You.