Hey guys, I have two questions about LSTM after the course.
When I looked at the code for the LSTM model, I was thinking about how the weights of LSTM cell are updated. I noticed that there are 5 dense layers after the LSTM cell with 200 ~ 1000 neurons and dropout ratio 0.8. By the chain rule, in back prop when the gradient reaches the LSTM cell, only roughly 0.03% of paths can survive. This means that the weight in LSTM cell maynot be changed much compared to the weights in dense layers. Is my understanding correct?
I found in the LSTM strategy, the role of LSTM might be a fancy replacement of the mean and variance of the past data, like data.rolling(20). I tested some cases and found LSTM may not deserve its complexity compared to the trivial mean strategy (compared with Sharpe ratio). I might not use LSTM strategy in a right way, so my question is how to manifest the advantages of using LSTM.
Your math is right, 0.2^5 ≈ 0.03% is the chance one specific path survives all 5 dropout layers. But that’s not the same as the gradient reaching the LSTM. Each layer has hundreds of neurons, so many paths survive together, and Keras rescales the kept neurons to make up for the dropped ones. So the LSTM isn’t stuck barely learning because of dropout. A better reason to suspect the dense layers dominate: the LSTM here has only 4 units, next to 200 to 1000 in the dense layers after it. That size gap is more likely why the dense layers do most of the work.
This matches what’s often seen in practice: LSTMs don’t always beat simple baselines on short-term, single-stock price prediction, since daily returns are noisy and this model is trained on just one stock with limited data. The strategy also trades the gap between predicted and actual price using rolling bands, which is already similar in spirit to a moving-average based approach. So your result makes sense. LSTMs tend to help more with richer data, like multiple stocks trained together or longer sequences, not necessarily one ticker’s daily prices. Worth trying the same comparison on a few more stocks before concluding either way.