Is DNN strategy reliable

General
1

Course Name: Neural Networks in Trading, Section No: 4, Unit No: 11, Unit type: Notebook

First, I want to point out a potential error in the code, which is
y = np.where(data.Close > data.Close.shift(5), 1, 0) should be
y=np.where(data.Close.shift(-5) > data.Close,1,0), the revised code is the true meaning of the description ‘the target variable will attempt to predict if the close price has increased or decreased after 5 trading days’.

I ran the code locally. I found that I got different results in different trials. Sometimes, the model is better, sometimes it is worse than the market return. I argue that the DNN is quite sensitive to the weights and not a reliable strategy.

2

Thanks for pointing this out, you’re right.

The line should be y = np.where(data.Close.shift(-5) > data.Close, 1, 0). The version in the notebook checks today’s price against the price from 5 days ago (looking backward), but the idea was to check the price 5 days ahead (looking forward). Your fix matches what the notebook is trying to teach. We’ve raised this internally and are getting it corrected.

On the changing results: part of it is that the notebook sets a random seed for NumPy but not for TensorFlow, so the model’s starting weights and dropout pattern still change every run. Adding this line near the top would make single runs repeatable:

python

import tensorflow as tf
tf.random.set_seed(42)

Beyond that, some variation is expected. This model is trained on a small amount of data and tested on only 200 days, so different runs can land on different results just by chance. That’s a real limitation of using one DNN run this way, and it’s a fair thing to notice. This is actually covered next in Section 7 (Cross Validation in Keras), which deals with exactly this kind of run-to-run variation through cross-validation and hyperparameter tuning, worth going through if you haven’t yet. A more reliable check would be running it several times (with seeds fixed) and averaging the results, or testing on a few different stocks or time periods, before drawing a conclusion either way

1 Like
3

Hi Sanya,

Thank you for your reply. It is helpful and constructive. After learning Section 7 (CV in Keras), I am impressed by the effectiveness and usefulness of hyperparameter tuning and cv in DNN. I also tried the model on other stocks, and the results look good to me.

I am also impressed but also worried about the Sharpe ratio in section 7. In the code, the Sharpe ratio can go up to over 10, which is so high that I have to doubt whether it is real. Maybe its bez it did not consider transaction fees? Could you please let me know how to understand this high Sharpe ratio? Thank you so much.

4

Good catch, that instinct to question a Sharpe above 10 is the right one.

The formula in this notebook is:

python

Sharpe = (np.nansum(strategy_returns) - intrest_rate) / strategy_returns.std()

It sums daily returns instead of averaging them, and never annualizes (no sqrt(252) scaling), so the numerator and denominator are on mismatched time scales.

The correct version, used in Section 4, is:

python

def annualized_sharpe_ratio(returns, N=252):
    return np.sqrt(N) * returns.mean() / returns.std()

Working back from the number this notebook printed, the corrected Sharpe should land somewhere around 0.8, a solid but realistic figure, rather than 5+. If you swap in the corrected formula and rerun it, would love to know what exact number you get. We’ve flagged this for a fix on our end too. Thanks for catching it.