Problems when reshaping model.predict() to single vector (Neural Networks course)

AI & Machine learning
1

Hi,



I am using your code from the RNN notebook to train a model on a dataset which, after reshaping to 3D to fit into Keras, has the following shape:

 

X_train shape is:  (2650, 20, 21)
y_train shape is:  (2650,)
X_test shape is:  (667, 20, 21)
y_test shape is:  (667,)

Thus, I am using 20 time steps with 21 features.

The problem is that when I perform the prediction I am not able to reshape 
the predicted values to match the y_test set:

pred = model.predict(X_test)
print('pred shape is: ', pred.shape)
pred shape is:  (667, 20, 1)

Then, if I reshape, as suggested in your notebook, I get:



predict_close = []
predict_close = np.reshape(model.predict(X_test), (1, -1))
predict_close = predict_close[0]
print('Reshaped pred shape is: ', predict_close.shape)
Reshaped pred shape is:  (13340,)

It multiplies 667 times 20.

What should I do to get the correct shape?

Thanks for your help!

Luca
2

Hi Luca,



It would be great if you could please share the relevant piece of code so that we can replicate the exact issue.



Thanks,

Akshay

3

Hi Akshay,



I have uploaded the whole notebook here: 



http://github.com/babyloncapital/python_projects/blob/a74391ce61ef93f5fd85620d52222d9c2d30c0c4/Notebook%20LSTM%20-%20exercise.ipynb



It's a little bit long, but you can only focus on the section "Deep Learning Model" (the last one).



I have also uploaded the final data set used here:



http://github.com/babyloncapital/python_projects/blob/a74391ce61ef93f5fd85620d52222d9c2d30c0c4/LSTM%20DataFrame.xlsx



This is because I have previously stored the features and target datasets in my local database. This way, it should be easy for you to replicate my code on your local machine. 



I am especially concerned about how I pre-processed features and target for the LSTM model, as here I am trying to predict returns 60 days in the future. Thus, I might have messed up with reshaping or scaling. Please, also note that I want to scale the whole features set based ONLY on the training set (as shown in the first part of the course with the classification model).



The model predicts very poorly, but I would like to know if it is due to mis-preprocessing or if it is the model per-se that should be tuned more.



Many thanks for your kind support here.



Luca

 

4

Hi Luca,



Thank you very much for sharing the code. We will go through it once and get back to you.



Thanks,

Akshay

5

Hi Akshay,



any news on the above?



Thanks!



Luca

6

Hi Luca,



From the code shared above, I can see that your first query is resolved, as the shapes are now in the proper order. However, if there is any specific reshaping issue, please feel free to mark the code cells so we can have a look at it. 



For the second query on model performance, you can try applying the model to different assets, check the vanishing/exploding gradient problem and apply the model with varying sets of features. You can have a look at this to learn more about the vanishing/exploding gradient problem as this is a common problem in machine learning.



Also, you can try using a different ML model, as a simpler model generally works well in the financial markets.



Hope this helps!



Thanks,

Akshay