Input shape problem

Course Name: Deep Reinforcement Learning in Trading, Section No: 22, Unit No: 6, Unit type: ZipFiles



I am trying to imporove my capstone projetct using a different data granularity. I wish to use data with the following granularity:

5 min

1 hr

12 hr 

6 days

In this way all granularities are multiplier of 12.

But when I run the complete algo after few trades I got one error about the input shape for the ANN see the following:

 Trade 001 | pos 1 | len 3 | approx cum ret -0.04% | trade ret -0.04% | eps 1.0010 | 2015-01-28 12:50:00 | 5004

Trade 002 | pos -1 | len 33 | approx cum ret -0.40% | trade ret -0.37% | eps 0.0635 | 2015-01-28 15:40:00 | 5038

Trade 003 | pos -1 | len 32 | approx cum ret -0.53% | trade ret -0.12% | eps 0.0133 | 2015-01-28 18:25:00 | 5071

Trade 004 | pos 1 | len 331 | approx cum ret 1.74% | trade ret 2.27% | eps 0.0049 | 2015-01-29 22:05:00 | 5403

Trade 005 | pos -1 | len 159 | approx cum ret 2.12% | trade ret 0.38% | eps 0.0026 | 2015-01-30 11:25:00 | 5563

Traceback (most recent call last):



  File "C:\Users\guido\QUANTRA\CAPSTONE\Capstone_Project\guido_best_1.py", line 688, in <module>

    run(bars5m, rl_config)



  File "C:\Users\guido\QUANTRA\CAPSTONE\Capstone_Project\guido_best_1.py", line 590, in run

    q = q_network.predict(state_t)



  File "C:\Users\guido\anaconda3\envs\quantra\lib\site-packages\keras\engine\training.py", line 1149, in predict

    x, _, _ = self._standardize_user_data(x)



  File "C:\Users\guido\anaconda3\envs\quantra\lib\site-packages\keras\engine\training.py", line 751, in _standardize_user_data

    exception_prefix='input')



  File "C:\Users\guido\anaconda3\envs\quantra\lib\site-packages\keras\engine\training_utils.py", line 138, in standardize_input_data

    str(data_shape))



ValueError: Error when checking input: expected dense_1_input to have shape (171,) but got array with shape (175,)



It looks that the input is expecting an array with less 4 dimensions. 4 dimensions are exactly the new granularity I added.

I have checked all the algo to see if I forgot to update any parameter but i did not find any error.

Do you have any suggestions?

Thank you

best regards

Guido



 

Hi Guido,



I will be glad to try and solve this problem. Can you please share your current notebook file (plus the supporting .py file if any) and the data you are working with at quantra@quantinsti.com



It is difficult to debug the shape mismatch errors in the absence of the codes.



Thanks!

Hi Guido!



We have received the code and data file and are working to resolve your query.



Thanks!

Hello Guido!



Your query has been resolved and you will be receiving the response mail shortly. For the benefit of other users, I will share the fix which was performed.



You had changed the assembly of states to something different, which is totally fine, but when the inclusion of 6 days candle was done, the problem started there. The main issue is that the data for 5 min bars is resampled to get the lookback period equivalent of 6-day candles. This needs to be ensured by taking a buffer of the start index. Similar to the explanation given here in a previous post.



The start index of 5000 was unable to accommodate this, and hence the shape error was encountered.



The start index was increased to 12000 and the code ran smoothly.



Another suggestion was made in the _get_last_N_timebars function where you can use the following:

        wdw5m = 9

        wdw1h = np.ceil(self.lkbk15.0/24)

        wdw12h = np.ceil(self.lkbk
15.0/2)

        wdw6d = np.ceil(self.lkbk15.06)



Please note that these are not fine-tuned, but take an arbitrarily larger data set so that the time delta shift and resampling can still give the complete data for the different time resolutions you have selected.



Hope this helps!



Thanks!