Course Name: Python For Trading!, Section No: 7, Unit No: 4, Unit type: Notebook
In Python, we can use the reindex method to detect missing data for the specific frequency. The NaN values can be filled using the previous values or by any other method.
Can someone please explain this with words?:
df.reindex(pd.date_range(start=df.index[0], end=df.index[-1], freq='1min'))
Hello Alvaro,
It's good to know you are making progress in such a great programming language.
Let's breakdown the code in two parts.
"""
pd.date_range(start=df.index[0], end=df.index[-1], freq='1min')
"""
This code creates a datetime series starting from date df.index[0] and ending at df.index[-1]. Frequency means that each datetime created within this range will be the sum of the its previous datetime and one minute.
For example, if we create the following code:
"""
pd.date_range(start='2018-01-01', end='2018-01-08', freq='D')
"""
Then the datetime series will be:
DatetimeIndex(['2018-01-01', '2018-01-02', '2018-01-03', '2018-01-04',
'2018-01-05', '2018-01-06', '2018-01-07', '2018-01-08'],
dtype='datetime64[ns]', freq='D')
In this case the frequency is daily, thus, the difference between each datetime is one day.
Another example
"""
pd.date_range(start='2018-01-01 12:00:00', end='2018-01-01 12:05:00', freq='1min')
"""
The result is:
DatetimeIndex(['2018-01-01 12:00:00', '2018-01-01 12:01:00',
'2018-01-01 12:02:00', '2018-01-01 12:03:00',
'2018-01-01 12:04:00', '2018-01-01 12:05:00'],
dtype='datetime64[ns]', freq='T')
As you can see the difference between two dates is one minute.
So the code "pd.date_range(start=df.index[0], end=df.index[-1], freq='1min')",
creates a datetime series which contains all the datetimes which must be between df.index[0] and df.index[-1] with a minute frequency.
Now, let's come back to the df.reindex(...) code.
1. df.reindex will create a new dataframe based on the complete datetime series created from pd.date_range(..)
2. The datetime series created with pd.date_range is more complete than the datetimes located in the previous df.index
3. Thus, since the old dataframe "df" doesn't have some datetime values that the pd.date_range has, then the new dataframe will contain rows with NaN values for all the "df" columns.
4. These rows with NaN values will belong to datetimes which the previous df.index doesn't have but the new pd.date_range has.
The previous df.index doesn't have the following datetimes: "2018-07-23 09:19:00" and "2018-07-23 09:22:00".
The pd.date_range does have these datetimes.
I hope that helps to explain what the code does.
In case you have more doubts, please write us back.
Thanks and regards,
Jose Carlos