Convert data in dataframe - Python

Question

I have a dataframe as defined below (df) with daily frequency and I would like to convert this to minute frequency, starting at 7:30 and ending at 15:00.

import pandas as pd
dict = [
        {'ticker':'jpm','date': '2016-11-28','returns': '0.2'},
{ 'ticker':'ge','date': '2016-11-28','returns': '0.2'},
{'ticker':'fb', 'date': '2016-11-28','returns': '0.2'},
{'ticker':'aapl', 'date': '2016-11-28','returns': '0.2'},
{'ticker':'msft','date': '2016-11-28','returns': '0.2'},
{'ticker':'amzn','date': '2016-11-28','returns': '0.2'},
{'ticker':'jpm','date': '2016-11-29','returns': '0.2'},
{'ticker':'ge', 'date': '2016-11-29','returns': '0.2'},
{'ticker':'fb','date': '2016-11-29','returns': '0.2'},
{'ticker':'aapl','date': '2016-11-29','returns': '0.2'},
{'ticker':'msft','date': '2016-11-29','returns': '0.2'},
{'ticker':'amzn','date': '2016-11-29','returns': '0.2'}
]
df = pd.DataFrame(dict)
df['date']      = pd.to_datetime(df['date'])
df=df.set_index(['date','ticker'], drop=True)  

How do I do this?

Answer 1

Reshape dataframe using DataFrame.unstack for DatetimeIndex, then set minute frequency by DataFrame.asfreq, filter times by DataFrame.between_time and last use DataFrame.stack for MultiIndex. Use something like this:

df1 = df.unstack().asfreq('Min', method='ffill').between_time('8:30','16:00').stack()
print (df1.head(10))

                           returns
date                ticker       
2016-11-28 07:30:00 aapl       0.2
                    amzn       0.2
                    fb         0.2
                    ge         0.2
                    jpm        0.2
                    msft       0.2
2016-11-28 07:31:00 aapl       0.2
                    amzn       0.2
                    fb         0.2
                    ge         0.2