How does time series analysis with statsmodels work

0 votes

I'm trying to do multiple regression with time series data, but when I add the time series column to my model, it ends up treating each unique value as a separate variable, like so (my 'date' column is of type datetime):

est = smf.ols(formula='r ~ spend + date', data=df).fit()
print est.summary()

coef    std err t   P>|t|   [95.0% Conf. Int.]
Intercept   -6.249e-10  inf -0  nan nan nan
date[T.Timestamp('2014-10-08 00:00:00')]    -2.571e-10  inf -0  nan nan nan
date[T.Timestamp('2014-10-15 00:00:00')]    9.441e-11   inf 0   nan nan nan
date[T.Timestamp('2014-10-22 00:00:00')]    5.619e-11   inf 0   nan nan nan
date[T.Timestamp('2014-10-29 00:00:00')]    -8.035e-12  inf -0  nan nan nan
date[T.Timestamp('2014-11-05 00:00:00')]    6.334e-11   inf 0   nan nan nan
date[T.Timestamp('2014-11-12 00:00:00')]    7.9e+04 inf 0   nan nan nan
date[T.Timestamp('2014-11-19 00:00:00')]    1.58e+05    inf 0   nan nan nan
date[T.Timestamp('2014-11-26 00:00:00')]    1.58e+05    inf 0   nan nan nan
date[T.Timestamp('2014-12-03 00:00:00')]    1.58e+05    inf 0   nan nan nan
date[T.Timestamp('2014-12-10 00:00:00')]    2.28e+05    inf 0   nan nan nan
date[T.Timestamp('2014-12-17 00:00:00')]    3.28e+05    inf 0   nan nan nan
date[T.Timestamp('2014-12-24 00:00:00')]    3.705e+05   inf 0   nan nan nan
spend   2.105e-10   inf 0   nan nan nan

I also tried statsmodel's tms package, but wasn't sure what to do about 'frequencies':

ar_model = sm.tsa.AR(df, freq='1')

ValueError: Pandas data cast to numpy dtype of object. Check input data with np.asarray(data).
Sep 4, 2018 in Python by bug_seeker
• 15,520 points

1 answer to this question.

0 votes

I'd really like to see a data sample as well as a code snippet to reproduce your error. Without that, my suggestion will not address your particular error message. It will, however, let you run a multiple regression analysis on a set of time series stored in a pandas dataframe. Assuming that you're using continuous and not categorical values in your time series, here is how I would do it using pandas and statsmodels:

A dataframe with random values:

# Imports
import pandas as pd
import numpy as np
import itertools

rows = 12
listVars= ['y','x1', 'x2', 'x3']
rng = pd.date_range('1/1/2017', periods=rows, freq='D')
df_1 = pd.DataFrame(np.random.randint(100,150,size=(rows, len(listVars))), columns=listVars) 
df_1 = df_1.set_index(rng)


The function below will let you specify a source dataframe as well as a dependent variable y and a selection of independent variables x1, x2. Using statsmodels, some desired results will be stored in a dataframe. There, R2 will be of type numeric, while the regression coefficients and p-values will be lists since the numbers of these estimates will vary with the number of independent variables you wish to include in your analysis.

def LinReg(df, y, x, const):

    betas = x.copy()

    # Model with out without a constant
    if const == True:
        x = sm.add_constant(df[x])
        model = sm.OLS(df[y], x).fit()
        model = sm.OLS(df[y], df[x]).fit()

    # Estimates of R2 and p
    res1 = {'Y': [y],
            'R2': [format(model.rsquared, '.4f')],
            'p': [model.pvalues.tolist()],
            'start': [df.index[0]], 
            'stop': [df.index[-1]],
            'obs' : [df.shape[0]],
            'X': [betas]}
    df_res1 = pd.DataFrame(data = res1)

    # Regression Coefficients
    theParams = model.params[0:]
    coefs = theParams.to_frame()
    df_coefs = pd.DataFrame(coefs.T)
    xNames = list(df_coefs)
    xValues = list(df_coefs.loc[0].values)
    xValues2 = [ '%.2f' % elem for elem in xValues ]
    res2 = {'Independent': [xNames],
            'beta': [xValues2]}
    df_res2 = pd.DataFrame(data = res2)

    # All results
    df_res = pd.concat([df_res1, df_res2], axis = 1)
    df_res = df_res.T
    df_res.columns = ['results']

Here's a test run:

df_regression = LinReg(df = df, y = 'y', x = ['x1', 'x2'], const = True)


R2                                                       0.3650
X                                                      [x1, x2]
Y                                                             y
obs                                                          12
p             [0.7417691742514285, 0.07989515781898897, 0.25...
start                                       2017-01-01 00:00:00
stop                                        2017-01-12 00:00:00
Independent                                     [const, x1, x2]
coefficients                                [16.29, 0.47, 0.37]

answered Sep 4, 2018 by Priyaj
• 58,090 points

Related Questions In Python

0 votes
1 answer

How does insertion work in Python?

Or, this one: def ins_sort(k): ...READ MORE

answered Oct 8, 2018 in Python by charlie_brown
• 7,720 points
0 votes
1 answer

How does % work in Python?

The % (modulo) operator yields the remainder ...READ MORE

answered Oct 10, 2018 in Python by SDeb
• 13,300 points
0 votes
1 answer

How to parse date/time string with timezone abbreviated name in Python?

The parse() function in dateutil can't handle ...READ MORE

answered Nov 27, 2018 in Python by Nymeria
• 3,560 points
0 votes
2 answers
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 7, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 4,214 views
0 votes
1 answer
0 votes
1 answer

How does the @property decorator work?

The property() function returns a special descriptor object: >>> property() <property object ...READ MORE

answered Sep 19, 2018 in Python by Priyaj
• 58,090 points
0 votes
1 answer

How does slice notation in Python work?

The Python tutorial talks about it (scroll down a ...READ MORE

answered Oct 31, 2018 in Python by Priyaj
• 58,090 points
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP