Linear Regression and data manipulation

Question

How could I improve the following code that runs a simple linear regression using matrix algebra? I import a .csv file (link here) called 'cdd.ny.csv', and perform the matrix calculations that solve for the coefficients (intercept and regressor) of Y = XB (i.e., $(X'X)^{-1}X'Y$):

import numpy
from numpy import *
import csv

df1 = csv.reader(open('cdd.ny.csv', 'rb'),delimiter=',')
tmp = list(df1)
b = numpy.array(tmp).astype('string')
b1 = b[1:,3:5]
b2 = numpy.array(b1).astype('float')

nrow = b1.shape[0]
intercept = ones( (nrow,1), dtype=int16 )

b3 = empty( (nrow,1), dtype = float )

i = 0
while i < nrow:
    b3[i,0] = b2[i,0]
    i = i + 1

X = numpy.concatenate((intercept, b3), axis=1)
X = matrix(X)

Y = b2[:,1]
Y = matrix(Y).T

m1 = dot(X.T,X).I
m2 = dot(X.T,Y)

beta = m1*m2

print beta

#[[-7.62101913]
# [ 0.5937734 ]]

To check my answer:

numpy.linalg.lstsq(X,Y)

Answer 1

import numpy
from numpy import *
import csv

df1 = csv.reader(open('cdd.ny.csv', 'rb'),delimiter=',')
tmp = list(df1)
b = numpy.array(tmp).astype('string')
b1 = b[1:,3:5]
b2 = numpy.array(b1).astype('float')

Firstly, I'd avoid all these abbreviated variables. It makes it hard to follow your code. You can also combine the lines a lot more

b2 = numpy.array(list(df1))[1:,3:5].astype('float')

That way we avoid creating so many variables.

nrow = b1.shape[0]
intercept = ones( (nrow,1), dtype=int16 )

b3 = empty( (nrow,1), dtype = float )

i = 0
while i < nrow:
    b3[i,0] = b2[i,0]
    i = i + 1

This whole can be replaced by b3 = b2[:,0]

X = numpy.concatenate((intercept, b3), axis=1)
X = matrix(X)

If you really want to use matrix, combine these two lines. But really, its probably better to use just array not matrix.

Y = b2[:,1]
Y = matrix(Y).T

m1 = dot(X.T,X).I
m2 = dot(X.T,Y)

beta = m1*m2



print beta