I have a data set with close to 6 million rows of user input. Specifically, users were supposed to type in their email addresses, but because there was not pattern validation put in place we have a few months worth of interesting input.
I've come up with a script that counts every character, then combines it that so I can see the distribution of all characters. This enables me to do further analysis and get a sense of the most common mistakes so I can begin to clean the data. My question is: how would you optimize the following for speed?
import pandas as pd
import numpy as np
from pandas import Series, DataFrame
from collections import Counter
df = pd.DataFrame({'input': ['Captain Jean-Luc Picard <[email protected]>','[email protected]','geordi @starfleet.com','[email protected]','rik#[email protected]'],
'metric1': np.random.randn(5).cumsum(),
'metric2': np.random.randn(5)})
l = []
for i in range(len(df.index.values)):
l.append(dict(Counter(df.ix[i,'input'])))
dist = pd.DataFrame(l).fillna(0)
dist = dist.sum(axis=0)
print(dist)
I've run this over ~1/3 of my dataset, and it takes a while; it's still tolerable, I'm just curious if anyone could make it faster.