# Python speed up dataframe writes

I'm aware that iterating frames in pandas is a very slow operation. I've heard vectorization speeds things up. I found a project that I'm using for sentiment analysis on reddit.

The Calculate dataframe method wasn't working for some reason, (maybe different version of panda) I wanted to refactor it to use vectorization but Sadly it took me 2 days to even get the code working.

def calculate_df(df):
data_df = df.filter(['tickers', 'score', 'sentiment'])

tickers_processed = pd.DataFrame(df.tickers.explode().value_counts())

tickers_processed = tickers_processed.rename(columns = {'tickers':'counts'})

tickers_processed['score'] = 0.0
tickers_processed['sentiment'] = 0.0

for idx, row_tick in enumerate(tickers_processed.itertuples(), 1):
upd_row_sent = row_tick.sentiment
upd_row_score = row_tick.score
for row_data in data_df.itertuples():
if(row_tick.Index in row_data.tickers):
upd_row_sent += row_data.sentiment['compound'] / row_tick.counts
upd_row_score += int(row_data.score) / row_tick.counts
tickers_processed.set_value(row_tick.Index, 'sentiment', upd_row_sent)
tickers_processed.set_value(row_tick.Index, 'score', upd_row_score)


Calling set value twice seems redundant, If I could calculate the dataset and update it using vectors I think the code would be a lot nicer

• Welcome to posting to CodeReview@SE at long last. I can't identify The Calculate dataframe method - am I missing something? – greybeard Feb 17 at 5:39
• @greybeard sorry the code is the contents of the method, ive updated to provide a link to the original code, this is the only portion I would like to refactor – johnny 5 Feb 17 at 14:28
• I edited the method "def line" into your post mainly to show you can copy&paste between CR post & (I)DE. I urge you to include the code following that from process.py, I'd probably choose a separate code block to tell supposed target from context. – greybeard Feb 17 at 15:47
• I can include more code, but I mostly just wanted to fix the double for loop, to leverage vertorization, I didn't want to include anything extra to confuse the reviewers – johnny 5 Feb 17 at 17:16
• (Reads like you didn't follow the above hyperlink to How do I ask a Good Question?) – greybeard Feb 17 at 17:53