I'm aware that iterating frames in pandas is a very slow operation. I've heard vectorization speeds things up. I found a project that I'm using for sentiment analysis on reddit.

The Calculate dataframe method wasn't working for some reason, (maybe different version of panda) I wanted to refactor it to use vectorization but Sadly it took me 2 days to even get the code working.

def calculate_df(df):
    data_df = df.filter(['tickers', 'score', 'sentiment'])
    tickers_processed = pd.DataFrame(df.tickers.explode().value_counts())
    tickers_processed = tickers_processed.rename(columns = {'tickers':'counts'})
    tickers_processed['score'] = 0.0
    tickers_processed['sentiment'] = 0.0
    for idx, row_tick in enumerate(tickers_processed.itertuples(), 1):
        upd_row_sent = row_tick.sentiment
        upd_row_score = row_tick.score
        for row_data in data_df.itertuples():
            if(row_tick.Index in row_data.tickers):
                upd_row_sent += row_data.sentiment['compound'] / row_tick.counts
                upd_row_score += int(row_data.score) / row_tick.counts
                tickers_processed.set_value(row_tick.Index, 'sentiment', upd_row_sent)
                tickers_processed.set_value(row_tick.Index, 'score', upd_row_score)

Calling set value twice seems redundant, If I could calculate the dataset and update it using vectors I think the code would be a lot nicer

  • \$\begingroup\$ Welcome to posting to CodeReview@SE at long last. I can't identify The Calculate dataframe method - am I missing something? \$\endgroup\$ – greybeard Feb 17 at 5:39
  • \$\begingroup\$ @greybeard sorry the code is the contents of the method, ive updated to provide a link to the original code, this is the only portion I would like to refactor \$\endgroup\$ – johnny 5 Feb 17 at 14:28
  • \$\begingroup\$ I edited the method "def line" into your post mainly to show you can copy&paste between CR post & (I)DE. I urge you to include the code following that from process.py, I'd probably choose a separate code block to tell supposed target from context. \$\endgroup\$ – greybeard Feb 17 at 15:47
  • \$\begingroup\$ I can include more code, but I mostly just wanted to fix the double for loop, to leverage vertorization, I didn't want to include anything extra to confuse the reviewers \$\endgroup\$ – johnny 5 Feb 17 at 17:16
  • \$\begingroup\$ (Reads like you didn't follow the above hyperlink to How do I ask a Good Question?) \$\endgroup\$ – greybeard Feb 17 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.