1
\$\begingroup\$

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0

Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns contain text posts of var length. I then loop through each of these rows and query the post against an imported function that returns a specific counter of variable length. The counter tells me about the presence of certain words in the text. On average the func takes less than 1 ms to return this counter. (Timer for the "Func" inserted at the end to prove this)

Overheads: The code i'm looking to improve is the loop. I accept a certain level of overhead with the "func" which can't be improved at this minute. I have considered looking at Spark or Dask to parallelize the loop and speed up the process. Suggestions are welcome

#Import data
import pandas as pd 
from func import func 
data = pd.read_csv('Dataset.csv')

print(len(data))
>> 1900000

print(data.columns)
>> Index(['type', 'body'], dtype='object')


#Create new DF
data2 = pd.Dataframe()

for post in data['post']:
    post = str(post)
    scores = func.countWords(posts)
    data2 = data2.append(scores,ignore_index=True)

print(scores)
>> Counter({0: 306,
     1: 185,
     2: 61,
     45: 31,
     87: 23,
     92: 5,
     94: 3,
     102: 30,})


 import time
 start = time.time()
 score = func.countWords("Slow down Sir, you're going to give yourself skin faliure!")
 end = time.time()
 print(end - start)
 >> 0.0019948482513427734
\$\endgroup\$

closed as off-topic by IEatBagels, dfhwze, Toby Speight, t3chb0t, VisualMelon Aug 15 at 11:36

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Lacks concrete context: Code Review requires concrete code from a project, with sufficient context for reviewers to understand how that code is used. Pseudocode, stub code, hypothetical code, obfuscated code, and generic best practices are outside the scope of this site." – IEatBagels, dfhwze, Toby Speight, t3chb0t, VisualMelon
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    \$\begingroup\$ Any appreciable speedup here will come from optimizing countWords - I"d advise that you post that function for review. \$\endgroup\$ – Zach Jul 31 '18 at 16:56
  • 1
    \$\begingroup\$ This question is being discussed on meta \$\endgroup\$ – IEatBagels Aug 14 at 13:04
0
\$\begingroup\$

Several errors:

I then loop through each of these loops

I take it you mean "I then loop through each of these columns".

for post in data['post]:

missing end quote mark

scores = Func.countWords(posts)

You imported func (lowercase) and now you're calling Func (uppercase)

data2 = data2.append(scores,ignore_index=True)

append should take a row-type object. If the function returns a numeric, then you shouldn't be appending it. Instead you can do:

def post_to_count(post):
      return func.countWord(str(post))

scores = data['post'].apply(post_to_count)
\$\endgroup\$
  • \$\begingroup\$ I've fixed those spelling and syntaxs errors. Your .apply works fine, but 1) it returns a combined dataframe, 2) does it increase speed up? \$\endgroup\$ – F.D Jul 31 '18 at 16:16
  • \$\begingroup\$ What about scores = data.post.str.apply(func.countWord)? \$\endgroup\$ – 409_Conflict Jul 31 '18 at 20:05
  • 3
    \$\begingroup\$ Hey there, you've answered the question in a state when it was off-topic for the site. In accordance with existing site policy the edit to the question that makes your answer basically moot is acceptable. Considering that your answer in it's current state doesn't really contain a lot of useful suggestions for the latest revision of the post, you may want to edit your answer to avoid downvotes (or the answer being deleted). For more information, see this meta discussion. Thanks! \$\endgroup\$ – Vogel612 Aug 14 at 13:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.