I am new in Python coding. I think the code could be written in a better and more compact form. It compiles quite slowly due to the method of removing stop-words.
I wanted to find the top 10 most frequent words from the column excluding the URL links, special characters, punctuations... and stop-words.
Any criticisms and suggestions to improve the efficiency & readability of my code would be greatly appreciated. Also, I want to know if there exists any dedicated python module to get the desired result easily.
I have a dataframe
df such that:
0 If I smelled the scent of hand sanitizers toda... 1 Hey @Yankees @YankeesPR and @MLB - wouldn't it... 2 @diane3443 @wdunlap @realDonaldTrump Trump nev... 3 @brookbanktv The one gift #COVID19 has give me... 4 25 July : Media Bulletin on Novel #CoronaVirus... ... 179103 Thanks @IamOhmai for nominating me for the @WH... 179104 2020! The year of insanity! Lol! #COVID19 http... 179105 @CTVNews A powerful painting by Juan Lucena. I... 179106 More than 1,200 students test positive for #CO... 179107 I stop when I see a Stop\n\n@SABCNews\n@Izinda... Name: text, Length: 179108, dtype: object
I have done it in the following way:
import pandas as pd import nltk import re import string from nltk.corpus import stopwords nltk.download('punkt') nltk.download('stopwords') from nltk.tokenize import word_tokenize stop_words = stopwords.words() def cleaning(text): # converting to lowercase, removing URL links, special characters, punctuations... text = text.lower() text = re.sub('https?://\S+|www\.\S+', '', text) text = re.sub('<.*?>+', '', text) text = re.sub('[%s]' % re.escape(string.punctuation), '', text) text = re.sub('\n', '', text) text = re.sub('[’“”…]', '', text) # removing the emojies # https://www.kaggle.com/alankritamishra/covid-19-tweet-sentiment-analysis#Sentiment-analysis emoji_pattern = re.compile("[" u"\U0001F600-\U0001F64F" # emoticons u"\U0001F300-\U0001F5FF" # symbols & pictographs u"\U0001F680-\U0001F6FF" # transport & map symbols u"\U0001F1E0-\U0001F1FF" # flags (iOS) u"\U00002702-\U000027B0" u"\U000024C2-\U0001F251" "]+", flags=re.UNICODE) text = emoji_pattern.sub(r'', text) # removing the stop-words text_tokens = word_tokenize(text) tokens_without_sw = [word for word in text_tokens if not word in stop_words] filtered_sentence = (" ").join(tokens_without_sw) text = filtered_sentence return text dt = df['text'].apply(cleaning) from collections import Counter p = Counter(" ".join(dt).split()).most_common(10) rslt = pd.DataFrame(p, columns=['Word', 'Frequency']) print(rslt)
Word Frequency 0 covid19 104546 1 cases 18150 2 new 14585 3 coronavirus 14189 4 amp 12227 5 people 9079 6 pandemic 7944 7 us 7223 8 deaths 7088 9 health 5231
An example IO of my function
inp = 'If I smelled the scent of hand sanitizers today on someone in the past, I would think they were so intoxicated that… https://t.co/QZvYbrOgb0' outp = cleaning(inp) print('Input:\n', inp) print('Output:\n', outp)
Input: If I smelled the scent of hand sanitizers today on someone in the past, I would think they were so intoxicated that… https://t.co/QZvYbrOgb0 Output: smelled scent hand sanitizers today someone past would think intoxicated