I created following script to clean text that I scraped. The clean text would ideally be lowercase words, without numbers and at maybe only commas and a dot at the end of a sentence. It should only have white-space between words and remove all "\n" elements from the text.
Particularly, I'm interested in feedback to the following code:
def cleaning(text): import string exclude = set(string.punctuation) import re # remove new line and digits with regular expression text = re.sub(r'\n', '', text) text = re.sub(r'\d', '', text) # remove patterns matching url format url_pattern = r'((http|ftp|https):\/\/)?[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&:/~\+#]*[\w\-\@?^=%&/~\+#])?' text = re.sub(url_pattern, ' ', text) # remove non-ascii characters text = ''.join(character for character in text if ord(character) < 128) # remove punctuations text = ''.join(character for character in text if character not in exclude) # standardize white space text = re.sub(r'\s+', ' ', text) # drop capitalization text = text.lower() #remove white space text = text.strip() return text
The script is cleaned via
cleaner = lambda x: cleaning(x) df['text_clean'] = df['text'].apply(cleaner) # Replace and remove empty rows df['text_clean'] = df['text_clean'].replace('', np.nan) df = df.dropna(how='any')
So far, the script does the job, which is great. However, how could the script above be improved, or be written cleaner?
Unclear seems the difference between
text = re.sub(r'\n', '', text) text = re.sub('\n', '', text)
text = re.sub(r'\s+', ' ', text) ... text = text.strip()