I created following script to clean text that I scraped. The clean text would ideally be lowercase words, without numbers and at maybe only commas and a dot at the end of a sentence. It should only have white-space between words and remove all "\n" elements from the text.
Particularly, I'm interested in feedback to the following code:
def cleaning(text):
import string
exclude = set(string.punctuation)
import re
# remove new line and digits with regular expression
text = re.sub(r'\n', '', text)
text = re.sub(r'\d', '', text)
# remove patterns matching url format
url_pattern = r'((http|ftp|https):\/\/)?[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&:/~\+#]*[\w\-\@?^=%&/~\+#])?'
text = re.sub(url_pattern, ' ', text)
# remove non-ascii characters
text = ''.join(character for character in text if ord(character) < 128)
# remove punctuations
text = ''.join(character for character in text if character not in exclude)
# standardize white space
text = re.sub(r'\s+', ' ', text)
# drop capitalization
text = text.lower()
#remove white space
text = text.strip()
return text
The script is cleaned via
cleaner = lambda x: cleaning(x)
df['text_clean'] = df['text'].apply(cleaner)
# Replace and remove empty rows
df['text_clean'] = df['text_clean'].replace('', np.nan)
df = df.dropna(how='any')
So far, the script does the job, which is great. However, how could the script above be improved, or be written cleaner?
Unclear seems the difference between
text = re.sub(r'\n', '', text)
text = re.sub('\n', '', text)
and whether
text = re.sub(r'\s+', ' ', text)
...
text = text.strip()
makes sense.
cleaner
is not necessary. Just do.apply(cleaning)
. \$\endgroup\$