I have a text analytics process that I complete for work. This process mines text for different Twitter accounts and finds patterns in the tweets. The actual machine learning pipeline is much more complicated than what's shown below, but this will have to suffice as my minimum reproducible example. In reality, there are a lot more instances when I asked for user input in the script, so I don't think things like user input should be stored in the same script as the functions.
Right now, I run this script on an ad-hoc basis, and a lot of the things are hard-coded. I want to be able to package the code I've written to make it more robust and stable.
How should the functions before
__main__ and the procedures in
__main__ be split up to best make this code a package? Should the functions be saved in a different script altogether, and then be imported when needed to a different script? Right now, I just run these scripts from the command line and am sort of lost as to where to go in terms of packaging it.
Text data cleaning script:
import pandas as pd from nltk.corpus import stopwords import re import string import os def strip_punct(string, regex): return regex.sub(' ', string) def clean_string(string, regex, stopwords): words_to_keep =  stripped_string = strip_punct(string, regex) string_words = stripped_string.split() for word in string_words: if word not in stopwords: words_to_keep.append(word) cleaned_string = " ".join(words_to_keep) return cleaned_string def clean_and_map_column(column, df, regex, stopwords): text_set = set(df[column]) text_dict = dict((string, clean_string(row, regex, stopwords)) for row in text_set) df[column] = df[column].map(text_dict) return df[column] if __name__ == '__main__': os.chdir("/path/to/my/folder") # regular expression containing punctuation to be removed from strings regexp = re.compile('[%s]' % re.escape(''.join(p for p in string.punctuation))) # set of stopwords to remove from text from NLTK stops = stopwords.words('english') # my data frame, containing only columns of text df = pd.read_csv("file.csv") # clean each column for col in df.columns.values: df[col] = clean_and_map_column(col, df, regexp) df.to_csv("cleaned_file.csv")
KMeans clustering script:
from sklearn.feature_extraction.text import CountVectorizer from sklearn.cluster import KMeans import pandas as pd import os if __name__ == '__main__': os.chdir("/path/to/my/folder") df = pd.read_csv("cleaned_file.csv") corpus_column = 'twitter_data' corpus = df[corpus_column] vectorizer = CountVectorizer() X = vectorizer.fit_transform(corpus) model = KMeans(k = 5) model.fit(X) df['cluster_id'] = model.labels_ df.to_csv('kmeans_file.csv")