8
\$\begingroup\$

Please keep in mind that I am very new to data science and completely new to NLP! I am trying to create a model to classify customer reviews as either negative or positive. However, my main problem is that most of them are in Norwegian, which seems not to be very well supported. I have found this repo https://github.com/ltgoslo/norsentlex/tree/master/Fullform containing both negative and positive lexicons and have decided to use it.

with open("Fullform_Positive_lexicon.txt") as f:
    reader = csv.reader(f)
    positive_lexicon = []
    for row in reader:
        print(str(row))
        positive_lexicon.append(row[0])

with open("Fullform_Negative_lexicon.txt") as f:
    reader = csv.reader(f)
    negative_lexicon = []
    for row in reader:
        negative_lexicon.append(row[0])

#adjusted to use it with NaiveBayesClassifier
def get_tokens_for_model(cleaned_tokens_list):
    final_token_list = [] 
    for token in cleaned_tokens_list:
        token_dict = {}
        token_dict.update({token : True})
        final_token_list.append(token_dict)
    return final_token_list

positive_tokens_for_model = get_tokens_for_model(positive_lexicon)
negative_tokens_for_model = get_tokens_for_model(negative_lexicon)

positive_dataset = [(token, "Positive")
                     for token in positive_tokens_for_model]

negative_dataset = [(token, "Negative")
                     for token in negative_tokens_for_model]

dataset = positive_dataset + negative_dataset
#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i + 1)  
    dataset[i], dataset[j] = dataset[j], dataset[i]  

train_data = dataset[:20000]
test_data = dataset[7742:]

classifier = NaiveBayesClassifier.train(train_data)

print("Accuracy is:", classify.accuracy(classifier, test_data))

So I am just training my model based on this lexicon and then I am trying to apply it to my comments. I am getting here 87%, which is not so bad. However, it looks worse when used with whole sentences.

with open("stopwords.csv") as f:
    reader = csv.reader(f)
    norwegian_stopwords = []
    for row in reader:
        norwegian_stopwords.append(row[0])

customer_feedback = pd.read_excel("classification_sample.xlsx")
customer_feedback = customer_feedback.dropna()
customer_feedback['Kommentar'] = customer_feedback['Kommentar'].apply(remove_punctuation)

list_of_comments = list(customer_feedback['Kommentar'])

for comment in list_of_comments:
    custom_tokens = word_tokenize(comment)
    filtered_tokens = []
    for word in custom_tokens:
        if word not in  norwegian_stopwords:
            filtered_tokens.append(word)

    classification = classifier.classify(dict([token, True] for token in filtered_tokens))
    probdist = classifier.prob_classify(dict([token, True] for token in filtered_tokens))
    pred_sentiment = probdist.max()
    print("Sentence: " + comment)
    print("Classified as: " + str(classification))
    print("Key words: " + str(custom_tokens))
    print("Probability: " + str(round(probdist.prob(pred_sentiment), 2)))
    print("-----------------------------------------------------------")

I know my code is not the highest quality right now(although suggestions regarding this are also appreciated). I am looking mostly for some feedback what more can I do to improve my accuracy. What is currently not very clear to me, is how to properly train the model on words and then achieve the very same accuracy on sentences.

\$\endgroup\$
  • \$\begingroup\$ Hi, I have forgot to copy this piece of code. Its already there:) \$\endgroup\$ – Grevioos Nov 14 '19 at 11:58
  • \$\begingroup\$ Where is word_tokenize from? Is it from nltk? Please include your imports. \$\endgroup\$ – Graipher Nov 14 '19 at 14:09
  • \$\begingroup\$ You would benefit from running your tokens through a lemmatizer or a stemmer. If the word "glad" is in your positive tokens list, you'll want to find "glade" and "gladere" too. \$\endgroup\$ – Jetpack Nov 14 '19 at 18:09
  • \$\begingroup\$ @Łukasz D. Tulikowski: have a word about suggested edit in CR chat? \$\endgroup\$ – greybeard Nov 15 '19 at 6:26
3
\$\begingroup\$

All of this:

def get_tokens_for_model(cleaned_tokens_list):
    final_token_list = [] 
    for token in cleaned_tokens_list:
        token_dict = {}
        token_dict.update({token : True})
        final_token_list.append(token_dict)
    return final_token_list

positive_tokens_for_model = get_tokens_for_model(positive_lexicon)
negative_tokens_for_model = get_tokens_for_model(negative_lexicon)

positive_dataset = [(token, "Positive")
                     for token in positive_tokens_for_model]

negative_dataset = [(token, "Negative")
                     for token in negative_tokens_for_model]

dataset = positive_dataset + negative_dataset
#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i + 1)  
    dataset[i], dataset[j] = dataset[j], dataset[i]  

Is just a very verbose way to write:

dataset = [({token : True}, "Positive") for token in positive_lexicon]
dataset.extend([({token : True}, "Negative") for token in negative_lexicon])
random.shuffle(dataset)

The shuffling probabilities might be a bit different, though. random.shuffle basically has for each element equal probability to end up at any index. I think your method has a bias, but I'm not quite sure.

At least your custom shuffling seems to be unbiased:

enter image description here

However, I had to fix it to this, because otherwise it raises an IndexError, because random.randint is inclusive of the end (in contrast to range, slices and random.randrange):

#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i)  
    dataset[i], dataset[j] = dataset[j], dataset[i]
|improve this answer|||||
\$\endgroup\$
3
\$\begingroup\$

Ways to significantly improve performance and processing flow:

Accumulating lexicon dataset phase

The first 2 for loops for accumulating positive_lexicon and negative_lexicon could at least be optimized with a list comprehension:

...
reader = csv.reader(f)
positive_lexicon = [row[0] for row in reader]

that's better, but not the best when reading large files. (we'll beat that in the final version below)

The next 4 for loops are redundantly traversing the same sequences for just enriching each entry/token with {token : True}, then - with respective keyword flag "Positive"/"Negative".

Since all intermediate datasets are not used anywhere further and just intended to compose a combined final dataset dataset = positive_dataset + negative_dataset - all those 6 for loops can be substituted with a single efficient generator function which will be consumed just once and return the needed entries.
The final version for the 1st processing phase (lexicons/accuracy):

pos_lexicon_fname = "Fullform_Positive_lexicon.txt"
neg_lexicon_fname = "Fullform_Negative_lexicon.txt"

def get_lexicon_dataset(pos_lexicon_fname, neg_lexicon_fname):
    with open(pos_lexicon_fname) as f:
        reader = csv.reader(f)
        for row in reader:
            yield ({row[0]: True}, "Positive")

    with open(neg_lexicon_fname) as f:
        reader = csv.reader(f)
        for row in reader:
            yield ({row[0]: True}, "Negative")

dataset = list(get_lexicon_dataset(pos_lexicon_fname, neg_lexicon_fname))
#shuffle dataset
...

Stopwords/tokenizing/probability phase:

norwegian_stopwords accumulation approach is better composed with a list comprehension:

with open("stopwords.csv") as f:
    reader = csv.reader(f)
    norwegian_stopwords = [row[0] row in reader]

The code fragment which runs on each iteration (of the outer for loop):

...
filtered_tokens = []
for word in custom_tokens:
    if word not in  norwegian_stopwords:
        filtered_tokens.append(word)

classification = classifier.classify(dict([token, True] for token in filtered_tokens))
probdist = classifier.prob_classify(dict([token, True] for token in filtered_tokens))

introduces 2 issues:

  • filtered_tokens is better accumulated with a list comprehension
  • the same filtered_tokens sequence is then redundantly traversed twice

But, considering that the target data structure for classifying operation is a dictionary of filtered tokens (dict([token, True], ...)) - the more optimized and straightforward way is to compose such a dictionary at once:

for comment in list_of_comments:
    custom_tokens = word_tokenize(comment)
    filtered_tokens_dict = {word: True for word in custom_tokens
                            if word not in norwegian_stopwords}

    classification = classifier.classify(filtered_tokens_dict)
    probdist = classifier.prob_classify(filtered_tokens_dict)
    pred_sentiment = probdist.max()
    print("Sentence: " + comment)
    ...

Things to remember:

  • Generator expressions don’t materialize the whole output sequence when they’re run. Instead, generator expressions evaluate to an iterator that yields one item at a time from the expression
  • Generators can produce a sequence of outputs for arbitrarily large inputs because their working memory doesn’t include all inputs and outputs
|improve this answer|||||
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.