# NLP sentiment analysis in Norwegian

Please keep in mind that I am very new to data science and completely new to NLP! I am trying to create a model to classify customer reviews as either negative or positive. However, my main problem is that most of them are in Norwegian, which seems not to be very well supported. I have found this repo https://github.com/ltgoslo/norsentlex/tree/master/Fullform containing both negative and positive lexicons and have decided to use it.

with open("Fullform_Positive_lexicon.txt") as f:
positive_lexicon = []
print(str(row))
positive_lexicon.append(row[0])

with open("Fullform_Negative_lexicon.txt") as f:
negative_lexicon = []
negative_lexicon.append(row[0])

#adjusted to use it with NaiveBayesClassifier
def get_tokens_for_model(cleaned_tokens_list):
final_token_list = []
for token in cleaned_tokens_list:
token_dict = {}
token_dict.update({token : True})
final_token_list.append(token_dict)
return final_token_list

positive_tokens_for_model = get_tokens_for_model(positive_lexicon)
negative_tokens_for_model = get_tokens_for_model(negative_lexicon)

positive_dataset = [(token, "Positive")
for token in positive_tokens_for_model]

negative_dataset = [(token, "Negative")
for token in negative_tokens_for_model]

dataset = positive_dataset + negative_dataset
#shuffle dataset
for i in range(len(dataset)-1, 0, -1):
j = random.randint(0, i + 1)
dataset[i], dataset[j] = dataset[j], dataset[i]

train_data = dataset[:20000]
test_data = dataset[7742:]

classifier = NaiveBayesClassifier.train(train_data)

print("Accuracy is:", classify.accuracy(classifier, test_data))


So I am just training my model based on this lexicon and then I am trying to apply it to my comments. I am getting here 87%, which is not so bad. However, it looks worse when used with whole sentences.

with open("stopwords.csv") as f:
norwegian_stopwords = []
norwegian_stopwords.append(row[0])

customer_feedback = customer_feedback.dropna()
customer_feedback['Kommentar'] = customer_feedback['Kommentar'].apply(remove_punctuation)

custom_tokens = word_tokenize(comment)
filtered_tokens = []
for word in custom_tokens:
if word not in  norwegian_stopwords:
filtered_tokens.append(word)

classification = classifier.classify(dict([token, True] for token in filtered_tokens))
probdist = classifier.prob_classify(dict([token, True] for token in filtered_tokens))
pred_sentiment = probdist.max()
print("Sentence: " + comment)
print("Classified as: " + str(classification))
print("Key words: " + str(custom_tokens))
print("Probability: " + str(round(probdist.prob(pred_sentiment), 2)))
print("-----------------------------------------------------------")


I know my code is not the highest quality right now(although suggestions regarding this are also appreciated). I am looking mostly for some feedback what more can I do to improve my accuracy. What is currently not very clear to me, is how to properly train the model on words and then achieve the very same accuracy on sentences.

• Hi, I have forgot to copy this piece of code. Its already there:) – Grevioos Nov 14 '19 at 11:58
• Where is word_tokenize from? Is it from nltk? Please include your imports. – Graipher Nov 14 '19 at 14:09
• You would benefit from running your tokens through a lemmatizer or a stemmer. If the word "glad" is in your positive tokens list, you'll want to find "glade" and "gladere" too. – Jetpack Nov 14 '19 at 18:09
• @Łukasz D. Tulikowski: have a word about suggested edit in CR chat? – greybeard Nov 15 '19 at 6:26

All of this:

def get_tokens_for_model(cleaned_tokens_list):
final_token_list = []
for token in cleaned_tokens_list:
token_dict = {}
token_dict.update({token : True})
final_token_list.append(token_dict)
return final_token_list

positive_tokens_for_model = get_tokens_for_model(positive_lexicon)
negative_tokens_for_model = get_tokens_for_model(negative_lexicon)

positive_dataset = [(token, "Positive")
for token in positive_tokens_for_model]

negative_dataset = [(token, "Negative")
for token in negative_tokens_for_model]

dataset = positive_dataset + negative_dataset
#shuffle dataset
for i in range(len(dataset)-1, 0, -1):
j = random.randint(0, i + 1)
dataset[i], dataset[j] = dataset[j], dataset[i]


Is just a very verbose way to write:

dataset = [({token : True}, "Positive") for token in positive_lexicon]
dataset.extend([({token : True}, "Negative") for token in negative_lexicon])
random.shuffle(dataset)


The shuffling probabilities might be a bit different, though. random.shuffle basically has for each element equal probability to end up at any index. I think your method has a bias, but I'm not quite sure.

At least your custom shuffling seems to be unbiased:

However, I had to fix it to this, because otherwise it raises an IndexError, because random.randint is inclusive of the end (in contrast to range, slices and random.randrange):

#shuffle dataset
for i in range(len(dataset)-1, 0, -1):
j = random.randint(0, i)
dataset[i], dataset[j] = dataset[j], dataset[i]


### Ways to significantly improve performance and processing flow:

Accumulating lexicon dataset phase

The first 2 for loops for accumulating positive_lexicon and negative_lexicon could at least be optimized with a list comprehension:

...
positive_lexicon = [row[0] for row in reader]


that's better, but not the best when reading large files. (we'll beat that in the final version below)

The next 4 for loops are redundantly traversing the same sequences for just enriching each entry/token with {token : True}, then - with respective keyword flag "Positive"/"Negative".

Since all intermediate datasets are not used anywhere further and just intended to compose a combined final dataset dataset = positive_dataset + negative_dataset - all those 6 for loops can be substituted with a single efficient generator function which will be consumed just once and return the needed entries.
The final version for the 1st processing phase (lexicons/accuracy):

pos_lexicon_fname = "Fullform_Positive_lexicon.txt"
neg_lexicon_fname = "Fullform_Negative_lexicon.txt"

def get_lexicon_dataset(pos_lexicon_fname, neg_lexicon_fname):
with open(pos_lexicon_fname) as f:
yield ({row[0]: True}, "Positive")

with open(neg_lexicon_fname) as f:
yield ({row[0]: True}, "Negative")

dataset = list(get_lexicon_dataset(pos_lexicon_fname, neg_lexicon_fname))
#shuffle dataset
...


Stopwords/tokenizing/probability phase:

norwegian_stopwords accumulation approach is better composed with a list comprehension:

with open("stopwords.csv") as f:
norwegian_stopwords = [row[0] row in reader]


The code fragment which runs on each iteration (of the outer for loop):

...
filtered_tokens = []
for word in custom_tokens:
if word not in  norwegian_stopwords:
filtered_tokens.append(word)

classification = classifier.classify(dict([token, True] for token in filtered_tokens))
probdist = classifier.prob_classify(dict([token, True] for token in filtered_tokens))


introduces 2 issues:

• filtered_tokens is better accumulated with a list comprehension
• the same filtered_tokens sequence is then redundantly traversed twice

But, considering that the target data structure for classifying operation is a dictionary of filtered tokens (dict([token, True], ...)) - the more optimized and straightforward way is to compose such a dictionary at once:

for comment in list_of_comments:
custom_tokens = word_tokenize(comment)
filtered_tokens_dict = {word: True for word in custom_tokens
if word not in norwegian_stopwords}

classification = classifier.classify(filtered_tokens_dict)
probdist = classifier.prob_classify(filtered_tokens_dict)
pred_sentiment = probdist.max()
print("Sentence: " + comment)
...


Things to remember:

• Generator expressions don’t materialize the whole output sequence when they’re run. Instead, generator expressions evaluate to an iterator that yields one item at a time from the expression
• Generators can produce a sequence of outputs for arbitrarily large inputs because their working memory doesn’t include all inputs and outputs