NLP sentiment analysis in Norwegian

Question

Please keep in mind that I am very new to data science and completely new to NLP! I am trying to create a model to classify customer reviews as either negative or positive. However, my main problem is that most of them are in Norwegian, which seems not to be very well supported. I have found this repo https://github.com/ltgoslo/norsentlex/tree/master/Fullform containing both negative and positive lexicons and have decided to use it.

with open("Fullform_Positive_lexicon.txt") as f:
    reader = csv.reader(f)
    positive_lexicon = []
    for row in reader:
        print(str(row))
        positive_lexicon.append(row[0])

with open("Fullform_Negative_lexicon.txt") as f:
    reader = csv.reader(f)
    negative_lexicon = []
    for row in reader:
        negative_lexicon.append(row[0])

#adjusted to use it with NaiveBayesClassifier
def get_tokens_for_model(cleaned_tokens_list):
    final_token_list = [] 
    for token in cleaned_tokens_list:
        token_dict = {}
        token_dict.update({token : True})
        final_token_list.append(token_dict)
    return final_token_list

positive_tokens_for_model = get_tokens_for_model(positive_lexicon)
negative_tokens_for_model = get_tokens_for_model(negative_lexicon)

positive_dataset = [(token, "Positive")
                     for token in positive_tokens_for_model]

negative_dataset = [(token, "Negative")
                     for token in negative_tokens_for_model]

dataset = positive_dataset + negative_dataset
#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i + 1)  
    dataset[i], dataset[j] = dataset[j], dataset[i]  

train_data = dataset[:20000]
test_data = dataset[7742:]

classifier = NaiveBayesClassifier.train(train_data)

print("Accuracy is:", classify.accuracy(classifier, test_data))

So I am just training my model based on this lexicon and then I am trying to apply it to my comments. I am getting here 87%, which is not so bad. However, it looks worse when used with whole sentences.

with open("stopwords.csv") as f:
    reader = csv.reader(f)
    norwegian_stopwords = []
    for row in reader:
        norwegian_stopwords.append(row[0])

customer_feedback = pd.read_excel("classification_sample.xlsx")
customer_feedback = customer_feedback.dropna()
customer_feedback['Kommentar'] = customer_feedback['Kommentar'].apply(remove_punctuation)

list_of_comments = list(customer_feedback['Kommentar'])

for comment in list_of_comments:
    custom_tokens = word_tokenize(comment)
    filtered_tokens = []
    for word in custom_tokens:
        if word not in  norwegian_stopwords:
            filtered_tokens.append(word)

    classification = classifier.classify(dict([token, True] for token in filtered_tokens))
    probdist = classifier.prob_classify(dict([token, True] for token in filtered_tokens))
    pred_sentiment = probdist.max()
    print("Sentence: " + comment)
    print("Classified as: " + str(classification))
    print("Key words: " + str(custom_tokens))
    print("Probability: " + str(round(probdist.prob(pred_sentiment), 2)))
    print("-----------------------------------------------------------")

I know my code is not the highest quality right now(although suggestions regarding this are also appreciated). I am looking mostly for some feedback what more can I do to improve my accuracy. What is currently not very clear to me, is how to properly train the model on words and then achieve the very same accuracy on sentences.

Answer 1

All of this:

def get_tokens_for_model(cleaned_tokens_list):
    final_token_list = [] 
    for token in cleaned_tokens_list:
        token_dict = {}
        token_dict.update({token : True})
        final_token_list.append(token_dict)
    return final_token_list

positive_tokens_for_model = get_tokens_for_model(positive_lexicon)
negative_tokens_for_model = get_tokens_for_model(negative_lexicon)

positive_dataset = [(token, "Positive")
                     for token in positive_tokens_for_model]

negative_dataset = [(token, "Negative")
                     for token in negative_tokens_for_model]

dataset = positive_dataset + negative_dataset
#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i + 1)  
    dataset[i], dataset[j] = dataset[j], dataset[i]  

Is just a very verbose way to write:

dataset = [({token : True}, "Positive") for token in positive_lexicon]
dataset.extend([({token : True}, "Negative") for token in negative_lexicon])
random.shuffle(dataset)

The shuffling probabilities might be a bit different, though. random.shuffle basically has for each element equal probability to end up at any index. I think your method has a bias, but I'm not quite sure.

At least your custom shuffling seems to be unbiased:

However, I had to fix it to this, because otherwise it raises an IndexError, because random.randint is inclusive of the end (in contrast to range, slices and random.randrange):

#shuffle dataset
for i in range(len(dataset)-1, 0, -1): 
    j = random.randint(0, i)  
    dataset[i], dataset[j] = dataset[j], dataset[i]

Answer 2

Ways to significantly improve performance and processing flow:

Accumulating lexicon dataset phase

The first 2 for loops for accumulating positive_lexicon and negative_lexicon could at least be optimized with a list comprehension:

...
reader = csv.reader(f)
positive_lexicon = [row[0] for row in reader]

that's better, but not the best when reading large files. (we'll beat that in the final version below)

The next 4 for loops are redundantly traversing the same sequences for just enriching each entry/token with {token : True}, then - with respective keyword flag "Positive"/"Negative".

Since all intermediate datasets are not used anywhere further and just intended to compose a combined final dataset dataset = positive_dataset + negative_dataset - all those 6 for loops can be substituted with a single efficient generator function which will be consumed just once and return the needed entries.
The final version for the 1st processing phase (lexicons/accuracy):

pos_lexicon_fname = "Fullform_Positive_lexicon.txt"
neg_lexicon_fname = "Fullform_Negative_lexicon.txt"

def get_lexicon_dataset(pos_lexicon_fname, neg_lexicon_fname):
    with open(pos_lexicon_fname) as f:
        reader = csv.reader(f)
        for row in reader:
            yield ({row[0]: True}, "Positive")

    with open(neg_lexicon_fname) as f:
        reader = csv.reader(f)
        for row in reader:
            yield ({row[0]: True}, "Negative")

dataset = list(get_lexicon_dataset(pos_lexicon_fname, neg_lexicon_fname))
#shuffle dataset
...

---

Stopwords/tokenizing/probability phase:

norwegian_stopwords accumulation approach is better composed with a list comprehension:

with open("stopwords.csv") as f:
    reader = csv.reader(f)
    norwegian_stopwords = [row[0] row in reader]

The code fragment which runs on each iteration (of the outer for loop):

...
filtered_tokens = []
for word in custom_tokens:
    if word not in  norwegian_stopwords:
        filtered_tokens.append(word)

classification = classifier.classify(dict([token, True] for token in filtered_tokens))
probdist = classifier.prob_classify(dict([token, True] for token in filtered_tokens))

introduces 2 issues:

• filtered_tokens is better accumulated with a list comprehension
• the same filtered_tokens sequence is then redundantly traversed twice

But, considering that the target data structure for classifying operation is a dictionary of filtered tokens (dict([token, True], ...)) - the more optimized and straightforward way is to compose such a dictionary at once:

for comment in list_of_comments:
    custom_tokens = word_tokenize(comment)
    filtered_tokens_dict = {word: True for word in custom_tokens
                            if word not in norwegian_stopwords}

    classification = classifier.classify(filtered_tokens_dict)
    probdist = classifier.prob_classify(filtered_tokens_dict)
    pred_sentiment = probdist.max()
    print("Sentence: " + comment)
    ...

---

Things to remember:

• Generator expressions don’t materialize the whole output sequence when they’re run. Instead, generator expressions evaluate to an iterator that yields one item at a time from the expression
• Generators can produce a sequence of outputs for arbitrarily large inputs because their working memory doesn’t include all inputs and outputs