2
\$\begingroup\$

Sentiment words behave very differently when under the semantic scope of negation. I want to use a slightly modified version of Das and Chen (2001) They detect words such as no, not, and never and then append a "neg"-suffix to every word appearing between a negation and a clause-level punctuation mark. This method is very static and I want to create something a little bit more dynamic with the help of dependency parsing from spaCy.

This is an example of a tweet that I will process:

RT @trader $AAPL 2012 is ooopen to Talk about patents with GOOG definitely not the treatment Samsung got heh someURL

See here the visualized dependency parser in action on my string. I am able to identify not as negation modifier of got with the following:

negation_tokens = [tok for tok in doc if tok.dep_ == 'neg']

Next, I want define the scope of the negation:

negation_head_tokens = [token.head for token in negation_tokens]   
for token in negation_head_tokens:
    end = token.i
    start = token.head.i + 1
    negated_tokens = doc[start:end]
    print(negated_tokens)
    #  ooopen to Talk about patents with GOOG definitely not the treatment Samsung

Now I have defined the scope, I want to add "not" to certain words conditional on their POS-tag:

list = ['ADJ', 'ADV', 'AUX', 'VERB']
for token in negated_tokens:
    for i in list:
        if token.pos_ == i:
           print('not'+token.text)
           # notooopen, notTalk, notdefinitely, notnot

Logically, I want to exclude notnot from the string.

Full script

import spacy

nlp = spacy.load("en_core_web_sm")

doc = nlp(u'RT @trader $AAPL 2012 is ooopen to Talk about patents with GOOG definitely not the treatment Samsung got heh someURL', disable=['ner'])
pos_list = ['ADJ', 'ADV', 'AUX', 'VERB']

def negation(doc):
    negation_tokens = [tok for tok in doc if tok.dep_ == 'neg']
    negation_cue = [token.text for token in negation_tokens]

    if not negation_tokens:   # no negation token(s) present in the string
        return doc 
    else:
        negation_head_tokens = [token.head for token in negation_tokens]
        new_doc = []

        for token in negation_head_tokens:
            end = token.i
            start = token.head.i + 1
            negated_tokens = doc[start:end]
        for token in doc:
           if token.text not in negation_cue:
               if token in negated_tokens:
                  if token.pos_ in pos_list:
                     new_doc.append('not'+token.text)
                     continue
                  else:
                     pass
               new_doc.append(token.text)
        return new_doc

negated_doc = negation(doc)
print(negated_doc)

Output:

['RT', '@trader', '$', 'AAPL', '2012', 'is', 'notooopen', 'to', 'notTalk', 'about', 'patents', 'with', 'GOOG', 'notdefinitely', 'the', 'treatment', 'Samsung', 'got', 'heh', 'someURL']

Questions

Q1: Is it possible to use a direct route instead of the creating a list with

negation_cue = [token.text for token in negation_tokens]

and then check

if token.text not in negation_cue:

Q2: Is it maybe faster to remove the negation_cue token at the end of the script from the returned doc?

Q3 / Most important question: Do you see any speed-improvements in my script

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.