2
\$\begingroup\$

I am currently extracting post on a biligual page on Facebook. Therefore, I have the problem of splitting the post in French and english before starting analysing them. I have construct a function which permit me to filter the sentence in english from all my sentence.

However it is quite complex and not accurate..

### import enchant
import nltk
### nltk.download()
### from nltk.corpus import words
from nltk.corpus import brown
import pandas as pd
import numpy as np
import re
  ### pop is an example of posts...

pop = ['hi','say','bonjour mister','sauf','tu am      belle','I suis cool','Worst trip ever je deteste AirTransat! ce sont les pires']


### Defining my function

def Extract_sentence_english(list_of_sentence):

    ### Removing punctuation, numeric and whitespace at the same time
    list_of_sentence = [re.sub(r'[^A-Za-z]+', ' ', x) for x in list_of_sentence]
    ### To lower
    list_of_sentence = map(lambda x: x.lower(), list_of_sentence)

    ### Split the list into words (splitting on space)
    split_list = [y for x in list_of_sentence for y in x.split(' ')] 
    ### Return a logical value (True if English, False if other)
    log = []
    for i in range(0,len(split_list)):
        k = split_list[i] in words.words()
        log.append(k)
    ### log contain the logical value for the words    
    ### Number of words per sentence
    df=[]
    for entry in list_of_sentence:
        df.append(len(entry.split()))

    df = pd.DataFrame(df)
    df.columns = ['number_of_words']

    df['index'] = range(0,len(df))

    dat = pd.DataFrame(df.loc[np.repeat(df.index.values,df['number_of_words'])])
    dat['logical'] = log
    dat['log_num'] = dat['logical'].astype(int)

    df['English_words_freq'] = dat.groupby(['index'], as_index = False).sum()['log_num']/df['number_of_words']
    final = pd.DataFrame(list_of_sentence)
    final['logical'] = df['English_words_freq'] > 0.75
    return(final)


Extract_sentence_english(pop)

Does somebody know any better way of doing this ?

\$\endgroup\$
  • \$\begingroup\$ Where do you make the distinction between what is an English versus a French word? \$\endgroup\$ – holroy May 11 '17 at 20:23
  • \$\begingroup\$ I have add the comment in the code. it is the line k= split_list[i] in words.words() \$\endgroup\$ – Nico Coallier May 11 '17 at 20:24
  • \$\begingroup\$ Stop changing the code after you're original code. You can comment (outside of code), but please don't change the code in the question. It's against the rules of Code Review. \$\endgroup\$ – holroy May 11 '17 at 20:37
  • \$\begingroup\$ Ohhh sorry , I didn't know , I'll be careful \$\endgroup\$ – Nico Coallier May 11 '17 at 20:38
  • 1
    \$\begingroup\$ @holroy Technically it is only against the rules if there is an answer. However, it's nicer to reviewers to not change the code in the question too much. If code is changed much it might be an indication that the question was posted too early. \$\endgroup\$ – Simon Forsberg May 11 '17 at 21:06
2
\$\begingroup\$

Here are some comments on your code. First some stylistic comments, and afterwards some related to the logic of your code.

  • Method names should follow snake_case – According to the PEP8 standard, your method should be named extract_sentence_english, and slightly better worded, extract_english_sentences
  • Naming is hard – Most of the names seems to be slightly random. Why log, split_list, df, dat... They are not conveying anything sensible, or indicate how you've processed the original sentences.
  • Add vertical space – You have some, but could have more vertical space.
  • Better comments – For example, related to the split_list (renamed to all_words and log (renamed to is_english_words), I would rather write something like this:

    # Make a list of all single words based upon the sentences
    all_words = [word for sentence in list_of_sentence
                         for word in sentence.split(' ')]
    
    # Make a corresponding list with True if the word is English,
    # or False if not
    is_english_words = [word in words.words() for word in all_words]
    

    Now the comments explain the contents of the list, and what is happening on the next code lines. I've also used a list comprehension to create the alternative to log, and with better variable names it is easier to see, and should not as expensive as the original for loop. By the way, range defaults to starting at 0 so it could simply have been range(len(split_list)).

    Same kind of logic (and list comprehension) could apply to the creation of df, which could be renamed to sentence_lengths.

  • Why all the DataFrame mumble jumble? – I don't really see the need for all the usage of DataFrame (and don't understand what is happening besides the reconnection of words and sentences). More on this in alternate solution below.

  • Please add space after commas, , – In some of the parameter listings you don't add space after the comma, which makes the code harder to read. Be consistent, and always add space after comma.

Alternative logic

If I've understood your request properly you are getting a whole lot of sentences, and want to return all the English sentences (and loose the French sentences?). You go about this task by removing everything which isn't a-z, splitting into words and checking against a dictionary, and if the English word frequency is above 0.75 you'll return that sentence (without punctuation).

Doing it that way eliminates on a good indicator of French sentences. French sentences uses loads of accented characters. English doesn't. That is a very good indicator (if the Facebook guys care to write those characters), which you are missing out on.

Your solution also checks against an entire dictionary, which is kind of expensive. A simpler solution would be to keep two smaller list of typical English words and French words, and make sure to remove non-French or non-English words from these lists. I know very little French, but here is an attempt at some trigger word lists:

english_words = ["you", "are", "not", "the", "is", "what", "who", "that"]
french_words = ["le", "ce", "un", "la", "et", "au", "à", "toute"]

The idea is to have enough specific typical words in either list, so that it eases detection of the language.

So how could this look in an alternate version? Well, here is some code for you to work with:

import re
import random

sample_sentences = ['hi there, how are you mister', 'some none triggering english sentence', 'And another long english statement', 'This is not plain English', 'say','bonjour mister','sauf','tu am      belle','I suis cool', 'Hey YOU', u'Worst trip.ever je, à deteste AirTransat! ce sont les pires', u"Toute personne a droit à l´éducation"]

ENGLISH_WORDS = ["you", "are", "not", "the", "is", "what", "who", "that"]
FRENCH_WORDS = ["le", "ce", "un", "la", "et", "au", u"à", u"é", "toute"]

REQUIRED_NUMBER_OF_WORDS = 1
REQUIRED_ENGLISH_PERCENTAGE = 0.75

def extract_english_sentences(sentences):

  # Compile a pattern of all non-alphanumerics in Unicode
  pattern = re.compile('\W+', re.UNICODE)

  result = []
  print sentences

  # Loop on each sentence, and determine if it English or not
  for sentence in sentences:
    # Get a simplified sentence with only unicode alphanumerics,
    # which are lowercased
    no_punctuation = pattern.sub(' ', sentence).lower()

    # Split into single words
    words_in_sentence = no_punctuation.split()
    number_words = len(words_in_sentence)

    print ("\n>{}<".format(no_punctuation.encode('utf8', 'replace')))

    is_english = None

    # Check whether word is in either trigger list
    for word in words_in_sentence:
      if word in FRENCH_WORDS: 
        print "in french words"
        is_english = False
        break

      if word in ENGLISH_WORDS:
        print "in english words"
        is_english = True
        break

    if is_english is not None and is_english:
      result.append(sentence)
      continue

    # If it didn't match anything in trigger list,
    # and we have more than 5 words, lets check the
    # percentage of English words
    if is_english is None and number_words > REQUIRED_NUMBER_OF_WORDS:
      # Count occurences of each word in the dictionary
      # give by words.words()
      #sum_english_words = sum(word in words.words() 
      #                          for word in words_in_sentence)

      # Don't have dictionary, guess... :-)
      sum_english_words = random.randint(1, number_words)
      print "{} vs {}".format(sum_english_words, number_words)

      if sum_english_words / number_words > REQUIRED_ENGLISH_PERCENTAGE: 
        print "Dictionary says English"
        result.append(sentence)
        continue

    if is_english is not None and not is_english:
      print "French!"
    else:
      print "Couldn't decide"

  return result

print extract_english_sentences(sample_sentences)

As I didn't have any words.words() available, I've commented out that segment, and use a random generator instead. This in combination a lot of print statements should show that in most cases it would continue with next sentence in a lot of the cases, and hopefully not fall through to the dictionary test or "Couldn't decide" very often.

When deployed to production all of the print statements should be removed. Do also note that this code is written in Python 2.7, so no parentheses are needed for the print statements, and some strings needs to be declared as unicode strings. But you didn't state which python version you are using, so I made this in the older variant. Hope this helps you along the way.

Alternate trigger word list tests

At the cost of looping twice through the words_in_sentence list, one could remove the inner for loop, and all other code related to is_english:

# Check whether word is in either trigger list
if any(word in FRENCH_WORDS for word in words_in_sentence):
  print "in french words"
  continue

if any(word in ENGLISH_WORDS for word in words_in_sentence):
  print "in english words"
  result.append(sentence)

Possible major speedup

I just realized that you could get a major speedup related to the dictionary lookup. It is very expensive to get the complete word list, words.words(), for every sentence. This should be done once, and created into a set, which then is used to check for existence within dictionary.

It's late in the night locally for me, so the following code is untested:

WORDS_SET = set(words.words())

...

sum_english_words = sum(word in WORDS_SET)

This should avoid the repeated loading of the words for every word of every sentence... Will try to test this tomorrow.

\$\endgroup\$
0
\$\begingroup\$

Amazing answer !!!

I am know able to sort all the english from the post I have extracted (processed 10 000 post in less then a minute).

NEST STEP is to make a class method to deal with bilingual (french/english) sentence.

  1. I want to include the function such as:
    • determine the dominant language of the post between french,english and spanish
    • extract the dominant language sentence (make extract_english_sentences more versatile...)

I have started working on the class and improve a bit the function you suggest . Instead of creating a list of words I have used french and english stopwords. I also added couple french words that were not in the stopwords and make the function return all the sentence so I can analyse seperatly futher on...

This suit my need for the moment but I am sharing the class so we can continue working on it for future needs. I think we could try to incorporate multiple language as suggested before.

I have incorporated the module into the class but for simplicity but it is probably incorrect...

class attribute_language_sentences():

    def __init__(self,sentences):
      self.sentences = sentences

    def attribution(self,REQUIRED_NUMBER_OF_WORDS= 1,REQUIRED_ENGLISH_PERCENTAGE= 0.85):

      ### This function will extract the english words from the sentences and return it into a list.

      from nltk.corpus import brown
      from nltk.corpus import stopwords
      import re
      import random

      # Compile a pattern of all non-alphanumerics in Unicode
      pattern = re.compile('\W+', re.UNICODE)

      # Making the list of words

      ENGLISH_WORDS = set(stopwords.words('english'))
      FRENCH_WORDS = set(stopwords.words('french'))
      FRENCH_WORDS_other = ["bravo","merci","mille","cent","dix","oui","non","bienvenue"]
      WORDS_SET = set(brown.words())

      result = []
      #print self.sentences
      index = 0

      # Loop on each sentence, and determine if it English or not
      for sentence in self.sentences:
        index = index + 1  
        # Get a simplified sentence with only unicode alphanumerics,
        #which are lowercased
        no_punctuation = pattern.sub(' ', sentence).lower()

        # Split into single words
        words_in_sentence = no_punctuation.split()
        number_words = len(words_in_sentence)

        #print ("\n>{}<".format(no_punctuation.encode('utf8', 'replace')))

        is_english = None

        # Check whether word is in either trigger list
        for word in words_in_sentence:
          if word in FRENCH_WORDS or word in FRENCH_WORDS_other: 
            #print "in french words"
            is_english = False
            break

          if word in ENGLISH_WORDS:
            #print "in english words"
            is_english = True
            break

        if is_english is not None and is_english:
          result.append((sentence,index,"English"))
          continue

        # If it didn't match anything in trigger list,
        # and we have more than 5 words, lets check the
        # percentage of English words
        if is_english is None and number_words > REQUIRED_NUMBER_OF_WORDS:
          # Count occurences of each word in the dictionary
          # give by words.words()
          sum_english_words = sum(word in WORDS_SET
                                    for word in words_in_sentence)

          #print "{} vs {}".format(sum_english_words, number_words)

          if sum_english_words / number_words > REQUIRED_ENGLISH_PERCENTAGE: 
            #print "Dictionary says English"
            result.append((sentence,index,"English"))
            continue

        if is_english is not None and not is_english:
            result.append((sentence,index,"French"))
        else:
            result.append((sentence,index,"NA"))

      return result

Thanks again for your comments and hope we can develop a class that could help other people in the future over the next weeks !

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.