Example data can be downloaded below.

The name of the file names (fileName) are important because I'll extract node (e.g. abessijn) and component (e.g. WR-P-E-A) from them into a dataframe. Examples:

abessijn.WR-P-E-A.lst
A-bom.WR-P-E-A.lst
acroniem.WR-P-E-C.lst
acroniem.WR-P-E-G.lst
adapter.WR-P-E-A.lst
adapter.WR-P-E-C.lst
adapter.WR-P-E-G.lst

<sentence>Vooral mijn abessijn ruikt heerlijk kruidig .. : ) )</sentence>
<sentence>Mijn abessijn denkt daar heel anders over .. : ) ) Maar mijn kinderen richt ik ook niet af , zit niet in mijn bloed .</sentence>

ID | filename             | node | component | precedingWord      | leftContext                               |  sentence
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1    adapter.WR-P-P-F.lst  adapter  WR-P-P-F   aanpassingseenheid  Een aanpassingseenheid (                      Een aanpassingseenheid ( adapter ) , 
2    adapter.WR-P-P-F.lst  adapter  WR-P-P-F   toestel             Het toestel (                                 Het toestel ( adapter ) draagt zorg voor de overbrenging van gegevens
3    adapter.WR-P-P-F.lst  adapter  WR-P-P-F   de                  de aansluiting tussen de sensor en de         de aansluiting tussen de sensor en de adapter , 
4    airbag.WS-U-E-A.lst   airbag   WS-U-E-A   den                 ja voor den                                   ja voor den airbag op te pompen eh :p
5    airbag.WS-U-E-A.lst   airbag   WS-U-E-A   ne                  Dobby , als ze valt heeft ze dan wel al ne    Dobby , als ze valt heeft ze dan wel al ne airbag hee

neuter = ["het", "Het"]
non_neuter = ["de","De"]

node    |   neuter   | nonNeuter   | unspecified
-------------------------------------------------
A-bom       0          4             2
acroniem    3          0             2
act         3          2             1

I came up with the following script (paste) to do that.

import os, pandas as pd, numpy as np, regex as re

from glob import glob
from datetime import datetime
from html import unescape

start_time = datetime.now()

# Create empty dataframe with correct column names
columnNames = ["fileName", "component", "precedingWord", "node", "leftContext", "sentence" ]
df = pd.DataFrame(data=np.zeros((0,len(columnNames))), columns=columnNames)

# Create correct path where to fetch files
subdir = "rawdata"
path = os.path.abspath(os.path.join(os.getcwd(), os.pardir, subdir))

# "Cache" regex
# See http://stackoverflow.com/q/452104/1150683
p_filename = re.compile(r"[./\\]")

p_sentence = re.compile(r"<sentence>(.*?)</sentence>")
p_typography = re.compile(r" (?:(?=[.,:;?!) ])|(?<=\( ))")
p_non_graph = re.compile(r"[^\x21-\x7E\s]")
p_quote = re.compile(r"\"")
p_ellipsis = re.compile(r"\.{3}(?=[^ ])")

p_last_word = re.compile(r"^.*\b(?<!-)(\w+(?:-\w+)*)[^\w]*$", re.U)

# Loop files in folder
for file in glob(path+"\\*.lst"):
    with open(file, encoding="utf-8") as f:
        [n, c] = p_filename.split(file.lower())[-3:-1]
        fn = ".".join([n, c])
        for line in f:
            s = p_sentence.search(unescape(line)).group(1)
            s = s.lower()
            s = p_typography.sub("", s)
            s = p_non_graph.sub("", s)
            s = p_quote.sub("'", s)
            s = p_ellipsis.sub("... ", s)

            if n in re.split(r"[ :?.,]", s):
                lc = re.split(r"(^| )" + n + "( |[!\",.:;?})\]])", s)[0]
    
                pw = p_last_word.sub("\\1", lc)
    
                df = df.append([dict(fileName=fn, component=c, 
                                   precedingWord=pw, node=n, 
                                   leftContext=lc, sentence=s)])
            continue

# Reset indices
df.reset_index(drop=True, inplace=True)

# Export dataset
df.to_csv("dataset/py-dataset.csv", sep="\t", encoding="utf-8")

# Let's make a frequency list
# Create new dataframe

# Define neuter and non_neuter
neuter = ["het"]
non_neuter = ["de"]

# Create crosstab
df.loc[df.precedingWord.isin(neuter), "gender"] = "neuter"
df.loc[df.precedingWord.isin(non_neuter), "gender"] = "non_neuter"
df.loc[df.precedingWord.isin(neuter + non_neuter)==0, "gender"] = "rest"

freqDf = pd.crosstab(df.node, df.gender)

freqDf.to_csv("dataset/py-frequencies.csv", sep="\t", encoding="utf-8")

# How long has the script been running?
time_difference = datetime.now() - start_time
print("Time difference of", time_difference)

The name of the file names (fileName) are important because I'll extract node (e.g. abessijn) and component (e.g. WR-P-E-A) from them into a dataframe.

Examples:

abessijn.WR-P-E-A.lst
A-bom.WR-P-E-A.lst
acroniem.WR-P-E-C.lst
acroniem.WR-P-E-G.lst
adapter.WR-P-E-A.lst
adapter.WR-P-E-C.lst
adapter.WR-P-E-G.lst

<sentence>Vooral mijn abessijn ruikt heerlijk kruidig .. : ) )</sentence>
<sentence>Mijn abessijn denkt daar heel anders over .. : ) ) Maar mijn kinderen richt ik ook niet af , zit niet in mijn bloed .</sentence>

ID | filename             | node | component | precedingWord      | leftContext                               |  sentence
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1    adapter.WR-P-P-F.lst  adapter  WR-P-P-F   aanpassingseenheid  Een aanpassingseenheid (                      Een aanpassingseenheid ( adapter ) , 
2    adapter.WR-P-P-F.lst  adapter  WR-P-P-F   toestel             Het toestel (                                 Het toestel ( adapter ) draagt zorg voor de overbrenging van gegevens
3    adapter.WR-P-P-F.lst  adapter  WR-P-P-F   de                  de aansluiting tussen de sensor en de         de aansluiting tussen de sensor en de adapter , 
4    airbag.WS-U-E-A.lst   airbag   WS-U-E-A   den                 ja voor den                                   ja voor den airbag op te pompen eh :p
5    airbag.WS-U-E-A.lst   airbag   WS-U-E-A   ne                  Dobby , als ze valt heeft ze dan wel al ne    Dobby , als ze valt heeft ze dan wel al ne airbag hee

neuter = ["het", "Het"]
non_neuter = ["de","De"]

node    |   neuter   | nonNeuter   | unspecified
-------------------------------------------------
A-bom       0          4             2
acroniem    3          0             2
act         3          2             1

I came up with the following script (paste) to do that:

import os, pandas as pd, numpy as np, regex as re

from glob import glob
from datetime import datetime
from html import unescape

start_time = datetime.now()

# Create empty dataframe with correct column names
columnNames = ["fileName", "component", "precedingWord", "node", "leftContext", "sentence" ]
df = pd.DataFrame(data=np.zeros((0,len(columnNames))), columns=columnNames)

# Create correct path where to fetch files
subdir = "rawdata"
path = os.path.abspath(os.path.join(os.getcwd(), os.pardir, subdir))

# "Cache" regex
# See http://stackoverflow.com/q/452104/1150683
p_filename = re.compile(r"[./\\]")

p_sentence = re.compile(r"<sentence>(.*?)</sentence>")
p_typography = re.compile(r" (?:(?=[.,:;?!) ])|(?<=\( ))")
p_non_graph = re.compile(r"[^\x21-\x7E\s]")
p_quote = re.compile(r"\"")
p_ellipsis = re.compile(r"\.{3}(?=[^ ])")

p_last_word = re.compile(r"^.*\b(?<!-)(\w+(?:-\w+)*)[^\w]*$", re.U)

# Loop files in folder
for file in glob(path+"\\*.lst"):
    with open(file, encoding="utf-8") as f:
        [n, c] = p_filename.split(file.lower())[-3:-1]
        fn = ".".join([n, c])
        for line in f:
            s = p_sentence.search(unescape(line)).group(1)
            s = s.lower()
            s = p_typography.sub("", s)
            s = p_non_graph.sub("", s)
            s = p_quote.sub("'", s)
            s = p_ellipsis.sub("... ", s)

            if n in re.split(r"[ :?.,]", s):
                lc = re.split(r"(^| )" + n + "( |[!\",.:;?})\]])", s)[0]
    
                pw = p_last_word.sub("\\1", lc)
    
                df = df.append([dict(fileName=fn, component=c, 
                                   precedingWord=pw, node=n, 
                                   leftContext=lc, sentence=s)])
            continue

# Reset indices
df.reset_index(drop=True, inplace=True)

# Export dataset
df.to_csv("dataset/py-dataset.csv", sep="\t", encoding="utf-8")

# Let's make a frequency list
# Create new dataframe

# Define neuter and non_neuter
neuter = ["het"]
non_neuter = ["de"]

# Create crosstab
df.loc[df.precedingWord.isin(neuter), "gender"] = "neuter"
df.loc[df.precedingWord.isin(non_neuter), "gender"] = "non_neuter"
df.loc[df.precedingWord.isin(neuter + non_neuter)==0, "gender"] = "rest"

freqDf = pd.crosstab(df.node, df.gender)

freqDf.to_csv("dataset/py-frequencies.csv", sep="\t", encoding="utf-8")

# How long has the script been running?
time_difference = datetime.now() - start_time
print("Time difference of", time_difference)

Seeing that the goal of Python was to make everything go smoothly, I was confused. The goal is to beat a similar R script in execution speed. I read Python was fast, so where did I go wrong? A colleague advised me to use Python which might be a faster case. I know you can't simply compare two languages that way, each has its merits. But in this specific case it seemed to me that Python ought to be faster in data crunching. My current Python code works, so there's no issue with lower() for me. Please tell me if I'm completely wrong. If someone could help me use some higher hierarchy functions as proposed on SO (imap, ifilter...) and explaining them that would be great.

Seeing that the goal of Python was to make everything go smoothly, I was confused. The goal is to beat a similar R script in execution speed. I read Python was fast, so where did I go wrong? A colleague advised me to use Python which might be a faster case. I know you can't simply compare two languages that way, each has its merits. But in this specific case it seemed to me that Python ought to be faster in data crunching. I figured that when both scripts are equally well optimised, Python should be faster in this case. However, my current Python code is badly optimised whereas my R script is decent enough. 

Please tell me if I'm completely wrong. If someone could help me use some higher hierarchy functions as proposed on SO (imap, ifilter...) and explaining them that would be great.