**Example data can be downloaded below.**
I am working on a project which crunches plain text files (`.lst`).
The name of the file names (`fileName`) are important because I'll extract `node` (e.g. `abessijn`) and `component` (e.g. `WR-P-E-A`) from them into a dataframe. Examples:
abessijn.WR-P-E-A.lst
A-bom.WR-P-E-A.lst
acroniem.WR-P-E-C.lst
acroniem.WR-P-E-G.lst
adapter.WR-P-E-A.lst
adapter.WR-P-E-C.lst
adapter.WR-P-E-G.lst
Each file consists of one or more line. Each line consists of a sentence (inside `<sentence>` tags). Example (`abessijn.WR-P-E-A.lst`)
<sentence>Vooral mijn abessijn ruikt heerlijk kruidig .. : ) )</sentence>
<sentence>Mijn abessijn denkt daar heel anders over .. : ) ) Maar mijn kinderen richt ik ook niet af , zit niet in mijn bloed .</sentence>
From each line I extract the sentence, do some small modifications to it, and call it `sentence`. Up next is an element called `leftContext`, which takes the first part of the split between `node` (e.g. `abessijn`) and the sentence it came from. Finally, from `leftContext` I get `precedingWord`, which is the word preceding `node` in `sentence`, or the right most word in `leftContext` (with some limitations such as the option of a compound formed with a hyphen). Example:
ID | filename | node | component | precedingWord | leftContext | sentence
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 adapter.WR-P-P-F.lst adapter WR-P-P-F aanpassingseenheid Een aanpassingseenheid ( Een aanpassingseenheid ( adapter ) ,
2 adapter.WR-P-P-F.lst adapter WR-P-P-F toestel Het toestel ( Het toestel ( adapter ) draagt zorg voor de overbrenging van gegevens
3 adapter.WR-P-P-F.lst adapter WR-P-P-F de de aansluiting tussen de sensor en de de aansluiting tussen de sensor en de adapter ,
4 airbag.WS-U-E-A.lst airbag WS-U-E-A den ja voor den ja voor den airbag op te pompen eh :p
5 airbag.WS-U-E-A.lst airbag WS-U-E-A ne Dobby , als ze valt heeft ze dan wel al ne Dobby , als ze valt heeft ze dan wel al ne airbag hee
That dataframe is exported as `dataset.csv`.
After that, the intention of my project comes at hand: I create a frequency table that takes `node` and `precedingWord` into account. From a variable I define `neuter` and `non_neuter`, e.g (in Python)
neuter = ["het", "Het"]
non_neuter = ["de","De"]
and a rest category `unspecified`. When `precedingWord` is an item from the list, assign it to the variable. Example of a frequency table output:
node | neuter | nonNeuter | unspecified
-------------------------------------------------
A-bom 0 4 2
acroniem 3 0 2
act 3 2 1
The frequency list is exported as `frequencies.csv`.
I came up with the following script ([paste][2]) to do that.
import os, pandas as pd, numpy as np, regex as re
from glob import glob
from datetime import datetime
from html import unescape
start_time = datetime.now()
# Create empty dataframe with correct column names
columnNames = ["fileName", "component", "precedingWord", "node", "leftContext", "sentence" ]
df = pd.DataFrame(data=np.zeros((0,len(columnNames))), columns=columnNames)
# Create correct path where to fetch files
subdir = "rawdata"
path = os.path.abspath(os.path.join(os.getcwd(), os.pardir, subdir))
# "Cache" regex
# See http://stackoverflow.com/q/452104/1150683
p_filename = re.compile(r"[./\\]")
p_sentence = re.compile(r"<sentence>(.*?)</sentence>")
p_typography = re.compile(r" (?:(?=[.,:;?!) ])|(?<=\( ))")
p_non_graph = re.compile(r"[^\x21-\x7E\s]")
p_quote = re.compile(r"\"")
p_ellipsis = re.compile(r"\.{3}(?=[^ ])")
p_last_word = re.compile(r"^.*\b(?<!-)(\w+(?:-\w+)*)[^\w]*$", re.U)
# Loop files in folder
for file in glob(path+"\\*.lst"):
with open(file, encoding="utf-8") as f:
[n, c] = p_filename.split(file.lower())[-3:-1]
fn = ".".join([n, c])
for line in f:
s = p_sentence.search(unescape(line)).group(1)
s = s.lower()
s = p_typography.sub("", s)
s = p_non_graph.sub("", s)
s = p_quote.sub("'", s)
s = p_ellipsis.sub("... ", s)
if n in re.split(r"[ :?.,]", s):
lc = re.split(r"(^| )" + n + "( |[!\",.:;?})\]])", s)[0]
pw = p_last_word.sub("\\1", lc)
df = df.append([dict(fileName=fn, component=c,
precedingWord=pw, node=n,
leftContext=lc, sentence=s)])
continue
# Reset indices
df.reset_index(drop=True, inplace=True)
# Export dataset
df.to_csv("dataset/py-dataset.csv", sep="\t", encoding="utf-8")
# Let's make a frequency list
# Create new dataframe
# Define neuter and non_neuter
neuter = ["het"]
non_neuter = ["de"]
# Create crosstab
df.loc[df.precedingWord.isin(neuter), "gender"] = "neuter"
df.loc[df.precedingWord.isin(non_neuter), "gender"] = "non_neuter"
df.loc[df.precedingWord.isin(neuter + non_neuter)==0, "gender"] = "rest"
freqDf = pd.crosstab(df.node, df.gender)
freqDf.to_csv("dataset/py-frequencies.csv", sep="\t", encoding="utf-8")
# How long has the script been running?
time_difference = datetime.now() - start_time
print("Time difference of", time_difference)
*I'm running Python in 32 bit, because I'd like to use the [nltk module][3] in the future and they discourage users to use 64 bit.*
Seeing that the goal of Python was to make everything go smoothly, I was confused. **The goal is to beat a similar R script in execution speed.** I read Python was fast, so where did I go wrong? A colleague advised me to use Python which might be a faster case. I know you can't simply compare two languages that way, each has its merits. But in this specific case it seemed to me that Python ought to be faster in data crunching. My current Python code works, so there's no issue with `lower()` for me. Please tell me if I'm completely wrong. If someone could help me use some higher hierarchy functions as proposed on SO (imap, ifilter...) **and explaining them** that would be great.
What is the problem? Is Python slower in reading files and lines, or in doing regexes? Or is R simply better equipped to dealing with dataframes and can't it be beaten by pandas? *Or* is my code simply badly optimised and should Python indeed be the victor?
You can download the test data I used [*here*][4].
I'm a bit of a beginner, so please take the time to explain how they work and/or why they're better.
[1]: http://pastebin.com/gtCrAdMq
[2]: http://pastebin.com/rPFJKuBV
[3]: http://www.nltk.org/
[4]: http://bramvanroy.be/files/testdata.zip