#Styling
Some minor edits, but you should import each module on a new line, especially if you're assigning them different names.
import os
import numpy as np
import pandas as pd
It seems redundant here to assign subdir
for just a single use. If there's plans to use it later sure, but as it stands this should be fine.
path = os.path.abspath(os.path.join(os.getcwd(), os.pardir, "rawdata"))
I know you said that you wanted to save on performance, but when you instantly call this function anyway you're always going to run these imports, so they should be at the top with the rest.
import regex as re
from html import unescape
The p_quote
regex string is unnecessary when you can just use '"'
. Python sees no real difference between strings bound in single or double quotes, meaning you can easily reference them like this.
p_quote = re.compile('"')
Don't use path + '/' + filename
to set a file path. Use the os
module. It performs certain operations based on the OS Python is being run on. os.path.join(path, filename)
will work cross platform.
Also there's no real point to opening the file in r+
mode, as that allows you to read and write but all you ever do is read the file anyway.
As for your naming conundrum, I'd just use the actual column names. They're not going to actually clash since you call the columns with strings so there's a clear delineation from what is a list and what's a string. And what can be clearer than using the same name. Also generally you can leave list
out of a variable name except if you actually need to distinguish it from others.
The exception to both in your case of course being that you have a separate list of filenames, so I've given that a different name but you might find a different name more suitable.
filenameDataList.append(filenameData)
components.append(component)
precedingWords.append(precedingWord)
nodes.append(node)
leftContexts.append(leftContext)
sentences.append(sentence)
...
df['sentence'] = sentences
#Minor Performance
These are two notes that wont significantly reduce your speed from file to file because they're not affecting the regex but they will save you a fraction of a second per file which can build up if you're running it on many files.
You don't need to set n
and c
as part of a list, you can just comma separate them and python will understand. And it's faster to not create an unneeded list.
[n, c] = p_filename.split(filename.lower())[-3:-1]
There's also no reason to use join
here. It's slower than just string concatenation since it's just two items and you're creating the list for the sake of the join.
fn = n + "." + c
#Refactoring
I would put all the import
s together as I said above. Also I would put all the regex compilations near the top. They're effectively a constant so it's best to always keep constants near the top of the file rather than in a function where they'll be recreated the exact same (not that this affects your performance). This would mean that you have this all before your data frame is created:
import numpy as np
import os
import pandas as pd
import regex as re
from html import unescape
from datetime import datetime
start_time = datetime.now()
p_filename = re.compile(r"[./\\]")
p_last_word = re.compile(r"^.*\b(?<!-)(\w+(?:-\w+)*)[^\w]*$", re.U)
p_sentence = re.compile(r"<sentence>(.*?)</sentence>")
p_typography = re.compile(r" (?:(?=[.,:;?!) ])|(?<=\( ))")
p_non_graph = re.compile(r"[^\x21-\x7E\s]")
p_quote = re.compile(r"\"")
p_ellipsis = re.compile(r"\.{3}(?=[^ ])")
As for splitting up functions, I don't personally think that's necessary as your functions aren't overly long and it's a relatively specific process without a lot of repeating code, so I would keep to the two separate ones you have here.