Memory usage
What I notice most, is that this is bound to use a lot of memory.
First you load all the files (at once) into memory.
Then you copy over (with modifications) the data into another variable. And another. And another. ...
This creates a few lists, all with the same total memory size as the entire documents you have read.
Ex-pen-sive.
Generators/Iterators to the rescue!
In Python, iterators are really nice. They allow something like continuations and just-in-time calculation. This causes memory usage to be a lot lower, on the (perhaps) extra cost of a bit more CPU processing. But... I think in this case it will overall be a saving because you have a lot less memory usage, so probably also less cache-misses.
Let's see what we can do about that.
The last step
nrfinal = []
for j in nr:
rem = 0
outr = ''
for i in j:
if ord(i)>= 48 and ord(i)<=57:
rem += 1
if rem == 1:
outr = outr+ '#'
else:
rem = 0
outr = outr+i
nrfinal.append(outr)
The easiest step is to first make it a function.
def calc_nrfinal(inp):
retval = []
for j in nr:
rem = 0
outr = ''
for i in j:
if ord(i)>= 48 and ord(i)<=57:
rem += 1
if rem == 1:
outr = outr+ '#'
else:
rem = 0
outr = outr+i
retval.append(outr)
return retval
nrfinal = calc_nrfinal(nr)
Now this does not save us memory. But by rewriting it a little bit, it will become a generator.
def calc_nrfinal(inp):
for j in nr:
rem = 0
outr = ''
for i in j:
if ord(i)>= 48 and ord(i)<=57:
rem += 1
if rem == 1:
outr = outr+ '#'
else:
rem = 0
outr = outr+i
yield outr
nrfinal = calc_nrfinal(nr)
The downside is that you can only iterate over nrfinal once, but you only need to do it once.
The inner loop
One thing that bothers me is the inner loop here. ord(i)>=49 and ord(i)<=57) could be written as 49 <= ord(i) <= 57 (one of Pythons strengths!). But... I'd rather write
outr = re.sub(r'[0-9]+', '#', j)
Giving us
def calc_nrfinal(inp):
for j in nr:
yield re.sub(r'[0-9]+', '#', j)
nrfinal = calc_nrfinal(nr)
But now, the whole generator-function is a bit annoying, and I'd like to switch to ... (drumroll)
Generator comprehensions
Defining lots of separate generator functions can be really annoying, especially when the body of the generator is quite small. There's a solution for that: generator comprehensions.
def f(iterable):
for val in iterable:
yield g(val)
k = f()
Can be rewritten as
k = (g(val) for val in iterable)
It even supports multiple fors and ifs inside the loops.
Anyway, I disgres. What I mean is:
nrfinal = (re.sub(r'[0-9]+', '#', j) for j in nr)
And, nrfinal will take almost no memory at all, because all the values are evaluated just-in-time. The downside is you can not iterate twice (because it is not stored).
Continuing in similar fashion:
Here I did the same for a couple of the pieces.
# -*- coding: utf-8 -*-
from __future__ import print_function
import os, codecs, re, string, mysql
import mysql.connector
'''Reading files with txt extension'''
y_ = ""
for root, dirs, files in os.walk("/Users/Documents/source-document/part1"):
for file in files:
if file.endswith(".txt"):
x_ = codecs.open(os.path.join(root,file),"r", "utf-8-sig")
for lines in x_.readlines():
y_ = y_ + lines
#print(tokenized_docs)
'''Tokenizing sentences of the text files'''
from nltk.tokenize import sent_tokenize
raw_docs = sent_tokenize(y_)
tokenized_docs = [sent_tokenize(y_) for sent in raw_docs]
'''Removing stop words'''
stopword_removed_sentences = []
from nltk.corpus import stopwords
stopset = stopwords.words("English")
for i in tokenized_docs[0]:
tokenized_docs = ' '.join([word for word in i.split() if word not in stopset])
stopword_removed_sentences.append(tokenized_docs)
''' Removing punctuation marks'''
regex = re.compile('[%s]' % re.escape(string.punctuation))
nw = []
for review in stopword_removed_sentences:
new_review = ''
for token in review:
new_token = regex.sub(u'', token)
if not new_token == u'':
new_review += new_token
nw.append(new_review)
lw = (i.lower() for i in nw)
nr = (re.sub(r'[^\[\]]+(?=\])', '#', j) for j in lw)
nrfinal = (re.sub('[0-9]+', '#', j) for j in nr)
'''Inserting into database'''
def connect():
for j in nrfinal:
conn = mysql.connector.connect(user = 'root', password = '', unix_socket = "/tmp/mysql.sock", database = 'Thesis' )
cursor = conn.cursor()
cursor.execute("""INSERT INTO splitted_sentences(sentence_id, splitted_sentences) VALUES(%s, %s)""",(cursor.lastrowid,j))
conn.commit()
conn.close()
if __name__ == '__main__':
connect()
Unnecessary looping
Here I got a bit lost, due to the double loop in nw and new_review... For this, I first need to understand the contents of stopword_removed_sentences. Let's look a few lines up:
stopword_removed_sentences = []
from nltk.corpus import stopwords
stopset = stopwords.words("English")
for i in tokenized_docs[0]:
tokenized_docs = ' '.join([word for word in i.split() if word not in stopset])
stopword_removed_sentences.append(tokenized_docs)
What this tells me:
stopwords_removed_sentences is a list.- The value of each of the lists is a string (
' '.join(...) must be a list).
Ok, for now I know enough. This tells me:
review is a string.token is a 1-character substring of review.
Looking back to the troublesome code:
regex = re.compile('[%s]' % re.escape(string.punctuation))
nw = []
for review in stopword_removed_sentences:
new_review = ''
for token in review:
new_token = regex.sub(u'', token)
if not new_token == u'':
new_review += new_token
nw.append(new_review)
This means we call regex.sub for all characters of review.
I have three different solutions:
new_review = regex.sub(u'', review)
Which reads very nice! Or
new_review = ''.join(token for token in review if token in string.punctuation)
Which is probably a bit more expensive due to the looping in Python (instead of C). Or,
new_review = review.translate(None, string.punctuation)
Which also saves us pre-compiling a regex. More readable, I think. Any are fine, but I think the review.translate is the easiest to look at.
Using that:
# -*- coding: utf-8 -*-
from __future__ import print_function
import os, codecs, re, string, mysql
import mysql.connector
'''Reading files with txt extension'''
y_ = ""
for root, dirs, files in os.walk("/Users/Documents/source-document/part1"):
for file in files:
if file.endswith(".txt"):
x_ = codecs.open(os.path.join(root,file),"r", "utf-8-sig")
for lines in x_.readlines():
y_ = y_ + lines
#print(tokenized_docs)
'''Tokenizing sentences of the text files'''
from nltk.tokenize import sent_tokenize
raw_docs = sent_tokenize(y_)
tokenized_docs = [sent_tokenize(y_) for sent in raw_docs]
'''Removing stop words'''
stopword_removed_sentences = []
from nltk.corpus import stopwords
stopset = stopwords.words("English")
for i in tokenized_docs[0]:
tokenized_docs = ' '.join([word for word in i.split() if word not in stopset])
stopword_removed_sentences.append(tokenized_docs)
nw = (j.translate(None, string.punctuation) for j in stopword_removed_sentences)
lw = (i.lower() for i in nw)
nr = (re.sub(r'[^\[\]]+(?=\])', '#', j) for j in lw)
nrfinal = (re.sub('[0-9]+', '#', j) for j in nr)
'''Inserting into database'''
def connect():
for j in nrfinal:
conn = mysql.connector.connect(user = 'root', password = '', unix_socket = "/tmp/mysql.sock", database = 'Thesis' )
cursor = conn.cursor()
cursor.execute("""INSERT INTO splitted_sentences(sentence_id, splitted_sentences) VALUES(%s, %s)""",(cursor.lastrowid,j))
conn.commit()
conn.close()
if __name__ == '__main__':
connect()
Quadratic run-time in the number of sentences.
Look at this code:
raw_docs = sent_tokenize(y_)
tokenized_docs = [sent_tokenize(y_) for sent in raw_docs]
First, you get the sentences from y_. Then, you again get the sentences from y_, but as many times as there are sentences in y_. Then, you only use the first item from the second set.
raw_docs = sent_tokenize(y_)
tokenized_docs = [sent_tokenize(y_) for sent in raw_docs]
'''Removing stop words'''
stopword_removed_sentences = []
from nltk.corpus import stopwords
stopset = stopwords.words("English")
for i in tokenized_docs[0]:
tokenized_docs = ' '.join([word for word in i.split() if word not in stopset])
stopword_removed_sentences.append(tokenized_docs)
Becomes
raw_docs = sent_tokenize(y_)
tokenized_docs = [sent_tokenize(y_) for sent in raw_docs]
'''Removing stop words'''
stopword_removed_sentences = []
from nltk.corpus import stopwords
stopset = stopwords.words("English")
for i in raw_docs:
tokenized_docs = ' '.join([word for word in i.split() if word not in stopset])
stopword_removed_sentences.append(tokenized_docs)
And now switch to generators again.
But, we also want generators
raw_docs = sent_tokenize(y_)
stopword_removed_sentences = (' '.join(word for word in i.split() if word not in stopset) for i in raw_docs)
Now this becomes a bit bulky, though. The inner text is quite long. Let's make that a function.
def strip_stopwords(sentence):
return ' '.join(word for word in sentence.split() if word not in stopset)
stopword_removed_sentences = (strip_stopwords(sentence) for sentence in raw_docs)
Recognizing a pattern.
Now, we end up with
...
stopword_removed_sentences = (strip_stopwords(sentence) for sentence in raw_docs)
nw = (j.translate(None, string.punctuation) for j in stopword_removed_sentences)
lw = (i.lower() for i in nw)
nr = (re.sub(r'[^\[\]]+(?=\])', '#', j) for j in lw)
nrfinal = (re.sub('[0-9]+', '#', j) for j in nr)
...
Let's see if we can make that more readable...
actions = [
strip_stopwords,
lambda sentence: sentence.translate(None, string.punctuation),
str.lower,
lambda blob: re.sub(r'[^\[\]]+(?=\])', '#', blob),
lambda blob: re.sub(r'[0-9]+', '#', blob),
]
def apply_all_actions(val):
for action in actions:
val = action(val)
return action
nrfinal = (apply_all_actions(val) for val in raw_docs)
(Here we abuse the fact that str.lower is the same as lambda f: f.lower() if it is certain that f is a string.)
Now it is really clear that nrfinal is constructed from raw_docs by a sequence of simple transformations on the elements. If you want, you can define functions for all of the separate actions, given
actions = [
strip_stopwords,
remove_punctuation,
lowercase,
(some magic name, I don't know what),
squash_numbers_to_hash,
]
But, this is a matter of choice.
Connection management.
Now, for something completely different.
def connect():
for j in nrfinal:
conn = mysql.connector.connect(user = 'root', password = '', unix_socket = "/tmp/mysql.sock", database = 'Thesis' )
cursor = conn.cursor()
cursor.execute("""INSERT INTO splitted_sentences(sentence_id, splitted_sentences) VALUES(%s, %s)""",(cursor.lastrowid,j))
conn.commit()
conn.close()
Let's read what this does:
- for every sentence in
nrfinal, do the following:
** Create a connection to a mysql host.
** Create a cursor.
** Insert a value.
** Commit the connection.
** Close the connection.
Connecting is expensive. Why not do that outside the loop?
def connect():
conn = mysql.connector.connect(user = 'root', password = '', unix_socket = "/tmp/mysql.sock", database = 'Thesis' )
for j in nrfinal:
cursor = conn.cursor()
cursor.execute("""INSERT INTO splitted_sentences(sentence_id, splitted_sentences) VALUES(%s, %s)""",(cursor.lastrowid,j))
conn.commit()
conn.close()
I'm hesitant to do this for the cursor, because that might break things. Why, you ask? Because of the cursor.lastrowid. Before a statement has been made, it is None. So we must mimic that.
def connect():
conn = mysql.connector.connect(user = 'root', password = '', unix_socket = "/tmp/mysql.sock", database = 'Thesis' )
cursor = conn.cursor()
for j in nrfinal:
cursor.execute("""INSERT INTO splitted_sentences(sentence_id, splitted_sentences) VALUES(%s, %s)""",(None,j))
conn.commit()
conn.close()
In fact, I think you want to make sentence_id an AUTO_INCREMENT. Then, write
def connect():
conn = mysql.connector.connect(user = 'root', password = '', unix_socket = "/tmp/mysql.sock", database = 'Thesis' )
cursor = conn.cursor()
for j in nrfinal:
cursor.execute("""INSERT INTO splitted_sentences(splitted_sentences) VALUES (%s)""",(j,))
conn.commit()
conn.close()
Executing many statements
The mysql.connector allows executemany.
nrtuples = ((j,) for j in nrfinal)
def connect():
conn = mysql.connector.connect(user = 'root', password = '', unix_socket = "/tmp/mysql.sock", database = 'Thesis' )
cursor = conn.cursor()
cursor.executemany("""INSERT INTO splitted_sentences(splitted_sentences) VALUES (%s)""", nrtuples)
conn.commit()
conn.close()
That should take care of the memory usage.
Loading all the files
y_ = ""
for root, dirs, files in os.walk("/Users/Documents/source-document/part1"):
for file in files:
if file.endswith(".txt"):
x_ = codecs.open(os.path.join(root,file),"r", "utf-8-sig")
for lines in x_.readlines():
y_ = y_ + lines
You load all files into memory at once. This is expansive. Rather not. Let's assume that there are no cross-file sentences going on (at least I hope not!).
def get_sentences():
for root, dirs, files in os.walk("/Users/Documents/source-document/part1"):
for file in files:
if file.endswith(".txt"):
x_ = codecs.open(os.path.join(root,file),"r", "utf-8-sig")
yield x_.read()
If you also know that there are no cross-line sentences, you can use
def get_sentences():
for root, dirs, files in os.walk("/Users/Documents/source-document/part1"):
for file in files:
if file.endswith(".txt"):
x_ = codecs.open(os.path.join(root,file),"r", "utf-8-sig")
for lines in x_.readlines():
yield lines
I'm not that big a fan of this deep nesting, and would prefer to also write a generator which yields the .txt files. But I hope you can continue from here.
Also, maybe you should use with codecs.open(....) as x_, that way the file gets closed when the iteration is done.
Further remarks.
There are certain style guidelines.
Keep your imports at the top of the file.
- First a module-level docstring (if present).
- Then, the imports from the standard library. (preferably sorted).
- Then, imports from other libraries.
- Then, imports from other parts of the project.
- Then class/function definitions (in whatever order you please)
- Finally, only a
if __name__ == '__main__': block. Preferably containing only a function-call to main() in the module.
docstrings are no comments. Comments are no docstrings.
In the code you have ''' some useful text'''. But you actually want to use a comment: # some useful text. Know to disambiguate between those.
Multi-line comments? Just start every line with #.
