At a high level, I think you're overusing regexes. You can split stuff into words earlier and then clean them, and you don't need to treat newlines as an explicit token. If you do want to split them as an explicit token called newline
to protect it from future regexes, you can handle that when you're constructing the markov chain by having a dedicated class, or just a string \n
.
class NewlineToken:
pass
I would take your input text and split it into words first, but do it a line at a time so I don't need to read the whole file into memory. Also, don't use file
as the name of a temporary file since it shadows a builtin.
The split
method when called with no arguments splits on whitespace.
with open("corpus.txt") as fh:
for line in fh:
words = line.split()
# do stuff with words
After that you can strip punctuation from your words and lowercase it.
word.lower().translate(None, string.punctuation)
I'd probably wrap that all up in a generator that looks something like this.
def stripped_words(path):
with open(path) as fh:
for line in fh:
words = line.split()
for word in words:
yield word.lower().translate(
None,
string.punctuation
)
More about string.punctuation
here https://stackoverflow.com/questions/265960/best-way-to-strip-punctuation-from-a-string-in-python
The markov chain construction is good. zip
automatically truncates the longer iterator and .setdefault
is really useful. I actually didn't know about that until reading this. You'd have to modify it a little bit if you're iterating over words from your file rather than working with an array.
This line selects a word uniformly at random regardless of how many times it appears in the text rather than selecting a word with probability equal to its frequency. I'm not sure what the requirements are, but it is something that jumped out at me.
# more frequent words aren't more likely to be chosen here
current_word = random.choice(list(markov_chain.keys()))
Some miscellaneous comments:
You don't need to escape everything in your first character class.
text = re.sub(r'([,.!?;:])',r' \1 ', text)
In this line
text = re.sub(r'[^a-zA-Z0-9|\,|\.|\!|\?|\;|\:]',r' ',text)
|
is part of your character class, I'm not sure whether you intended it because it isn't escaped like your other non-alphanumeric characters are.