# Markov text generator program in Python

This is my first non-trivial program in my Python. I am coming from a Java background and I might have messed up or ignored some conventions. I would like to hear feedback on my code.

import nltk
import random

file_name = input()
file = open(file_name, "r", encoding="utf-8")
# nltk.trigrams returns a list of 3-tuples
file.close()

model = {}
for trigram in trigrams:
head = trigram[0] + " " + trigram[1]
tail = trigram[2]

sentence_ending_punctuation = (".", "!", "?")
for key in model.keys():
if key[0].isupper() and not key.split(" ")[0].endswith(sentence_ending_punctuation):

# Generate 10 pseudo-sentences based on model
for _ in range(10):
tokens = []
# Chooses a random starting head from list
while True:
# Randomly select elements from list taking their weights into account
most_probable_tail = random.choices(possible_tails, weights, k=1)[0]
# print("Most probable tail: ", most_probable_tail)

if most_probable_tail.endswith(sentence_ending_punctuation) and len(tokens) >= 5:
tokens.append(most_probable_tail)
# print("Chosen tail and ending sentence: ", most_probable_tail)
break
elif not most_probable_tail.endswith(sentence_ending_punctuation):
tokens.append(most_probable_tail)
# print("Chosen tail: ", most_probable_tail)
elif most_probable_tail.endswith(sentence_ending_punctuation) and len(tokens) < 5:
# print("Ignoring tail: ", most_probable_tail)
tokens = []

pseudo_sentence = " ".join(tokens)
print(pseudo_sentence)

$$`$$

Here's a kinda stream-of-consciousness review:

input() takes an argument, typically a prompt or question, so the user knows enter something. You could also check sys.argv for a filename on the command line.

Avoid using names of built in functions (e.g., file) for a variable name.

open() is a context manager, so it can be used in a with statement. In the current code, an exception anywhere in list(nltk.trigrams(file.read().split())) could result in the file being left open. Using a with statement ensures the file is closed:

with open(file_name, "r", encoding="utf-8") as input_file:
# nltk.trigrams returns a list of 3-tuples

### model data structure

Get to know the collections module in the standard library. Here, defaultdict and Counter would be useful.

A tuple can be a dictionary key, so concatenating the first two items in a trigram is not necessary.

import collections

model = collections.defaultdict(collections.Counter)

for trigram in trigrams:
model[trigram[:2]].update(trigram[2:])

Or, to be a little clearer:

for word1, word2, word3 in trigrams:
model[(word1, word2)].update((word3,))

Then, first words can be collected at the same time:

for word1, word2, word3 in trigrams:
model[(word1, word2)].update((word3,))

if word1[0].isupper() and not word1.endswith(sentence_ending_punctuation):

### generating sentences

If you intend to make lots of sentences, recreating lists of candidate tails and their weights every time may slow things down. Consider restructuring the model to better suit the end use. This can be done all at once:

new_model = {}
for key, counter in model.items():
new_model[key] = (list(counter.keys()), list(accumulate(counter.values())))

most_probable_tail is a misnomer; it's the chosen tail.

The logic of the if ... elif ... elif statement is not easy to follow, and .endswith(sentence_ending_punctuation) potentially gets called three times.

tokens = []
# Chooses a random starting head from list

while True:

chosen_tail = random.choices(possible_tails, cum_weights=weights, k=1)

if not chosen_tail.endswith(sentence_ending_punctuation):
tokens.append(most_probable_tail)

else:
if len(tokens) >= 5:
tokens.append(chosen_tail)
break

else:
tokens = []

pseudo_sentence = " ".join(tokens)

### other thoughts

You can store the first word of a sentence in the model with a key of ('', '') and the second word with a key of ('', word1). Then they don't need to be handled separately when generating the sentences.

NLTK has a function word_tokenize() that might be helpful. It breaks text into tokens such as words, numbers, and punctuation. It recognized things like Dr.

• I double-checked after reading your answer, and file has been removed from the built-in functions somewhere between Python 2.7 and 3.5, neither of which is still supported. There's no harm avoiding using it as a variable using a somewhat recent version of Python but it shouldn't wreck havoc when that happens.
– Anab
Aug 31 '21 at 21:00

In the following, I assume that you use Python3 rather than Python2, though I can't say for sure that it makes a difference.

First, when you parse your file, you could use a context manager:

file_name = input()
with open(file_name, "r", encoding="utf-8") as file:
# nltk.trigrams returns a list of 3-tuples

That way, even if an exception is raised by the assignment to trigrams (in either of the 4 method calls), the file stream is properly closed.

You could simplify your model generation using defaultdict from the collections package. Your way of doing it is actually wrong and I can't say for sure that this option is more pythonic but it might be interesting to know.

import collections
model = collections.defaultdict(lambda : collections.defaultdict(int))
for trigram in trigrams:
head = trigram[0] + " " + trigram[1]
tail = trigram[2]

This does not change the behavior of your algorithm, it just feels a bit simpler to me.

But you can do something more memory-efficient:

import collections
model = collections.defaultdict(lambda : collections.defaultdict(int))
file_name = input()
with open(file_name, "r", encoding="utf-8") as file:
# nltk.trigrams returns a list of 3-tuples