I suspect there might be a much more efficient way to accomplish this task. I would appreciate a more clever hint for performance increase. I am also looking for feedback on style and simplification where appropriate.
I assume that the set of what counts as English words is a given and that I don't need to do something clever like an actual dictionary or use Google n-grams. That being said, the results could differ depending on the domain and also what is accepted as a word.
The original problem set M=3 (3-gram characters), which makes the results not so interesting (ans=
[(u'the', 191279), (u'and', 92292), (u'ing', 57323), (u'her', 44627), (u'hat', 38609)]).
#gets the top N most common substrings of M characters in English words from nltk import FreqDist from nltk.corpus import gutenberg def get_words(): for fileid in gutenberg.fileids(): for word in gutenberg.words(fileid): yield word def ngrams(N, which_list, strict=False): list_size = len(which_list) stop = list_size if strict: stop -= (N - 1) for i in xrange(0, stop): element = which_list[i] ngram = [element] j = 1 index = j + i while j < N and index < list_size: ngram.append(which_list[index]) j += 1 index += 1 yield ''.join(ngram) def m_most_common_ngram_chars(M=5, N=3): n_grams =  for word in get_words(): for ngram in ngrams(N, word, strict=True): n_grams.append(ngram) f = FreqDist(n_grams) return f.most_common(M) l = m_most_common_ngram_chars(M=5, N=3) print l