0
\$\begingroup\$

I wrote this script to figure out which users have the largest vocabulary on my irc server. I would like to see if there's any way to clarify what I consider a very quick script. It outputs a CSV without a header of user, their unique words, most common words and the use of those words.

import sys
import json
import string

from collections import Counter, defaultdict
from nltk.corpus import stopwords

english_stopwords = stopwords.words('english')


x = defaultdict(Counter)
removepunct = str.maketrans('', '', string.punctuation)

for file in sys.argv[1:]:
    with open(file) as f:
        for line in f.readlines():
            line = line.strip().lower()
            # check for system msg in first 20 chars
            if '--' in line[:40]:
                continue
            line = line.translate(removepunct)
            words = line.split()
            if len(words) < 6:
                continue

            user = words[2]
            if user[0] in '@+%&~_[':
                user = user[1:]

            the_user = x[user]
            for word in words[3:]:
                if len(word) > 5 and not(word.startswith('http')) and word not in english_stopwords:
                    the_user[word] += 1

y = [(len(words), user) for user, words in x.items()]
high5 = sorted(y, reverse=True)[:25]
for unique_words, user in high5:
    for word, count in x[user].most_common(10):
        print(user,unique_words,word,count, sep=',')
\$\endgroup\$
2
  • \$\begingroup\$ Will this count every typo as a new word? Is that intentional? \$\endgroup\$
    – Mast
    Jun 12 at 6:57
  • \$\begingroup\$ @Mast Maybe I should add some kind of stemming to it. \$\endgroup\$
    – HSchmale
    Jun 12 at 14:21

1 Answer 1

1
\$\begingroup\$

Add comments.

At the top of the file, explain what it does. Mention the expected input format and the output format. Include paths or "stdin and stdout".

It's a quick script. Add enough that you can understand it in five years. Optimizing throwaway scripts for readability is not worthwhile.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.