5
\$\begingroup\$

I have written a program that takes the original file (the script of 'the bee movie') and does a word count, a unique word count and replaces some of the words. This is really just a rushed version, so if there are any errors or ways of fixing/making the code more efficient, please let me know.

Link to the script: https://github.com/esker-luminous/Bee-Movie-Knock-off/blob/main/script.py - it is a very large file.

Code:

file = open('script.py', 'r')

words = []
punctuation = '1234567890!@#$%^&*()_-+=-/*{}[];:\'"\<>~`|?'

for line in file:
    line = line.lower()
    for char in punctuation:
        line = line.replace(char, '')

    words += line.split()

print(len(words))
print(len(set(words)))
file.seek(0)

text = file.read()
print(text)
file.close()

word_replacement ={'bee': 'turtle', 'hive': 'cow', 'flower': 'algae', 'bees': 'turtle', 'flowers' : 'algae'}
for wor_repl in word_replacement.keys():
    text = text.replace(wor_repl, word_replacement[wor_repl])
file = open('knock off.py', 'w')
file.write(text)
file.close()
\$\endgroup\$
3
\$\begingroup\$

For the part where you first read the file, you could write it like this:

import collections
import re

file = open('bee_movie.txt', 'r')

text = ""
words = collections.Counter()
punctuation = '1234567890!@#$%^&*()_-+=-/*{}[];:\'"\\<>~`|?'
all_upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
all_lower = "abcdefghijklmnopqrstuvwxyz"
translation = str.maketrans(all_upper, all_lower, punctuation)

for line in file:
    text += line # concatenating like this is optimized in python 3
    words.update(line.translate(translation).split())

print(sum(words.values()))
print(len(words))
file.close()

For the part where you’re replacing words, it would be faster if you used re.sub. When you use text.replace, it needs to remake the entire script every time it’s used.

Also you should save the text files as .txt files, they don’t need the .py extension.

Btw you can import string and use string.ascii_uppercase, string.ascii_lowercase, and string.punctuation if you don’t want to type out all the characters.

\$\endgroup\$
10
  • 1
    \$\begingroup\$ @Peilonrayz thank you for your input! \$\endgroup\$ Jan 13 at 16:02
  • 1
    \$\begingroup\$ @my_first_c_program thank you so much for letting me know! appreciate it :)) \$\endgroup\$ Jan 13 at 16:03
  • 1
    \$\begingroup\$ Could you source "concatenating like this is optimized in python 3" last I heard it was a CPython implementation detail, but I couldn't find anything about it when I looked for it last. \$\endgroup\$
    – Peilonrayz
    Jan 13 at 16:05
  • 1
    \$\begingroup\$ Please be careful about conflating CPython and Python. That post only talks about CPython. \$\endgroup\$
    – Peilonrayz
    Jan 13 at 19:35
  • 2
    \$\begingroup\$ maketrans was interesting for me to learn about; I'd never encountered it before. On the whole I actually like it better than using a regular expression for this application. \$\endgroup\$
    – Reinderien
    Jan 13 at 22:13
3
\$\begingroup\$

it is a very large file

Perhaps if you're running on a microcontroller. Otherwise, 52kB is not considered large, and can comfortably fit in memory many thousands of times over.

Consider the following adjustments:

  • Do not iterate over lines, particularly since your input file has a weird format where most sentences are already broken up into many separate lines this is not efficient. Just read the entire thing into memory.
  • Tell the user the meaning of the two lengths you're outputting.
  • Avoid explicit close, and use a with context manager instead.
  • Do not re-read your file; just hold onto an unmodified version of its contents in memory.
  • Do not call keys; instead call items which gives you both the key and value.
  • Avoid looping over every single punctuation character. Instead, use str.transate to remove every match in one pass.

Suggested:

from string import ascii_lowercase, ascii_uppercase

with open('bee-movie.txt') as f:
    text = f.read()

trans = str.maketrans(ascii_uppercase, ascii_lowercase, r'1234567890!@#$%^&*()_-+=-/*{}[];:\'"\<>~`|?')
no_punc_text = str.translate(text, trans)

words = no_punc_text.split()
print('Total words:', len(words))
print('Total unique words:', len(set(words)))

word_replacement = {'bee': 'turtle', 'hive': 'cow', 'flower': 'algae',
                    'bees': 'turtles', 'flowers': 'algae'}
for source, dest in word_replacement.items():
    text = text.replace(source, dest)

with open('turtle-movie.txt', 'w') as f:
    f.write(text)
\$\endgroup\$
1
  • 1
    \$\begingroup\$ thank you so much for this! appreciate it 🙃 \$\endgroup\$ Jan 13 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.