A Bee Movie Knock-off

Question

I have written a program that takes the original file (the script of 'the bee movie') and does a word count, a unique word count and replaces some of the words. This is really just a rushed version, so if there are any errors or ways of fixing/making the code more efficient, please let me know.

Link to the script: https://github.com/esker-luminous/Bee-Movie-Knock-off/blob/main/script.py - it is a very large file.

Code:

file = open('script.py', 'r')

words = []
punctuation = '1234567890!@#$%^&*()_-+=-/*{}[];:\'"\<>~`|?'

for line in file:
    line = line.lower()
    for char in punctuation:
        line = line.replace(char, '')

    words += line.split()

print(len(words))
print(len(set(words)))
file.seek(0)

text = file.read()
print(text)
file.close()

word_replacement ={'bee': 'turtle', 'hive': 'cow', 'flower': 'algae', 'bees': 'turtle', 'flowers' : 'algae'}
for wor_repl in word_replacement.keys():
    text = text.replace(wor_repl, word_replacement[wor_repl])
file = open('knock off.py', 'w')
file.write(text)
file.close()

Answer 1

For the part where you first read the file, you could write it like this:

import collections
import re

file = open('bee_movie.txt', 'r')

text = ""
words = collections.Counter()
punctuation = '1234567890!@#$%^&*()_-+=-/*{}[];:\'"\\<>~`|?'
all_upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
all_lower = "abcdefghijklmnopqrstuvwxyz"
translation = str.maketrans(all_upper, all_lower, punctuation)

for line in file:
    text += line # concatenating like this is optimized in python 3
    words.update(line.translate(translation).split())

print(sum(words.values()))
print(len(words))
file.close()

For the part where you’re replacing words, it would be faster if you used re.sub. When you use text.replace, it needs to remake the entire script every time it’s used.

Also you should save the text files as .txt files, they don’t need the .py extension.

Btw you can import string and use string.ascii_uppercase, string.ascii_lowercase, and string.punctuation if you don’t want to type out all the characters.

Answer 2

it is a very large file

Perhaps if you're running on a microcontroller. Otherwise, 52kB is not considered large, and can comfortably fit in memory many thousands of times over.

Consider the following adjustments:

• Do not iterate over lines, particularly since your input file has a weird format where most sentences are already broken up into many separate lines this is not efficient. Just read the entire thing into memory.
• Tell the user the meaning of the two lengths you're outputting.
• Avoid explicit close, and use a with context manager instead.
• Do not re-read your file; just hold onto an unmodified version of its contents in memory.
• Do not call keys; instead call items which gives you both the key and value.
• Avoid looping over every single punctuation character. Instead, use str.transate to remove every match in one pass.

Suggested:

from string import ascii_lowercase, ascii_uppercase

with open('bee-movie.txt') as f:
    text = f.read()

trans = str.maketrans(ascii_uppercase, ascii_lowercase, r'1234567890!@#$%^&*()_-+=-/*{}[];:\'"\<>~`|?')
no_punc_text = str.translate(text, trans)

words = no_punc_text.split()
print('Total words:', len(words))
print('Total unique words:', len(set(words)))

word_replacement = {'bee': 'turtle', 'hive': 'cow', 'flower': 'algae',
                    'bees': 'turtles', 'flowers': 'algae'}
for source, dest in word_replacement.items():
    text = text.replace(source, dest)

with open('turtle-movie.txt', 'w') as f:
    f.write(text)