16
\$\begingroup\$

Audiobooks make my life easier when I have idle ears and a mind free to wander, but something better for my eyes to do.

This program is one of my more recent efforts to allow most of the stories written on popular websites to be transformed into a more accessible audio format so that they can be enjoyed without having to steal the important function of vision from the user.

Concerns:

  • Primary Concerns:

    • Actual usefulness of program

    • Can I clean up the code in any further way so as to make it easier to read?

    • How the program holds up to real-world inputs from various sources and how to handle an unexpected error

  • Secondary Concerns:

    • Making it more convenient to use

    • Cross platform compatibility (Possibly)

    • Dealing with inputs I have not accounted for (URLs with no 'http://' in them)

  • Tertiary Concerns:

    • Making sure I have formatted my code for readability.

    • Are there any parts of my code that need more explanation?

Core Components:

  • Automated required module retrieval

    • Will fetch required modules if they are not installed on the system
  • say() Function

    • Uses Google's Text-to-Speech engine if online to read text and a local solution using Microsoft's built in Text-to-Speech engine if offline
  • getStory Class

    • Handles the retrieval and classification of text from the various websites or formats
  • OmniReader Function

    • Handles how to output the text from these websites or formats

Pros:

  • You can listen to stories as they are being downloaded by navigating to your C:\Users\'Username' directory, where an MP3 will be stored containing your story name and chapter number.

  • The succeeding chapters in a fanfiction.net story will be downloaded in preparation. (Is this something that I should remove?)

Cons:

  • If a URL or path is not formatted correctly for the program to read, it will treat that URL or path as text and will instead read it out loud.

  • Will spit out large blocks of story-text. (Would it be better for me to instead print out a progress report on what my code is doing?)

  • In the case of the fanfiction.net function, it will take an amount of time to process the text and then open up the MP3 after the file is completely downloaded.

    • If possible, I want to create the MP3, have the audio start downloading, then have the program open the MP3 while it is still being downloaded so as to avoid a long wait for the download to be completed.
  • Works better for some websites than others (fanfiction.net > wattpad)

  • No generalized text-grabber system. The fallback is to simply speak the input.

Long-Term Goal:

  • Have a more generalized solution for most websites or file formats while still retaining a precise solution for those, more popular websites or formats.

Supported Websites/Filetypes:

Code:

import os
import time
from subprocess import call
from platform import python_version
home = os.path.expanduser("~")


if '2.' == python_version()[:1]:
    print("This program requires Python 3 in order to properly function.\nA backwards compatible version may be available in the future.")
    input("End of line.")
    sys.exit()
else:
    try:
        from urllib.request import urlopen
    except:
        from urllib import urlopen

try:
    from gtts import gTTS
except:
    answer = input("Your system does not have Google's Text to Speech API installed. Do you want to install it?")
    if 'y' or 'Y' in answer:
        os.system('python -m pip install --upgrade gTTS')
    else:
        print("Without the Google Text to Speech API, this process will not sound natural.")

try:
    from bs4 import BeautifulSoup
except:
    answer = input("Your system does not have BeautifulSoup installed. Do you want to install it?")
    if 'y' or 'Y' in answer:
        os.system('python -m pip install --upgrade beautifulsoup4')
    else:
        print("Without BeautifulSoup, this process will not be able to pull text from websites successfully.")

try:
    import PyPDF2
except:
    answer = input("Your system does not have PyPDF2 installed. Do you want to install it?")
    if 'y' or 'Y' in answer:
        os.system('python -m pip install --upgrade PyPDF2')
    else:
        print("Without PyPDF2, this process will not be able to process PDF files successfully.")



speech_system = 'google'
def say(message,title='Speak',speech_system='google'):
    if speech_system == 'google':
        try:
            # Create the MP3 file which will speak the text
            title += '.mp3'
            tts = gTTS(message)
            tts.save(home+'\\'+title)
            os.system("start /MIN {}".format(home+'\\'+title))
        except:
            print("Vocalization failed.")
    else:
        try:
            # Create the Visual Basic code which will speak the text
            with open(title + '.vbs', 'w') as file:
                file.write(
                        """speaks="{}"\n
                           Dim speaks, speech\n
                           Set speech=CreateObject("sapi.spvoice")\n
                           speech.Speak speaks""".format( str(message) ))
            # Execute the file
            call(['cscript.exe', title + '.vbs'])
        except:
            print("Vocalization failed.")


class getStory:
    def __init__(self,url):
        self.url = url
        self.speech = speech_system
        self.text = 'Please initialize.'
        if 'wattpad' in self.url:
            self.type = 'wattpad'
        elif 'fanfiction' in self.url and 'wattpad' not in self.url:
            self.type = 'fanfiction'
        elif 'deviantart' in self.url:
            self.type = 'deviantart'
        elif 'pdf' in self.url:
            self.type = 'pdf'
        else:
            self.type = 'text'
        if 'http://' in url or 'https://' in url:
            self.pathtype = 'url'
        else:
            self.pathtype = 'local'

    def initialize(self):
        if self.type == 'wattpad':
            self.wattpad()
        elif self.type == 'fanfiction':
            self.fanfiction()
        elif self.type == 'deviantart':
            self.deviantart()
        elif self.type == 'pdf':
            self.pdf_inititialize()
        else:
            pass

    def fanfiction(self):
        # Opens and parses the URL with BeautifulSoup
        soup = BeautifulSoup(urlopen(str(self.url)))
        # Finds the path and contents given by the search
        try:
            self.text = soup.find(class_='storytext').text
        except:
            print('Retrieval Failed.')

    def deviantart(self):
        try:
            soup = BeautifulSoup(urlopen(str(self.url)))
            self.text = soup.select('#devskin > div > div > div.gr-body > div > div > div')[0].text
        except:
            print('Retrieval Failed.')

    def wattpad(self, page=0, mode='singular'):  # Modes are singular and plural
        if page:
            page = '/page/' + str(page)
        else:
            page = ''
        #Opens and parses the URL with BeautifulSoup
        soup = BeautifulSoup(urlopen(str(self.url + page)))
        #Finds the path and contents given by the search
        if mode == 'singular':
            self.text = soup.find(class_="panel panel-reading")
        elif mode == 'plural':
            self.text = soup.find_all(class_="panel panel-reading")

    def pdf_inititialize(self):
        try: #Safety first!
            os.remove(os.getcwd() + '\\PDF2BEREAD.pdf')
        except:
            pass
        if self.pathtype == 'url':
            #Download the PDF from the web
            path = urlopen(self.url)
            with open('PDF2BEREAD.pdf','wb') as file:
                file.write(path.read())
            self.url = os.getcwd() + '\\PDF2BEREAD.pdf'

    def pdf(self,page):
        self.text = PyPDF2.PdfFileReader(self.url).getPage(page).extractText().replace('\u2122',"'")

    def text(self):
        self.text = self.url

    def parse(self):
        #Removes all unicode characters, nonprintable characters, and HTML code
        text = str(bytes(self.text,'utf-8').decode('unicode_escape').encode('ascii','ignore').decode('utf-8'))
        #Removes newline and return characters
        if speech_system == 'local':
            self.text = text.replace('\n',' ').replace('\r',' ').replace('"',"'").replace('.','. ').replace('.   .   . ','').replace("\'", '').replace('\"', '').replace('Klenon','Klenn une').replace('Tali','Tahlie').replace('tali','tahlie').replace ('Yalo',' Yah-lo ').replace('Garrus','Gae-rrus').replace('Vakarian','Vah-kare-eean').replace('Noveria',' No-veir-eaah ').replace('Binary','Bi-nary').replace('caf ', 'cafe ')
        else:
            self.text = text.replace('\n',' ').replace('\r',' ').replace('"',"'").replace('.','. ').replace('.   .   . ','').replace("\'", '').replace('\"', '').replace('Tali','Tahhlee').replace('tali','Tahhlee').replace('caf ', 'cafe ')

def OmniReader(text):
    story = getStory(text)

    if story.type == 'wattpad':
        number_of_pages = int(input("How many pages are in the story: ")) + 1
        # Iterates through the pages of the story
        for each_page in range(number_of_pages):
            if each_page:
                #Designed to cope with Wattpad's annoying multi-page system
                story.wattpad(each_page,'plural')[1]
            else:
                #Meant for the first page
                story.wattpad()
            #Get all the text in one array
            paragraphs = story.text.find_all('p')
            #Iterates through the paragraphs in each page of the story
            for each_paragraph in range(len(paragraphs)):
                #Get all the text segments
                paragraphs[each_paragraph] = paragraphs[each_paragraph].text
            text = ' '.join(paragraphs)
            #Helps to save special characters before the purge
            text = text.replace(':', ': ').replace('!', '! ').replace('?', '? ').replace('. . .', '...').replace('1',
                                                                                                                 '1 ').replace(
                   '2', '2 ').replace('3', '3 ').replace('4', '4 ').replace('5', '5 ').replace('6','6 ').replace('7',
                                                                                                                 '7 ').replace(
                   '8', '8 ').replace('9', '9 ').replace('0', '0 ').replace('!  ', '! ').replace('?  ','? ').replace(':  ',
                                                                                                                     ': ').replace(
                   ';', '; ').replace(';  ', '; ')
            text = bytes(text, 'utf-8').decode('unicode_escape').encode('ascii', 'ignore').decode('utf-8')
            print(text)
            #Say it for me, baby.
            say(text,speech_system=speech_system)
        input("End of Line.")

    elif story.type == 'fanfiction':
        #Loop through each chapter in a fanfiction and save the audio reading of each
        try:
            while 1:
                url = story.url.split('/')
                story.fanfiction()
                #Set up the name for each audio recording
                title = url[-1] + '_' + url[-2]
                story.parse()
                print(story.text)
                say(story.text,title,speech_system)
                #Iterate to the next chapter and reset the URL
                url[-2] = str(int(url[-2]) + 1)
                story.url = '/'.join(url)
        except:
            input("End of Line.")

    elif story.type == 'pdf':
        first_page = int(input("Please enter the beginning page: ")) - 1
        for each_page in range(first_page,PyPDF2.PdfFileReader(self.url).getNumPages()):
            story.pdf(each_page)
            print('\n \t \t' + str(each_page + 1) + '\n')
            print(story.text)
            story.parse()
            say(story.text,speech_system=speech_system)
        input("End of line.")

    elif story.type == 'deviantart':
        story.deviantart()
        url = story.url.split('/')
        title = url[-1]
        story.parse()
        print(story.text)
        say(story.text,title,speech_system)
        input("End of line.")

    elif story.type == 'text':
        print(story.url)
        story.text()
        story.text = story.url
        story.parse()
        say(story.text,speech_system=speech_system)
        input("End of line.")

try:
    say("Initializing...")
except:
    speech_system = 'local'

text = input("State your request, creator: ")
OmniReader(text)
\$\endgroup\$
  • 1
    \$\begingroup\$ Do not remove code that has already been reviewed. For legality issues, use the Contact Us form to have this evaluated by SE. \$\endgroup\$ – Jamal Jul 15 '15 at 17:16
9
\$\begingroup\$

I'm just going to review these eight lines of code. You'll see that there's plenty here for one answer.

try:
    from gtts import gTTS
except:
    answer = input("Your system does not have Google's Text to Speech API installed. Do you want to install it?")
    if 'y' or 'Y' in answer:
        os.system('python -m pip install --upgrade gTTS')
    else:
        print("Without the Google Text to Speech API, this process will not sound natural.")
  1. If you restricted your lines to 79 columns, as recommended by the Python style guide (PEP8), then we wouldn't have to scroll the code horizontally to read it here.

  2. The prompt is also more than 79 columns wide, which means that on most users' terminals it will wrap. Better to split this prompt:

    print("Your system does not have Google's Text to Speech API installed.")
    answer = input("Do you want to install it?")
    

    or use textwrap.fill to split it into lines.

  3. A bare except: is a bad idea: it catches all exceptions, including KeyboardInterrupt and SystemExit. This means that you might suppress genuine problems that you would have preferred to know about. It is better to catch just the exception that you are interested in, which is ImportError in this case.

  4. The test:

    if 'y' or 'Y' in answer:
    

    is always true! That's because it parses like this:

    if 'y' or ('Y' in answer):
    

    and 'y' is a non-empty string, so it converts to true. The code needs to say:

    if 'y' in answer or 'Y' in answer:
    
  5. The test 'y' in answer accepts too many strings, I think. If the user enters nay or no way or not on your life then this test will still evaluate to true. Better to be more restrictive, for example answer.startswith(('Y', 'y')).

  6. If the user chooses to install the gTTS package, the program does not make a second attempt to import it. So the user will be disappointed: it will appear that the installation was ineffective.

  7. Running the command python relies on the Python interpreter being on PATH, and also on this Python interpreter being the right one in the common case where there are several installed versions of Python on the same machine. It would be better to use sys.executable.

  8. os.system can be wasteful of resources (because it starts a shell to run the command) and risky (because the command gets parsed by the shell and this can go wrong). In this case neither of these problems is severe, but it's worth practicing good security habits in easy situations, so I recommend subprocess.call instead.

  9. After a failure to import, this code prints a warning, but then just carries on. This will just lead to a NameError later on when the program tries to evaluate gTTS. It would be better to exit the program, for example by re-raising the exception.

  10. These eight lines of code are essentially repeated three times for three different modules. It would be better to extract the common code and make it into a function. This is slightly tricky because of the need to import a module given its name as a string, but we can use importlib.import_module, like this:

    from importlib import import_module
    import subprocess
    import sys
    from textwrap import fill
    
    def import_or_install(module, package, description, capability):
        """Import module. If it fails to import, prompt user to install
        package and try again. The description argument gives a brief
        human-readable description of the module, and capability gives a
        human-readable description of what it can do.
    
        """
        try:
            import_module(module)
        except ImportError:
            print(fill("Your system does not have {} installed. "
                       "Without this module, this process will not {}."
                       .format(description, capability)))
            answer = input("Do you want to install it?")
            if answer.startswith(('Y', 'y')):
                subprocess.call([sys.executable, '-m', 'pip', 'install',
                                 '--upgrade', package])
                import_module(module)
            else:
                raise
    

    Now it's possible to write:

    import_or_install('gtts', 'gTTS', 'Google Text to Speech API',
                      'sound natural')
    from gtts import gTTS
    
    import_or_install('bs4', 'beautifulsoup4', 'Beautiful Soup',
                      'be able to pull text from websites'):
    from bs4 import BeautifulSoup
    
    import_or_install('PyPDF2', 'PyPDF2', 'PyPDF2',
                      'be able to process PDF files'):
    import PyPDF2
    

    Note that this still has an import statement after calling import_or_install because we need to assign a name to the module. Calling importlib.import_module loads the module (and puts an entry in sys.modules), but does not actually assign it a name.

\$\endgroup\$
  • \$\begingroup\$ Why is the plain import statement (not the from ... import) after the call to import_or_install still needed? \$\endgroup\$ – mkrieger1 Jul 3 '15 at 11:59
  • \$\begingroup\$ See updated answer. \$\endgroup\$ – Gareth Rees Jul 3 '15 at 12:07
  • \$\begingroup\$ Thank you, this cleans up the top section and makes further expansion much simpler. \$\endgroup\$ – Zenohm Jul 3 '15 at 19:25
3
\$\begingroup\$

This is really cool!

There's a few things that could be fixed up:


You should have whitespace between each variable in your parameters:

def say(message,title='Speak',speech_system='google'):

into:

def say(message, title='speak', speech_system='google'):    
home+'\\'+title

into:

home + '\\' + title

but, you should have extra whitespace after/before your brackets:

.format( str(message) ))

into: .format(str(message)))


You can use array format or or format for if-else conditions:

if 'http://' in url or 'https://' in url:

into: if 'http://' or 'https://' in url: There's probably a better way of doing that, perhaps substrings.


You could turn the following into a ternary statement, if you like:

if page:
    page = '/page/' + str(page)
else:
    page = ''
page = '' if not page else '/page/' + str(page)</s>

I would suggest changing your really long .replace().replace() structure into an array structure like follows:

changes = {
     "Binary": "Bi-nary",
     ".   .   . ": "",
   }
return [string.replace(k, v) for k, v in changes.items()]

Other than that, your code looks nice and clean!

\$\endgroup\$
2
\$\begingroup\$

This looks like one cool project!

First, you should use if __name__ == "main":. This will help if you ever split your code into multiple files, and is just the correct thing to do.

Second, you have a large section of if/elif where you determine what you are reading. You could clean this up a bit be creating more methods to call, instead of having one monster method to contain the logic for everything related to this. Splitting your code up into smaller subsections will also help keep bugs away.

Third, you have some duplication in the afore-mentioned if blocks. Some of this may be able to be removed by slight rearrangements and moving identical sections to the end of the method, possibly not.

Fourth, in the afore-mentioned if blocks, you don't have a final else. What do you do if the type of book isn't found? Just exit? Perhaps you could have an else saying that the book type isn't supported.

\$\endgroup\$
  • \$\begingroup\$ The else statement is given in the __init__ of the getStory class. \$\endgroup\$ – Zenohm Jul 3 '15 at 4:39
  • \$\begingroup\$ if __name__ == "__main__" Thanks for the suggestion, I implemented this and a main() function in the more recent versions. \$\endgroup\$ – Zenohm Jul 25 '15 at 23:03
2
\$\begingroup\$

I'm guessing if 'http://' in url or 'https://' in url should actually be : if url.startswith('http://') or 'url.startswith('https://').

Also, from the startswith documentation :

str.startswith(prefix[, start[, end]])

Return True if string starts with the prefix, otherwise return False. prefix can also be a tuple of prefixes to look for. With optional start, test string beginning at that position. With optional end, stop comparing string at that position.

Changed in version 2.5: Accept tuples as prefix.

So this can be :

if url.startswith(('http://', 'https://')

\$\endgroup\$
2
\$\begingroup\$

I'd replace usages of os.system with the subprocess module (which you're already using!). This is recommended by the Python Standard Library documentation linked below and in PEP 324

https://docs.python.org/2/library/subprocess.html#replacing-os-system

\$\endgroup\$
  • \$\begingroup\$ Welcome to Code Review! Your answer could be even better if you briefly mentioned why you should avoid os.system. \$\endgroup\$ – 200_success Jul 3 '15 at 10:13
  • 1
    \$\begingroup\$ Updated to include mention of why to avoid os.system :) \$\endgroup\$ – Lorcan O'Neill Jul 6 '15 at 10:51
2
\$\begingroup\$

I was originally planning on a comment, but it turned into an answer.

WARNING: Legal advice from a non-lawyer! Some of this information might not be correct in your jurisdiction.

While not exactly a concern with the code itself, I do want to call out the issue of copyright and authorship here. I've checked the Fanfiction, WattPad and DeviantArt ToS pages. FF and WP both have 4 rules in their terms which will give you problems, even with the same number and wording:

4A: You will not distribute any of the content on our site on any medium without our permission;

4C: You will only access our content through our website (so, not through any programs);

4E: You will not use any scrapers on our website, unless you have our permission.

5D: you will not copy any of the content on our site.

DeviantArt has an entirely different format on their ToS, but they still say that you can't reproduce any of their user submitted content in any medium without their express permission.

In addition, there are internationally agreed upon copyright laws that support the above statements, for example, the US copyright Act of 1976. Now, I do admit that you might be able to claim that this is similar to the biggest non-fair use exemption to copyright: the infamous home copy you're allowed to have. There is also the idea that this might be a transformative use, since this can turn any article on those websites into an audiobook that is more accessible to those with poor vision. However, as far as I'm aware, you need permission to make audiobooks.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.