2
\$\begingroup\$

I'm going to be working on a much larger version of this program and I just wanted to see if there was anything I should change in my style of coding before I made anything larger.

If it wasn't obvious, this code goes through each comic on A Softer World and downloads each comic.

Oh and with larger projects, should I try and fit my main() function into a class or is having a large main() function normal with programs?

from BeautifulSoup import BeautifulSoup as bs
from urllib import urlretrieve
from urllib2 import urlopen
from os import path, mkdir
from threading import Thread
from Queue import Queue

class Worker(Thread):

    def __init__(self, queue):
        self.queue = queue
        self.url = 'http://asofterworld.com/index.php?id='
        Thread.__init__(self)

    def run(self):
        while 'running':
            number = self.queue.get()
            soup = bs(urlopen(self.url + number))
            match = soup.find('p', {'id' : 'thecomic'}).find('img')

            filename = match['src'].split('/')[-1]
            urlretrieve(match['src'], 'Comic/' + filename)
            self.queue.task_done()

def main():

    queue = Queue()

    if not path.isdir('Comic'):
        mkdir('Comic')

    for number in xrange(1, 977):
        queue.put(str(number))

    for threads in xrange(20):
        thread = Worker(queue)
        thread.setDaemon(True)
        thread.start()

    queue.join()

main()
\$\endgroup\$
  • \$\begingroup\$ Can you tell us something about the much larger version, to give more context to this code? \$\endgroup\$ – Janne Karila Jun 10 '13 at 6:27
  • \$\begingroup\$ I'm hoping to have it go through multiple sites and follow links from there to download random images. Like a web crawler, but instead of indexing things, it will just randomly download images it finds. \$\endgroup\$ – Leinad177 Jun 10 '13 at 6:38
  • \$\begingroup\$ One thing that can be improved is the title of your question. Since all questions here ask for code review, instead try describing what your code does. I've changed it for you, but feel free to improve it further. (We also use proper English here, so please do capitalize correctly: i-->I) \$\endgroup\$ – Adam Jun 12 '13 at 7:56
1
\$\begingroup\$
  • The names Worker and run are very general, while what they are doing is actually very specific (crawl a single web site for images matching a pattern). It would be better if the names reflected their content.
  • The magic numbers and strings in the code should either be parameters or constants. The thread count, for example, would be a good candidate for a parameter (since you might handle threading differently when importing Worker), while the different parts of the URL could be class constants.
  • Regarding main, it should IMO only contain that which is necessary for running from a shell (as opposed to importing the functionality in another Python program). That is, when importing everything except main there should be no missing pieces to get the same functionality as when running from a shell.
\$\endgroup\$
1
\$\begingroup\$
  • The observation made by @l0b0 regarding main is correct. But Ionly makes sense when the module can be imported without executing main. For that use the following pattern:

    def main():
        # your actual 'standalone' invocation steps
    
    if __name__ == '__main__':
        main()
    
  • importing BeautifulSoup seems to imply you are still using version 3, which has been replaced with version 4 over a year ago and only receives bugfixes. You should switch to bs4 before expanding the program.

  • you should put the actual url retrieval code in separate class. At some point (for this or other similar sites), the calls to urllib should be done with mechanize (e.g. keeping login state) or even selenium (if you need to have working javascript support to get to your target on your site).

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.