Criticize my threaded image downloader

Question

I'm going to be working on a much larger version of this program and I just wanted to see if there was anything I should change in my style of coding before I made anything larger.

If it wasn't obvious, this code goes through each comic on A Softer World and downloads each comic.

Oh and with larger projects, should I try and fit my main() function into a class or is having a large main() function normal with programs?

from BeautifulSoup import BeautifulSoup as bs
from urllib import urlretrieve
from urllib2 import urlopen
from os import path, mkdir
from threading import Thread
from Queue import Queue

class Worker(Thread):

    def __init__(self, queue):
        self.queue = queue
        self.url = 'http://asofterworld.com/index.php?id='
        Thread.__init__(self)

    def run(self):
        while 'running':
            number = self.queue.get()
            soup = bs(urlopen(self.url + number))
            match = soup.find('p', {'id' : 'thecomic'}).find('img')

            filename = match['src'].split('/')[-1]
            urlretrieve(match['src'], 'Comic/' + filename)
            self.queue.task_done()

def main():

    queue = Queue()

    if not path.isdir('Comic'):
        mkdir('Comic')

    for number in xrange(1, 977):
        queue.put(str(number))

    for threads in xrange(20):
        thread = Worker(queue)
        thread.setDaemon(True)
        thread.start()

    queue.join()

main()

Answer 1

• The names Worker and run are very general, while what they are doing is actually very specific (crawl a single web site for images matching a pattern). It would be better if the names reflected their content.
• The magic numbers and strings in the code should either be parameters or constants. The thread count, for example, would be a good candidate for a parameter (since you might handle threading differently when importing Worker), while the different parts of the URL could be class constants.
• Regarding main, it should IMO only contain that which is necessary for running from a shell (as opposed to importing the functionality in another Python program). That is, when importing everything except main there should be no missing pieces to get the same functionality as when running from a shell.

Answer 2

• The observation made by @l0b0 regarding main is correct. But Ionly makes sense when the module can be imported without executing main. For that use the following pattern:

  def main():
      # your actual 'standalone' invocation steps
  
  if __name__ == '__main__':
      main()
  

• importing BeautifulSoup seems to imply you are still using version 3, which has been replaced with version 4 over a year ago and only receives bugfixes. You should switch to bs4 before expanding the program.

• you should put the actual url retrieval code in separate class. At some point (for this or other similar sites), the calls to urllib should be done with mechanize (e.g. keeping login state) or even selenium (if you need to have working javascript support to get to your target on your site).