I've been working for a program over the last few days which should download a range of pages from the webcomic Homestuck. I've created a working version in python 3, but it is horribly inefficient. Can anyone see ways to improve and shorten this code?

import urllib.request
range1 = int(input("Enter the 1st page you want: "))
range2 = int(input("Enter the last page you want: ")) + 1
current = range1 + 1900
final = range2 + 1900
page = ''
nextPage = ''
while current != final:
    page = str(current)
    nextPage = str(current+1)
    while len(page) != 6:
        page = '0'+ page
    while len(nextPage) != 6:
        nextPage = '0'+ nextPage
    html = 'http://www.mspaintadventures.com/?s=6&p='+page
    site = urllib.request.urlopen(html)
    s = site.read()
    s = s.decode("utf8")
    s = s.replace("<!-- end comic content -->", "<!-- begin comic content -->")
    s = s.replace("http://cdn.mspaintadventures.com/storyfiles/hs2/", "")
    s = s.replace("?s=6&p=" + str(nextPage), str(int(nextPage))+".html")
    s = s.replace(page+"/"+page, page)
    a,b,c = s.split('<!-- begin comic content -->')
    b = "<title> Page " + page + "</title>" + b
    t = open(str(current)+".html", 'w+')
    page = str((int(page)-1900))
    while len(page) != 5:
        page = '0'+ page

    t = open(str(current)+".html", 'a')
    t.write("<object width='1000' height='1000'> <param name='movie' value='"+swfname+"'>")
    t.write("<embed src="+swfname+" width=650 height=450>")
        img = "http://cdn.mspaintadventures.com/storyfiles/hs2/"+page+".gif"
        urllib.request.urlretrieve(img, page+".gif")
            img = "http://cdn.mspaintadventures.com/storyfiles/hs2/"+page+"_1.gif"
            urllib.request.urlretrieve(img, page+"_1.gif")
            img = "http://cdn.mspaintadventures.com/storyfiles/hs2/"+page+"_2.gif"
            urllib.request.urlretrieve(img, page+"_2.gif")
                img = "http://cdn.mspaintadventures.com/storyfiles/hs2/"+page+"/"+page+".swf"
                urllib.request.urlretrieve(img, page+".swf")
                print("Image "+img+" failed to download")
    print ("Page " + str(page) + " of " + str(final-1901) + " downloaded")

The main performance issue is the blocking nature of your script. You don't process the next url until you are done with the current. Think of using asynchronous tools like Scrapy web-scraping framework which is based on twisted; or something like grequests.

Other notes:

  • if you would stick to synchronous approach, switch to requests, initialize a session (requests.Session) once and reuse - this should be faster than using urllib.request
  • when you generate HTML files, pre-define a template with placeholders, render the template on the fly filling up the placeholders. You may use a template engine like mako or Jinja2, or use the built-in str.format()
  • it also looks like you are reopening each of the generated files twice - once for the initial write in the w+ mode and then to append in a mode
  • \$\begingroup\$ Thanks, I'll have a go. As for your 3rd note, is there a better way than this? \$\endgroup\$ – Jammy Dodger Feb 13 '17 at 22:15
  • \$\begingroup\$ @JammyDodger I was thinking about keeping the file open until you are done writing into it..open once per iteration..thanks. \$\endgroup\$ – alecxe Feb 13 '17 at 22:17
  • \$\begingroup\$ Is there a way to append to a file without reopening? \$\endgroup\$ – Jammy Dodger Feb 13 '17 at 22:19
  • \$\begingroup\$ @JammyDodger I think you can keep the file open in the w mode and continue writing into it..may be I'm missing something.. \$\endgroup\$ – alecxe Feb 13 '17 at 22:20
  • \$\begingroup\$ But when I write in 'w' mode, it overwrites the file. Or maybe if I keep the file open it will just append? \$\endgroup\$ – Jammy Dodger Feb 13 '17 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.