6
\$\begingroup\$

I have written some code that scrapes data on asteroids, but the problem is that is super slow! I understand that it has a lot to scrape, but as of now it has been running for 5 days and is bot even a tenth of the way through. Here is my code, the part I'm talking about is under GET-EPHEMERIDES:

from mechanize import Browser
from bs4 import BeautifulSoup
import datetime
from dateutil.relativedelta import relativedelta
##from SendEmail import Send
import os
import httplib
import time
import sys

print "###################################################################################\n                        Scrapes targets from MPO Opposition Database                       \n                                                                                   \n                            and Ephemerides from MPC's website                            \n                              Author: Tarik Joseph Zegmott                           \n###################################################################################\n\n"      

#-----EXTRACTION-CODE-FOR-MINORPLANET.INFO------------------------------------------

def extract(soup):
    table = soup.find('table', border=1)
    for row in table.findAll('tr')[1:]: # uses BS findAll() to pull all tr tags (html row) into a list, the [1:] modifier skips the first lin which is just a header
        col = row.findAll('td') # A list that will grab all the td tags (html column)
        num = col[0].font.string
        name = col[1].font.string
        odate = col[2].font.string # Opposition Date (mm/dd.d)
        omag = col[3].font.string # Opposition Mag (V)
        mddate = col[4].font.string # Date of Minimum Distance (mm/dd.d)
        mdist = col[5].font.string # Minimum Distance from Earth (AU)
        bdate = col[7].font.string # Date of Brightest Apparition (mm/dd.d)
        bmag = col[8].font.string # Brightest Magnitude (V)
        bdec = col[9].font.string # Declination on Date of Brightest Apparition
        record = (num, name, odate, omag, mddate, mdist, bdate, bmag, bdec)
        print >> outfile, "|".join(record) # pipe-delimiter using .join method

#-----------------------------------------------------------------------------------
def dif_month(d1, d2): # Calculates number of months between two dates
    return (d1.year - d2.year)*12 +d1.month - d2.month + 1

#-----SCRAPES-MINORPLANET.INFO-FOR-EACH-MONTH---------------------------------------
'''
    This sections uses BeautifulSoup and mechanize to scrape minorplanet.info's ephemerides service for a list of NEAs that will be observable during the specified dates. The NEAs are recorded in text files based on the month and year they're visible.
'''
####################################################################################
dayS = 01; monthS = 01; yearS = 2015 # Start Date
dayE = 31; monthE = 12; yearE = 2035 # End Date
SDate = datetime.datetime(yearS, monthS, dayS)
EDate = datetime.datetime(yearE, monthE, dayE)
#Date = []
total_months = float(dif_month(EDate, SDate))

while True:
    try:
        scrapemp = raw_input('Do you want to scrape minorplanet.info? (yes/no) ')
        if scrapemp not in ['yes', 'no']:
            raise ValueError("You did not enter 'yes' or 'no'. Please note: case sensitive.")
        break
    except ValueError, e:
        print e
        print "Try again..."

if scrapemp == 'yes':
    if os.path.exists('./Targets') is False:
        print "Creating directory ./Targets"
        os.mkdir('./Targets')
    print "Changing directory to ./Targets\n"
    os.chdir('./Targets')
    month_count = 0.

    sys.stdout.write("\rWriting asteroids to 2015-01_targets.txt  [0.00%]")
    sys.stdout.flush() # shows what is written to standard output
    while SDate <= EDate:
##      print "Looking for asteroids with apparitions in "+SDate.strftime('%B %Y')+"..."
        outfile = open(SDate.strftime('%Y-%m')+'_targets.txt', 'w')
        mech = Browser() # uses mechanize's Browser
        url = 'http://www.minorplanet.info/PHP/mpo_opps_query.php?script=Script&DataOutput[]=HTML&SortOrder[]=BDate&NumberedStatus[]=Both&SearchYear[]='+SDate.strftime('%Y')+'&SearchMonth[]=%02d&Group[]=NEA&MinMag=0&MaxMag=25&MinDec=-90&MaxDec=90'%SDate.month

        while True: # Reruns this loop until 'try' works, but can make run indefinately (not sure why?)
            try:
                page = mech.open(url) # grabs the location of our scrape target
            except httplib.BadStatusLine:
                print "\nError Encountered Retrying...\n"
                time.sleep(2)
                continue
            break   
        html = page.read() # returns all of the pages html
        soup = BeautifulSoup(html)
        try:
            extract(soup)
        except AttributeError:
            print "\nWebsite could be having troubles, please check."
            raise KeyboardInterrupt
##      print "Asteroids written to "+SDate.strftime('%Y-%m')+"_targets.txt\n"  
        outfile.close()
        SDate += relativedelta(months=1) # Advance Date by 1 month
        mech.close()
        month_count += 1 # for the percentage done
        sys.stdout.write("\rWriting asteroids to "+SDate.strftime('%Y-%m')+"_targets.txt  [%.2f%%]" %((month_count/total_months)*100.)) #writes percentage to standard output
        sys.stdout.flush() # shows what is written to standard output


    print "Changing directory to ..\n"
    os.chdir('..')

    ##Send('Asteroid Program', '[Nombre], \n \t Scraping of minorplanet.info from '+datetime.date(yearS, monthS, dayS).strftime('%B %Y')+' to '+datetime.date(yearE, monthE, dayE).strftime('%B %Y')+' has been completed.')

    raw_input('This is your chance to make a copy of the files in ./Targets.\nPress Enter To Continue: ')

#-----GET-EPHEMERIDES---------------------------------------------------------------
'''
   Makes use of BeautifulSoup and mechanize to scrape data from the MPC's ephemerides service, by looking up the names and dates scraped from minorplanet.info - that are stored in the text files from the ./Targets directory - and then creates new text files of each object's ephemerides from multiple locations and multi times throughout the night that are in the local timezone, which are stored in the directory "./Ephemerides" 
'''
####################################################################################
if os.path.exists('./Ephemerides') is False:
    print "Creating directory ./Ephemerides"
    os.mkdir('./Ephemerides')
print "Changing directory to ./Ephemerides\n"
os.chdir('./Ephemerides')

SDate = datetime.datetime(yearS, monthS, dayS) # Reset start date

times =['-2', '2', '6'] # Used in combination with time zone (locations[2])  to get 2200, 0200, 0600 local time
locations = [['695', '703', 'E12', 'K94', '807', '568', '950', '381'], ['Kitt Peak, Arizona', 'Catalina, Arizona', 'Siding Spring, Australia', 'Sutherland, South Africa', 'Cerro Tololo, Chile', 'Mauna Kea, Hawaii', 'La Palma, Canary Islands', 'Tokyo Kiso, Japan'], ['-7', '-7', '10', '2', '-3', '-10', '0', '9']] # [[Code], [Name], [GMT +- x (timezone)]]
while True:
    try:
        ephem = raw_input('Do you want to get ephemerides? (yes/no) ') 
        if ephem not in ['yes', 'no']:
            raise ValueError("You did not enter 'yes' or 'no'. Please note: case sensitive.")
        break
    except ValueError, e:
        print e
        print "Try again..."

if ephem =='yes':
    month_count = 0.
    print "Ephemerides are saved in folders for each year-month combination."
    while SDate <= EDate:
        count = 0.; subcount = 0. # for progress percentage
        date = open('../Targets/'+SDate.strftime('%Y-%m')+'_targets.txt', 'r')
        for i in date:
            count += 1.
        date.seek(0)
        if os.path.exists('./'+SDate.strftime('%Y_%m')) is False:
            print "Creating directory ./"+SDate.strftime('%Y_%m')
            os.mkdir('./'+SDate.strftime('%Y_%m'))
        os.chdir('./'+SDate.strftime('%Y_%m'))

        for line in date:
            line = line.strip()
            col = line.split("|")
            num = col[0]
            name = col[1]
            bdate = col[6]
            bdatemonth = bdate[:2]; bdateday = bdate[3:-2]
            brdate = datetime.datetime(int(SDate.strftime('%Y')), int(bdatemonth), int(bdateday))
            indate = brdate + relativedelta(days=-30)
            for loc, locname, loctime in zip(locations[0], locations[1], locations[2]):
                for t in times:
                    sys.stdout.write("\rMonth: "+SDate.strftime('%B')+", Year: "+SDate.strftime('%Y')+"; Current NEA: "+name+"  [%.2f%%]     [%.2f%%]"%((subcount/count)*100., (month_count/total_months)*100.))
                    sys.stdout.flush() # shows what is written to standard output
                    outfile = open(name+'_'+locname+'_at_local_midnight_+'+t+'.txt', 'w')
                    mech = Browser()
                    temp = int(loctime) + int(t)
                    time2use = str(temp)
                    url = "http://mpc.eps.harvard.edu/cgi-bin/mpeph2.cgi?ty=e&TextArea="+name.replace(' ', '%20')+"&d="+indate.strftime('%Y-%m-%d')+"&l=60&i=&u=d&uto="+time2use+"&c="+loc+"&long=&lat=&alt=&raty=a&s=t&ibh=y&m=m&adir=S&oed=&e=-2&resoc=&tit=&bu=&ch=c&ce=f&js=f"
                    while True: # Reruns this loop until 'try' works
                        try:
                            page = mech.open(url) # grabs the location of our scrape target
                        except httplib.BadStatusLine: # Catches the error 'badstatusline'
                            print "\nError Encountered Retrying...\n"
                            time.sleep(1)
                            continue
                        break
                    html = page.read()
                    soup = BeautifulSoup(html)
                    outfile.write(soup.pre.get_text()) # ...Suppressed... includes previous dates too
                    outfile.close()
                    mech.close()
            subcount += 1 # for the percentage done
        month_count += 1 # for the percentage done
        date.close()
        SDate += relativedelta(months = 1)

        os.chdir('..')

If you could help me identify any bottlenecks and just generally improve it to run quick, it would be much appreciated.

\$\endgroup\$
  • 1
    \$\begingroup\$ Welcome to CodeReview.SE ! Your code does not seem to run for me because yearS is not defined. Can you please have a look at this ? Also, I recommand using checkers such as pep8, pylint, pychecker or pyflakes on your code before submitting it as it might help you to find many issues. \$\endgroup\$ – SylvainD Dec 16 '14 at 21:23
  • \$\begingroup\$ Ahh sorry, I tried to cut out the part that I was having trouble with without putting up all the code, but I'll edit my question and put it all up. \$\endgroup\$ – Tazza2092 Dec 17 '14 at 15:34
3
\$\begingroup\$
  1. Remove nasty comments. For example, your "block separator comments", ###...### are completely, and utterly useless. Remove them. Another thing would be to not create "title comments", #----...----. Comments should be helpful, many of yours are not.
  2. Again, another tip on comments. Many of your comments describe things that are already obvious from looking at the code. For example, you have a comment that says this: # Reruns this loop until 'try' works, but can make run indefinately (not sure why?). It's clear that once the try block works, that the loop is exited. Obvious comments like these can be removed.
  3. Variables and functions should be named in the style of snake_case, and classes should be in the style of PascalCase. If a variable is constant, it should be UPPERCASE_SNAKE_CASE.
  4. You need better variable names. For example, d1, or d2 are completely unclear as to what their purposes are. Variable names should be long, but not too long, and as descriptive as possible. You also have many other places where you could do renaming, d1, and d2 are just a few examples.
  5. When getting user input, instead of making the user enter everything in lowercase, lower the text using str.lower(). Here's an example: user_input = raw_input("> ").lower().
  6. At the beginning of your code, you print the same character many times. Instead of repeating this character over and over again in your string, use string multiplication. For example, if I wanted to print 50 spaces, I would do print " " * 50.
  7. Why on Earth are you using sys.stdout.write()? Just use print to print something. sys.stdout.write() is unnecessary.
  8. Rather than checking if something equals false, as you're doing on this line: if os.path.exists('./Targets') is False:, you can just do if not os.path.exists("./Targets"):.
  9. Finally, you have many PEP8 errors. There are way too many to list in one answer, so I'm going to link the style guide instead, and you can read through it. You can find PEP8 here.
|improve this answer|||||
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.