eBay scraper with BeautifulSoup

Question

A friend of mine needs a scraper that reads product titles, prices and pictures from eBay and saves them to an Excel-ready .csv file. The program reads product pages into a list and then loops through it to get the relevant data, which it then saves to a .csv file.

import pdb
import time
import urllib
import urllib2
import re
import sys
import os
import shutil
import codecs
import unicodedata
from BeautifulSoup import BeautifulSoup

#debugger, just in case :S
#pdb.set_trace()
print("eBay allGrab Scraper by Ben Fishman, 2013\n")
print("\n")

#Create Input.csv
file = open("input.csv", "w")
file.close()
#Set counter to 0
i=0

urlinn = raw_input('Number of search pages you want to enter:\n')
urlinnn=int(urlinn)
for i in range(0, urlinnn):
 print "Please copy & paste search page URL #",i+1,":"
 urlin = raw_input('')
 soup = BeautifulSoup(urllib2.urlopen(urlin).read())
 for link in soup.findAll('a',{'itemprop':'name'}):
     url = link.get('href')
     file = open("Input.csv", "a")
     file.write(url)
     file.write(",")
     file.close()
#Delete the last comma so it doesn't screw up the processing later
with open("input.csv", 'rb+') as filehandle:
    filehandle.seek(-1, os.SEEK_END)
    filehandle.truncate()    
print "Done."   








dlcheck = raw_input('Do you want to download the product pictures? (y/n)\n')
print("\n")
#if the images folder doesn't exist, create a new one
if dlcheck == "y":
 if not os.path.exists('images'):
  os.mkdir('images')


print("Reading input file...")
#open and read Input.csv, split at commas and create a list of URLs

try:
 with open('input.csv') as f:
     content = f.read().split(',')
 print("Input file read.")
 print("Creating output file...")
 #create the Output.csv file and write a header
 file = open("Output.csv", "w")
 file.write('Name,Price,icIMG URL\n')
 file.close()
#get list lenght
 llen=content.__len__()
 #throw an error if the list is invalid
 if llen==1:
  print "Hey! The Input file is empty!"
  print "Please put some URLs in there."
  wait = input("Press Enter to exit and try again.")
#state the number of detected valid URLs
 print "Detected",llen," valid URLs."
 print "\n"
 print "\n"

#fatal error... Hope this doesn't happen :(
except IOError:
   print 'Oh dear.'
   print 'Something went horribly wrong.\n'
   print 'Did you place the Input.csv file into the dist folder \nlike the readme told you to?'
   print 'Do you have the Input file open in some program?'
   print 'Did you give the Input file the correct name?\n'
   wait = input("Press Enter to exit and try again.")

#set counter "i" to 0 and error checker "checker" to None
i=0
errcheck = None

#The main loop. Go through the loop until every list item has been processed
try:
 for i in range(0, llen):
  print "Reading URL #",i+1,"/",llen
  url=content[i]
  doc = urllib2.urlopen(url).read()
  soup = BeautifulSoup(''.join(doc))
#Find the title and write it to the Output file
  try:
   s = soup.find('h1').text

   print s

   file = codecs.open('Output.csv','a','windows-1252')
   file.write('"')
   file.write(s)
   file.write('"')
   file.write(',')
   file.close()
  except:
   print "Product title not readable! Writing as UTF-8 instead... " 
   file = codecs.open('Output.csv','a','utf-8')
   file.write('"')
   file.write(s)
   file.write('"')
   file.write(',')
   file.close()
   errcheck = True
#Find the price and write it to the Output file   
  try:
   ppr = soup.find('span',{'class':'notranslate'}).text
   print "Found article price:"
   print ppr
   newpr = ppr.replace("EUR ", "")
   file = open("Output.csv", "a")
   file.write('"')
   file.write(newpr)
   file.write('"')
   file.write(",")
   file.close()
  except:
    print 'No product price found. Are you sure this URL leads to an eBay article?\n'
    file = open("Output.csv", "a")
    file.write('"')
    file.write(' ')
    file.write('"')
    file.write(",")
    file.close()
    errcheck = True
#Find the icIMG URL and write it to the Output file     
  try:
   imgurl = soup.find('img',id="icImg")['src']
   print "Found picture URL:"
   print imgurl
   file = open("Output.csv", "a")
   file.write('"')
   file.write(imgurl)
   file.write('"')
   file.write(",\n")
   file.close()
   i+=1
   if dlcheck == "y":
    print "Downloading image"

    urllib.urlretrieve(imgurl, os.path.join("images", str(i)+".jpg"))
   print "\n"
  except:
    print 'No icImg ID found. Are you sure this URL leads to an eBay article?\n'
    errcheck = True
#Fatal error again :(
except:   
   print 'Oh dear.\n'
   print 'Something went horribly wrong.\n'
   print 'The Input file is corrupt!'
   print 'Did you check the Input.csv file for mistakes?'
   print 'Pay attention to double commas!'
   file = open("Output.csv", "w")
   file.write('')
   file.close()
   wait = input("Press Enter to exit and try again.")  


#Done :D
print "\n"
print "\n"
print "Done! \n"
print "The data has been saved to Output.csv \n"
#If the program saved images, point the user to their direction
if dlcheck == "y":
 print "The images have been saved to the /images/ folder. \n"
#Warn the user if there have been errors.
if errcheck == True:
 print "WARNING:"
 print "There were encoding errors during processing."
 print "That means that some product titles may not be written correctly\n into the .csv file."
 print "I recommend checking the Output.csv file for mistakes\n and fixing them manually."
#The end!
wait = input("Press Enter to continue.")

This is my first bit of code ever and I'm sure it's full of bad coding practices and weak points. The program is notably slow (fetching the data from the product pages is slow as hell, and I don't know why) and it's pretty prone to crashes.

Answer 1

First things first

When your scripts start to grow unruly like this it is usually a good idea to try and break them down into their component parts and compartmentalize functionality. I'm going to go through just a couple parts of your script with a few ideas that might help to 'modularize'. Doing so can help you with debugging in the future, as well as clarifying your thought process on what you are currently doing or not doing.

I haven't tried to radically re-implement what you're doing but your use of a CSV file as a container for the search page links seems a little unnecessary. Much of your script is spent doing something like:

parsed html >> CSV >> list

It could be easier to instead work directly on the list and data - unless you want to use the CSV files for something else later. The list of HTML links won't be a memory issue unless it explodes to thousands and thousands of links (a 10k character string takes 10033 bytes on my machine)

Modules You may be interested in

Two modules I would recommend are the CSV module, which simplifies much of what you're doing and extends your ability to work with CSV files. Also, for these examples I've used the Requests library rather than urllib/urllib2. This is more a style choice than a critique of your code - but you might look into it and see which you prefer.

Parse HTML function

def build_list_of_links(ebay_page_url):
    page = requests.get(page_url).text
    soup = BeautifulSoup(page)
    list_of_links = []
    for item in soup.find_all('a', {'itemprop':'name'}):
        list_of_links.append(item.get('href'))
    return(list_of_links)

Arguably, the first useful thing your script does is parse the webpage, which is what I've amended above. You mention that your code is currently prone to crashing. This won't necessarily do anything to improve reliability but it can provide a clearer picture of where things are breaking. I have opted to build a list of links rather than writing to a file.

But writing those links to a CSV file could be done like this:

Writing CSV

def write_links_file(output_links_file, links_list):
    with open(file_destination, "a") as output_file:
        link_writer = csv.writer(output_file)
        link_writer.writerow(links_list)

The main thing to note here is the use of the With Statement. It simplifies operations on files by removing the need to open/close the file manually. By calling

    with open(....)

the file will remain open within the scope of the statement.

    file = open("input.csv", "w")
    file.close()

Is unnecessary with the with statement, simply by calling with open... the file will be created if it does not exist currently. By designating the "a" - append flag, you can ensure that you don't lose any data this way either (as you have done).

To achieve the same thing that you have currently you could then just chain those two functions together:

    write_links_file("/some/dir/", build_list_of_links("some.page"))

Nothing says this is the best way to do it - but it is much more apparent what is happening and where the execution is taking place.

Jumping down a bit further, I've written your image directory check into a function:

If Not vs. If/Else

def create_img_dir():
  save_imgages = raw_input('Do you want to download the product pictures? (y/n)\n\n')
  if save_imgages == "y":
      if os.path.exists('images'):
          pass
      else:
          os.mkdir('images')
  else:
    ...

This is a readability issue and my personal preference, but instead of

    if not os.path.exists

I would opt for:

    if os.path.exists
        pass
    else
        os.mkdir

which is functionally identical but now reads more like what is actually happening (to me anyway).

Various things to note:

• instead of

  content.__len__() 
  

  you can call

  len(content)
  

• instead of

  for i in range(0, urlinnn) 
  

  is the same as

  for i in range(urlinnn)
  

• urlin/urlinn/urlinnn are totally inscrutable variable names

• Writing:

   print 'Oh dear.\n'
   print 'Something went horribly wrong.\n'
   print 'The Input file is corrupt!'
   print 'Did you check the Input.csv file for mistakes?'
   print 'Pay attention to double commas!'
  

  can also be written with a single print statement like this:

   print("Oh dear.\n\
   Something went horribly wrong.\n\
   The Input file is corrupt!\n\
   Did you check the Input.csv file for mistakes?\n\
   Pay attention to double commas!")
  

You've actually done most of the hard work already in getting it to work. I think a good next step would be to go through your script and adjust things into logical 'partitions' (this section reads/writes CSV, this sections gets webpage data, this section logs errors etc.). That and improving your variable names!