wxPython Item Information Scraper

Question

1. What is the best method/practice to keep, let's say, over 3,000 lines of code organized for readability?
2. What kind of NO-NOs in regards to coding habits should I get rid of before they become bad habits?
3. Inversely what kind of good coding habits should I be picking up?
4. Any glaringly obvious mistakes? Any helpful links for further reading would be appreciated.

I am including code that I have been working on for the past couple weeks. Basically it will help to increase the efficiency of my workflow from pulling inventory off trucks, to getting it to where it needs to go.

The following code navigates to a website and extracts information about an Item.

# -*- coding: utf8 -*-
import csv, time, os, random, difflib, re, json
from itertools import islice
import httplib
import urllib2
from bs4 import BeautifulSoup as BS
from shutil import copyfile

def get_product_info(item_number):
    ''' Returns a dictionary of item information '''



    # offer_id_orig = '#NK420#########'
    offer_id_orig = item_number
    item_uts_number = offer_id_orig[1:6]
    print offer_id_orig


    # DOWNLOAD PRODUCT INFO IF WE DON"T HAVE IT
    def download_page_info(item_uts_number):        
        #import time, os, random
        #import httplib
        #import urllib2
        # ITEM_NUMBER MUST BE 4NK4200000100 FORMAT
        # assigned_row_titles 'offer_id_orig' : current_row[2]



        print item_uts_number


        httplib.HTTPConnection.debuglevel = 1

        # take multiple arguments? 

        fp = os.path.join(folder,item_uts_number)
        print "Fetching:"+fp+"\n"
        try:
            request = urllib2.Request('http://www.fingerhut.com/product/'+item_uts_number.upper()+'.uts')
            request.add_header('User-Agent','jmunsch_thnx_v2.0 +http://jamesmunsch.com/')
            opener = urllib2.build_opener()
            data = opener.open(request).read()
            with open(fp,'w+') as f:
               f.write(data)
            time.sleep(1+random.random())
            print "Fetched."
        except Exception,e:print e
        return data


    # Get the images
    def get_images(fp,item_number):
        #### TODO ###
        # get the correct schema
        # will need to use selenium for this? since the page loads with jquery and javascript
        ################## 3-16-14
        # labeling schema for images itemnumber_letter_999

        #### This is not require if I get image url from the original init variables from html/json
#           letters = [ 'A','B','C','D','E','F','G','VA','VB','VC','VD','VE','VF','VG' ]
#           for letter in letters:

#               url_string = item_number+"_"+letter+"_999?"
            #+"&scl=1"
            #+"$swatch$"
#               imgName = item_number+"_"+letter+"_999"


            try:
                # build the url 
                image_url = "http://s7d5.scene7.com/is/image/bluestembrands/"+url_string
                print image_url
                # build request header
                user_agent = 'jmunsch_v3 (+http://jamesmunsch.com/)'
                headers = { 'User-Agent' : user_agent }
                imgRequest = urllib2.Request(image_url, headers=headers)
                # get the image data
                imgData = urllib2.urlopen(imgRequest).read()
                #write teh image data
                with open(os.path.join(fp,imgName),'w+') as imgFile:
                    imgFile.write(imgData)

            # except problem i.e. wrong filepath / 403 / incorrect request  
            except Exception,e:
                print "Oops: "+str(e)

    # Get Item info ( description, specs, picUrl )
    def extract_product_info(data):
        ''' Extract product info from the data from download_page_info '''
        #from bs4 import BeautifulSoup as BS
        #id="productTabs_tab1"
        #id="productTabs_tab3"





        # Grab Description
        try:
            json_data = data.split("page.init(")[1].split("});")[0]
            json_data = json_data + "}"
            print "Trying to load json"
            json_data = json.loads(json_data.replace('\\uXXXX',''))
            description = json_data['product']['description']
            d_soup = BS(description)
            # Remove tags with information that can lead back to FH
            for tag in d_soup.find_all('h3'):
                tag.replaceWith('')
            for tag in d_soup.find_all('script'):
                tag.replaceWith('')
            for tag in d_soup.find_all('a'):
                tag.replaceWith('')
            for tag in d_soup.find_all('li'):
                if "Available" in tag.text:
                    tag.replaceWith('')

            description = d_soup
        except Exception,e:
            print e
            print 'unable to get description'
            #print description
            print json_data
            description = None

        # Grab Specifications
        try:
            json_data = data.split("page.init(")[1].split("});")[0]
            json_data = json_data + "}"
            print "Trying to load json"
            json_data = json.loads(json_data)
            specifications = json_data['product']['specifications']
            specs ="<br>".join(x for x in specifications)
        except Exception,e:
            print e
            print 'unable to get specifications'
            print json_data
            #print specs
            specs = None




        # Grab Images
        image_list = []
        try:
            json_data = data.split("page.init(")[1].split("});")[0]
            json_data = json_data + "}"
            print "Trying to load json"
            json_data = json.loads(json_data)
            images = json_data['product']['media']['image']

            if len(images) is not 0:    
                for url in images:
                    tmp_str = url['hiRes'].encode('utf8').split('?')[0].split('/')[-1]
                    #print tmp_str
                    image_list.append(tmp_str)
            if len(images) is 0:
                print " images is empty "
                split_data = data.split('\"')
                for line in split_data:
                    if ( offer_id_orig[1:6] and "is/bluestembrands" ) in line:
                        image_list.append(line.split('/')[-1].split('?')[0])
            if len(image_list) is 0:
                raise NameError('imgList0')


        except NameError,n:
            print "\n###"+str(n)
            if 'imgList0' in n:
                print '#~#   Didnt find any images...? Good luck.'
                image_list = [offer_id_orig]
                pass
        except Exception,e:
            print e
            #print json_data
            print " Unable to get image_list "
            image_list = None


        image_list = list(set(image_list))
        if len(image_list) is not 0:
            print "Got Images."

        results = { 'description' : description ,
                    'specs' : specs ,
                    'image_list' : image_list 
                    }

        return results



    # Check if item page has been downloaded
    # Get page data
    # Make item folder
    try:
        folder = 'item_number_pages'
        os.mkdir(folder)
    except Exception,e:print e


    fp = os.path.join(folder,item_uts_number)

    # Check if the file exists if not download it
    if os.path.isfile(fp) is True:
        data = open(fp,'r').read()
    elif os.path.isfile(fp) is False:
        data = download_page_info(item_uts_number)              
    else:
        print "what?"
        exit()
    # extract product info returns
    # 'description','specs','image_list'
    #
    ########################################

    results = extract_product_info(data)
    print results
    return results

And the wxPython code that I am hacking together will display the Item Picture and allow me to make changes to the item information. I also plan to create a dialogue that shows all the images for a particular SKU, some features for inventorying/receiving these items, and eventually write this out to a csv the code below is unfinished:

import wx, urllib2, os
from PIL import Image

class PhotoCtrl(wx.Frame):
    def __init__(self, parent, title):
        wx.Frame.__init__(self, parent, title=title)
        #self.frame = wx.Frame(None,'Sky Group Scanner')
        #self.frame = wx.Frame(None, title='Photo Control')

        self.panel = wx.Panel(self)
        self.PhotoMaxSize = 240
        self.createWidgets()
        self.Show()


    def createWidgets(self):

        filepath = fp

        if fp is not None:
            img = wx.Image(fp, wx.BITMAP_TYPE_ANY)
        if fp is None:
            img = wx.EmptyImage(240,240)

        self.imageCtrl = wx.StaticBitmap(self.panel, wx.ID_ANY, 
                                         wx.BitmapFromImage(img))

        instructions = 'Is this image correct?'
        instructLbl = wx.StaticText(self.panel, label=instructions)
        # menubar
        #####################
        file = wx.Menu()
        edit = wx.Menu()
        help = wx.Menu()
        file.Append(101,'&Open', 'Open a file')
        file.Append(102,'&Save', 'Save a file')
        file.AppendSeparator()
        quit = wx.MenuItem(file, 105, '&Quit\tCtrl+Q', 'Quit the Application')
        file.AppendItem(quit)
        menubar = wx.MenuBar()
        menubar.Append(file, '&File')
        menubar.Append(edit,'&Edit')
        menubar.Append(help,'&Help')
        self.SetMenuBar(menubar)
        self.CreateStatusBar()
        #photo txt and browse Button
        ##################
        self.photoTxt = wx.TextCtrl(self.panel, size=(200,-1))
        browseBtn = wx.Button(self.panel, label='Browse')
        browseBtn.Bind(wx.EVT_BUTTON, self.onBrowse)
        SelectImageBtn = wx.Button(self.panel, label="Select Image Choices")
        SelectImageBtn.Bind(wx.EVT_BUTTON, self.onBrowse)
        # Description Text
        ##########################
        descriptionTextField = wx.TextCtrl(self.panel,size=(300,500), style=wx.TE_MULTILINE | wx.TE_RICH2)
        descriptionTextField.AppendText(str(itemInfo['specs'].decode('utf8')+itemInfo['description'].decode('utf8')))
        # Add Widgets
        #########################
        self.mainSizer = wx.BoxSizer(wx.HORIZONTAL)
        self.lSizer = wx.BoxSizer(wx.VERTICAL)
        self.rSizer = wx.BoxSizer(wx.HORIZONTAL)

        self.mainSizer.Add(wx.StaticLine(self.panel, wx.ID_ANY),
                           0, wx.ALL|wx.EXPAND, 5)

        self.lSizer.Add(instructLbl, 0, wx.ALL, 5)
        self.lSizer.Add(self.imageCtrl, 0, wx.ALL, 5)
        self.lSizer.Add(self.photoTxt, 0, wx.ALL, 5)
        self.lSizer.Add(browseBtn, 0, wx.ALL, 5)
        self.lSizer.Add(SelectImageBtn, 0, wx.ALL, 5)

        self.rSizer.Add(descriptionTextField, 0, wx.ALL, 5)

        self.mainSizer.Add(self.lSizer, flag=wx.ALIGN_LEFT)
        self.mainSizer.Add(self.rSizer, flag=wx.ALIGN_RIGHT)



        self.panel.SetSizer(self.mainSizer)
        self.mainSizer.Fit(self)

        self.panel.Layout()

    def onBrowse(self, event):
        """ 
        Browse for file
        """
        wildcard = "JPEG files (*.jpg)|*.jpg"
        dialog = wx.FileDialog(None, "Choose a file",
                               wildcard=wildcard,
                               style=wx.OPEN)
        if dialog.ShowModal() == wx.ID_OK:
            self.photoTxt.SetValue(dialog.GetPath())
        dialog.Destroy() 
        self.onView()


    def onView(self):
        #filepath = self.photoTxt.GetValue()
        #img = wx.Image(filepath, wx.BITMAP_TYPE_ANY)

        filepath = fp
        img = wx.Image(fp, wx.BITMAP_TYPE_ANY)
        # scale the image, preserving the aspect ratio
        W = img.GetWidth()
        H = img.GetHeight()
        if W > H:
            NewW = self.PhotoMaxSize
            NewH = self.PhotoMaxSize * H / W
        else:
            NewH = self.PhotoMaxSize
            NewW = self.PhotoMaxSize * W / H
        img = img.Scale(NewW,NewH)

        self.imageCtrl.SetBitmap(wx.BitmapFromImage(img))
        self.panel.Refresh()




def get_image(item_info,item_number):
    global fp
    global itemInfo
    global itemNumber

    itemInfo = item_info
    itemNumber = item_number

    try:
        uts_number = item_info['image_list'][0]
        uts_ = uts_number + '?'
        folder = 'item_number_pages'
        imgName = item_number+'.JPEG'
        fp = os.path.join(folder,imgName)

        if os.path.isfile(fp) is False:

            print uts_number
            # build the url 
            image_url = "http://s7d5.scene7.com/is/image/bluestembrands/"+uts_
            print image_url
            # build request header
            user_agent = 'jmunsch_v3 (+http://jamesmunsch.com/)'
            headers = { 'User-Agent' : user_agent }
            imgRequest = urllib2.Request(image_url, headers=headers)
            # get the image data
            imgData = urllib2.urlopen(imgRequest).read()


            #write teh image data
            # uh... windows requires wb ... 
            with open(fp,'wb') as imgFile:
                print('Writing Image:' + fp)
                imgFile.write(imgData)

    # except problem i.e. wrong filepath / 403 / incorrect request  
    except Exception,e:
        print "display_image: Oops1: "+str(e)

    app = wx.App(False)
    frame = PhotoCtrl(None,'Sky Group Scanner')
    app.MainLoop()
    return

if __name__ == '__main__':
    global fp
    global itemInfo
    global itemNumber
    fp = None
    itemInfo = {'description':'Empty','specs':'Empty'}
    itemNumber = "Empty"
    app = wx.App(False)
    frame = PhotoCtrl(None,'A Simple Scanner')
    app.MainLoop()

Update:

Based off of the recommendations of @alexwlchan I have begun using a dedicated IDE for python called Ninja-IDE. It has been pretty handy in comparison to using NotePad++. It is a little slow in regards to searching for multiple instances of a search term and the implemented terminal doesn't remember history, but overall it has been a good fit. The IDE I think is written in python so could be coded and possibly merged if the Ninja team is interested, but I haven't dug into that code.

I also began using github a little more extensively for peace of mind and also to have backups of backups and to keep a basepoint for a working version in case I really mess my code somewhere. http://github.com/jmunsch/Sky_Group_Inventory_Scanner-wxpython

And also instead of littering my code with a bunch of print statements for debugging, I have been using import pdb; pdb.set_trace() right before the code that I want to step through and debug.

Also, Modularizing the code has been really helpful.

Answer 1

As a piece of code for a “beginner” (reading from the tag), this is pretty good. I’ll start by offering some suggestions on your specific questions, and then make some comments on the code itself:

1. What is the best method/practice to keep, let's say, over 3,000 lines of code organized for readability?

   • Scratch readability for now, just keeping 3,000 lines of code organised will get painful quickly. Are you using some sort of version control system (VCS) with the code? If not, I’d recommend looking into one. As you maintain and extend the code, you can keep “snapshots” of working versions or experimental techniques. Using a VCS, and committing regularly, is a really good habit for larger codebases.

     The Rypress tutorial is a good introduction to Git, a fairly common VCS. You can either run a VCS locally, or Bitbucket offers free private repositories. (Bitbucket also have tutorials for Git and Mercurial when you sign up.)

   • Put TODO notes in a separate file. Otherwise it gets hard to find and will get lost in the mix.

   • Write docstrings for all of your non-trivial functions. You remember what onView() is meant to do right now, but you might find it harder to remember in six months time when you’re debugging it. Read PEP 257 for the Python standards for writing docstrings.

   • Don’t litter the file with multiple newlines between functions or within functions. One is quite enough, and doesn’t break the file up so much.

2. What kind of NO-NOs in regards to coding habits should I get rid of before they become bad habits?

   • Don’t just copy-and-paste special strings into the code; declare them as global variables at the top of the file and use them as appropriate. This means you can only change them once when you need to.

     For example, these lines from the two different files give me pause for thought:

     request.add_header('User-Agent','jmunsch_thnx_v2.0 +http://jamesmunsch.com/')
     user_agent = 'jmunsch_v3 (+http://jamesmunsch.com/)'
     

     Is there a reason for the two user agent strings to have a different format? And why do they have different version numbers? etc.

   • Have you read PEP 8? This is the Python style guide. Overall you’re pretty good, but there are one or two things that could be tweaked:

     • Spacing around commas goes like an English sentence: none before, one after. So except Exception,e:print e becomes

       except Exception, e:
           print e
       

       Also, put blocks after a colon like this on a new line.

     • Same goes for whitespace around binary operators. So print "Oops: "+str(e) becomes print "Oops: " + str(e).

     Style issues aren’t going to break your code, but it will make it easier for other people to read and debug your code (and conversely, it will make it easier for you to read other people’s Python), because the styles match.

3. Inversely what kind of good coding habits should I be picking up?

   Don’t do what I said not to do in 2. ;)

4. Any glaringly obvious mistakes?

   Beyond what I’ve already mentioned, nothing stands out as being terrible. (At least, to my eyes.)

   If you want more on good programming practice, then I’d suggest reading a copy of The Pragmatic Programmer, which contains some excellent advice on good style (including most of what I wrote above, and a lot more).