# wxPython Item Information Scraper

1. What is the best method/practice to keep, let's say, over 3,000 lines of code organized for readability?
2. What kind of NO-NOs in regards to coding habits should I get rid of before they become bad habits?
3. Inversely what kind of good coding habits should I be picking up?
4. Any glaringly obvious mistakes? Any helpful links for further reading would be appreciated.

I am including code that I have been working on for the past couple weeks. Basically it will help to increase the efficiency of my workflow from pulling inventory off trucks, to getting it to where it needs to go.

The following code navigates to a website and extracts information about an Item.

# -*- coding: utf8 -*-
import csv, time, os, random, difflib, re, json
from itertools import islice
import httplib
import urllib2
from bs4 import BeautifulSoup as BS
from shutil import copyfile

def get_product_info(item_number):
''' Returns a dictionary of item information '''

# offer_id_orig = '#NK420#########'
offer_id_orig = item_number
item_uts_number = offer_id_orig[1:6]
print offer_id_orig

# DOWNLOAD PRODUCT INFO IF WE DON"T HAVE IT
#import time, os, random
#import httplib
#import urllib2
# ITEM_NUMBER MUST BE 4NK4200000100 FORMAT
# assigned_row_titles 'offer_id_orig' : current_row[2]

print item_uts_number

httplib.HTTPConnection.debuglevel = 1

# take multiple arguments?

fp = os.path.join(folder,item_uts_number)
print "Fetching:"+fp+"\n"
try:
request = urllib2.Request('http://www.fingerhut.com/product/'+item_uts_number.upper()+'.uts')
opener = urllib2.build_opener()
with open(fp,'w+') as f:
f.write(data)
time.sleep(1+random.random())
print "Fetched."
except Exception,e:print e
return data

# Get the images
def get_images(fp,item_number):
#### TODO ###
# get the correct schema
# will need to use selenium for this? since the page loads with jquery and javascript
################## 3-16-14
# labeling schema for images itemnumber_letter_999

#### This is not require if I get image url from the original init variables from html/json
#           letters = [ 'A','B','C','D','E','F','G','VA','VB','VC','VD','VE','VF','VG' ]
#           for letter in letters:

#               url_string = item_number+"_"+letter+"_999?"
#+"&scl=1"
#+"$swatch$"
#               imgName = item_number+"_"+letter+"_999"

try:
# build the url
image_url = "http://s7d5.scene7.com/is/image/bluestembrands/"+url_string
print image_url
# build request header
user_agent = 'jmunsch_v3 (+http://jamesmunsch.com/)'
headers = { 'User-Agent' : user_agent }
# get the image data
#write teh image data
with open(os.path.join(fp,imgName),'w+') as imgFile:
imgFile.write(imgData)

# except problem i.e. wrong filepath / 403 / incorrect request
except Exception,e:
print "Oops: "+str(e)

# Get Item info ( description, specs, picUrl )
def extract_product_info(data):
''' Extract product info from the data from download_page_info '''
#from bs4 import BeautifulSoup as BS
#id="productTabs_tab1"
#id="productTabs_tab3"

# Grab Description
try:
json_data = data.split("page.init(")[1].split("});")[0]
json_data = json_data + "}"
print "Trying to load json"
description = json_data['product']['description']
d_soup = BS(description)
# Remove tags with information that can lead back to FH
for tag in d_soup.find_all('h3'):
tag.replaceWith('')
for tag in d_soup.find_all('script'):
tag.replaceWith('')
for tag in d_soup.find_all('a'):
tag.replaceWith('')
for tag in d_soup.find_all('li'):
if "Available" in tag.text:
tag.replaceWith('')

description = d_soup
except Exception,e:
print e
print 'unable to get description'
#print description
print json_data
description = None

# Grab Specifications
try:
json_data = data.split("page.init(")[1].split("});")[0]
json_data = json_data + "}"
print "Trying to load json"
specifications = json_data['product']['specifications']
specs ="<br>".join(x for x in specifications)
except Exception,e:
print e
print 'unable to get specifications'
print json_data
#print specs
specs = None

# Grab Images
image_list = []
try:
json_data = data.split("page.init(")[1].split("});")[0]
json_data = json_data + "}"
print "Trying to load json"
images = json_data['product']['media']['image']

if len(images) is not 0:
for url in images:
tmp_str = url['hiRes'].encode('utf8').split('?')[0].split('/')[-1]
#print tmp_str
image_list.append(tmp_str)
if len(images) is 0:
print " images is empty "
split_data = data.split('\"')
for line in split_data:
if ( offer_id_orig[1:6] and "is/bluestembrands" ) in line:
image_list.append(line.split('/')[-1].split('?')[0])
if len(image_list) is 0:
raise NameError('imgList0')

except NameError,n:
print "\n###"+str(n)
if 'imgList0' in n:
print '#~#   Didnt find any images...? Good luck.'
image_list = [offer_id_orig]
pass
except Exception,e:
print e
#print json_data
print " Unable to get image_list "
image_list = None

image_list = list(set(image_list))
if len(image_list) is not 0:
print "Got Images."

results = { 'description' : description ,
'specs' : specs ,
'image_list' : image_list
}

return results

# Check if item page has been downloaded
# Get page data
# Make item folder
try:
folder = 'item_number_pages'
os.mkdir(folder)
except Exception,e:print e

fp = os.path.join(folder,item_uts_number)

# Check if the file exists if not download it
if os.path.isfile(fp) is True:
elif os.path.isfile(fp) is False:
else:
print "what?"
exit()
# extract product info returns
# 'description','specs','image_list'
#
########################################

results = extract_product_info(data)
print results
return results


And the wxPython code that I am hacking together will display the Item Picture and allow me to make changes to the item information. I also plan to create a dialogue that shows all the images for a particular SKU, some features for inventorying/receiving these items, and eventually write this out to a csv the code below is unfinished:

import wx, urllib2, os
from PIL import Image

class PhotoCtrl(wx.Frame):
def __init__(self, parent, title):
wx.Frame.__init__(self, parent, title=title)
#self.frame = wx.Frame(None,'Sky Group Scanner')
#self.frame = wx.Frame(None, title='Photo Control')

self.panel = wx.Panel(self)
self.PhotoMaxSize = 240
self.createWidgets()
self.Show()

def createWidgets(self):

filepath = fp

if fp is not None:
img = wx.Image(fp, wx.BITMAP_TYPE_ANY)
if fp is None:
img = wx.EmptyImage(240,240)

self.imageCtrl = wx.StaticBitmap(self.panel, wx.ID_ANY,
wx.BitmapFromImage(img))

instructions = 'Is this image correct?'
instructLbl = wx.StaticText(self.panel, label=instructions)
#####################
file.Append(101,'&Open', 'Open a file')
file.Append(102,'&Save', 'Save a file')
file.AppendSeparator()
quit = wx.MenuItem(file, 105, '&Quit\tCtrl+Q', 'Quit the Application')
file.AppendItem(quit)
self.CreateStatusBar()
#photo txt and browse Button
##################
self.photoTxt = wx.TextCtrl(self.panel, size=(200,-1))
browseBtn = wx.Button(self.panel, label='Browse')
browseBtn.Bind(wx.EVT_BUTTON, self.onBrowse)
SelectImageBtn = wx.Button(self.panel, label="Select Image Choices")
SelectImageBtn.Bind(wx.EVT_BUTTON, self.onBrowse)
# Description Text
##########################
descriptionTextField = wx.TextCtrl(self.panel,size=(300,500), style=wx.TE_MULTILINE | wx.TE_RICH2)
descriptionTextField.AppendText(str(itemInfo['specs'].decode('utf8')+itemInfo['description'].decode('utf8')))
#########################
self.mainSizer = wx.BoxSizer(wx.HORIZONTAL)
self.lSizer = wx.BoxSizer(wx.VERTICAL)
self.rSizer = wx.BoxSizer(wx.HORIZONTAL)

0, wx.ALL|wx.EXPAND, 5)

self.lSizer.Add(instructLbl, 0, wx.ALL, 5)
self.lSizer.Add(self.imageCtrl, 0, wx.ALL, 5)
self.lSizer.Add(self.photoTxt, 0, wx.ALL, 5)
self.lSizer.Add(browseBtn, 0, wx.ALL, 5)
self.lSizer.Add(SelectImageBtn, 0, wx.ALL, 5)

self.rSizer.Add(descriptionTextField, 0, wx.ALL, 5)

self.panel.SetSizer(self.mainSizer)
self.mainSizer.Fit(self)

self.panel.Layout()

def onBrowse(self, event):
"""
Browse for file
"""
wildcard = "JPEG files (*.jpg)|*.jpg"
dialog = wx.FileDialog(None, "Choose a file",
wildcard=wildcard,
style=wx.OPEN)
if dialog.ShowModal() == wx.ID_OK:
self.photoTxt.SetValue(dialog.GetPath())
dialog.Destroy()
self.onView()

def onView(self):
#filepath = self.photoTxt.GetValue()
#img = wx.Image(filepath, wx.BITMAP_TYPE_ANY)

filepath = fp
img = wx.Image(fp, wx.BITMAP_TYPE_ANY)
# scale the image, preserving the aspect ratio
W = img.GetWidth()
H = img.GetHeight()
if W > H:
NewW = self.PhotoMaxSize
NewH = self.PhotoMaxSize * H / W
else:
NewH = self.PhotoMaxSize
NewW = self.PhotoMaxSize * W / H
img = img.Scale(NewW,NewH)

self.imageCtrl.SetBitmap(wx.BitmapFromImage(img))
self.panel.Refresh()

def get_image(item_info,item_number):
global fp
global itemInfo
global itemNumber

itemInfo = item_info
itemNumber = item_number

try:
uts_number = item_info['image_list'][0]
uts_ = uts_number + '?'
folder = 'item_number_pages'
imgName = item_number+'.JPEG'
fp = os.path.join(folder,imgName)

if os.path.isfile(fp) is False:

print uts_number
# build the url
image_url = "http://s7d5.scene7.com/is/image/bluestembrands/"+uts_
print image_url
# build request header
user_agent = 'jmunsch_v3 (+http://jamesmunsch.com/)'
headers = { 'User-Agent' : user_agent }
# get the image data

#write teh image data
# uh... windows requires wb ...
with open(fp,'wb') as imgFile:
print('Writing Image:' + fp)
imgFile.write(imgData)

# except problem i.e. wrong filepath / 403 / incorrect request
except Exception,e:
print "display_image: Oops1: "+str(e)

app = wx.App(False)
frame = PhotoCtrl(None,'Sky Group Scanner')
app.MainLoop()
return

if __name__ == '__main__':
global fp
global itemInfo
global itemNumber
fp = None
itemInfo = {'description':'Empty','specs':'Empty'}
itemNumber = "Empty"
app = wx.App(False)
frame = PhotoCtrl(None,'A Simple Scanner')
app.MainLoop()


# Update:

Based off of the recommendations of @alexwlchan I have begun using a dedicated IDE for python called Ninja-IDE. It has been pretty handy in comparison to using NotePad++. It is a little slow in regards to searching for multiple instances of a search term and the implemented terminal doesn't remember history, but overall it has been a good fit. The IDE I think is written in python so could be coded and possibly merged if the Ninja team is interested, but I haven't dug into that code.

I also began using github a little more extensively for peace of mind and also to have backups of backups and to keep a basepoint for a working version in case I really mess my code somewhere. http://github.com/jmunsch/Sky_Group_Inventory_Scanner-wxpython

And also instead of littering my code with a bunch of print statements for debugging, I have been using import pdb; pdb.set_trace() right before the code that I want to step through and debug.

Also, Modularizing the code has been really helpful.

• Could you please explain the purpose of this code? Titles here should also reflect on that.
– Jamal
Apr 3, 2014 at 23:14
• Welcome to CR! Thanks for putting in the effort, +1 ;) Apr 4, 2014 at 0:22

As a piece of code for a “beginner” (reading from the tag), this is pretty good. I’ll start by offering some suggestions on your specific questions, and then make some comments on the code itself:

1. What is the best method/practice to keep, let's say, over 3,000 lines of code organized for readability?

• Scratch readability for now, just keeping 3,000 lines of code organised will get painful quickly. Are you using some sort of version control system (VCS) with the code? If not, I’d recommend looking into one. As you maintain and extend the code, you can keep “snapshots” of working versions or experimental techniques. Using a VCS, and committing regularly, is a really good habit for larger codebases.

The Rypress tutorial is a good introduction to Git, a fairly common VCS. You can either run a VCS locally, or Bitbucket offers free private repositories. (Bitbucket also have tutorials for Git and Mercurial when you sign up.)

• Put TODO notes in a separate file. Otherwise it gets hard to find and will get lost in the mix.

• Write docstrings for all of your non-trivial functions. You remember what onView() is meant to do right now, but you might find it harder to remember in six months time when you’re debugging it. Read PEP 257 for the Python standards for writing docstrings.

• Don’t litter the file with multiple newlines between functions or within functions. One is quite enough, and doesn’t break the file up so much.

2. What kind of NO-NOs in regards to coding habits should I get rid of before they become bad habits?

• Don’t just copy-and-paste special strings into the code; declare them as global variables at the top of the file and use them as appropriate. This means you can only change them once when you need to.

For example, these lines from the two different files give me pause for thought:

request.add_header('User-Agent','jmunsch_thnx_v2.0 +http://jamesmunsch.com/')
user_agent = 'jmunsch_v3 (+http://jamesmunsch.com/)'


Is there a reason for the two user agent strings to have a different format? And why do they have different version numbers? etc.

• Have you read PEP 8? This is the Python style guide. Overall you’re pretty good, but there are one or two things that could be tweaked:

• Spacing around commas goes like an English sentence: none before, one after. So except Exception,e:print e becomes

except Exception, e:
print e


Also, put blocks after a colon like this on a new line.

• Same goes for whitespace around binary operators. So print "Oops: "+str(e) becomes print "Oops: " + str(e).

Style issues aren’t going to break your code, but it will make it easier for other people to read and debug your code (and conversely, it will make it easier for you to read other people’s Python), because the styles match.

3. Inversely what kind of good coding habits should I be picking up?

Don’t do what I said not to do in 2. ;)

4. Any glaringly obvious mistakes?

Beyond what I’ve already mentioned, nothing stands out as being terrible. (At least, to my eyes.)

If you want more on good programming practice, then I’d suggest reading a copy of The Pragmatic Programmer, which contains some excellent advice on good style (including most of what I wrote above, and a lot more).

• Nice. The Rypress git tutorial looks really great. I've messed with git a little bit, but not to keep code organized with versions. I've just been saving copies with the date and throwing them in a folder. I look forward to reading the book suggestion and styling guide. I really appreciate your time and suggestions. Apr 4, 2014 at 14:44
• @alexslchan I also started using an IDE... I had been using notepad++. It incorporates a PEP8 format checker, plugins for Git, and a bunch of other goodies that I haven't dived into yet, but this looks promising. Using NINJA-IDE since I have been coding mostly in python. Apr 7, 2014 at 16:35