# Content filtering of webpage

This is code for filtering the data of webpage, for the web crawler I made for my project. I know python scripts can lag than other languages, but this takes a lot of time when processing even a single page.

I don't want to use any other external libraries for filtering content. Is there any way my current code can be improved to be cleaner and faster?

# -*- coding: utf-8 -*-
import urllib
def content(page,url):#FILTERS THE CONTENT OF THE REMAINING PORTION
flg=0
#REMOVES &nsbp LIKE CHARACTERS
while page.find("&",flg)!=-1:
page.replace('&nbsp;','')
start=page.find("&",flg)
end=page.find(";",start+1)
if (end-start)<10:  #USED IF HERE TO CNFRM TAGS-->REMOVE IF NOT NEEDED
pageO=page[:start]
pageT=page[end+1:]
page=pageO+pageT
flg=start+1#TO CONTINUE FROM NEXT POS
else:
flg+=1
flg=0

#REMOVES CONTENT BETWEEN SCRIPT TAGS
while page.find("<script",flg)!=-1:
start=page.find("<script",flg)
end=page.find("</script>",flg)
end=end+9
i,k=0,end-start
page=list(page)
while i<k:
page.pop(start)
i=i+1
page=''.join(page)
flg=start

#REMOVES CONTENT BETWEEN STYLE TAGS
flg=0
while page.find("<style",flg)!=-1:
start=page.find("<style",flg)
end=page.find("</style>",flg)
end=end+9
i,k=0,end-start
page=list(page)
while i<k:
page.pop(start)
i=i+1
page=''.join(page)
flg=start

#REMOVES THE TAGS
s_list = list(page)
i,j = 0,0
while i < len(s_list):
# find the <
if s_list[i] == '<':
while s_list[i] != '>':# and i!=(len(s_list)-1):
# remove everything between the < and the >
s_list.pop(i)

# make sure we get rid of the > to
s_list.pop(i)
else:
i=i+1
#-------------------------------------------------------------------

#REMOVES WHITESPACES
s_list="".join(s_list)
lst=s_list.split()
#CONVERT TO LOWERCASE
i=0
while i<len(lst):
lst[i]=lst[i].lower()
i+=1

#REMOVES DUPLICATES
lst=list(set(lst))

#REMOVE COMMON WORDS
phrase=['to','a','an','the',\
'for','from','that','their',\
'i','my','your','you','mine',\
'we','okay','yes','no','as',\
'if','but','why','can','now',\
'are','is','also',',','.',';',\
':','?','|','/','\n','\t']

i=0
while i<len(lst):
if lst[i] in phrase:
lst.pop(i)
else:
i+=1
print lst
print len(lst)

def pageContent(url):#EXTRACTS HTML CODE
f = urllib.urlopen(url)
f.close()
#page=page.replace(u'\xa0', ' ').encode('utf-8','ignore')
return page

page=pageContent(url)
content(page,url)


Please mind the comments. I left it so it could be of some help.

• Obligatory suggestion to read and consider following the style guide. You say "outside libraries" - is the standard library OK? – jonrsharpe Aug 8 '14 at 13:06
• Why not external libraries? BeautifulSoup can be helpful here. – jcollado Aug 8 '14 at 14:18
• @jcollado i know about BeautifulSoup but my mentor has told me to do this without using any other libraries. – Chetan Aug 10 '14 at 9:43
• @jonrsharpe yeah standard libraries are OK. – Chetan Aug 10 '14 at 9:44

Per my comment, follow the style guide - for example, there should be whitespace around = when assigning, and after commas:

i, k = 0, end - start


content is not a good name for your function. You should be more descriptive of what it actually does (perhaps filter_content?) and add a docstring providing more information. Throughout your code there are temporary variables with cryptic names (s_list? lst?) that could be changed to make things much clearer - I was wondering why flg isn't Boolean, and it turns out that it isn't actually a flag.

Your approach to removing HTML tags (picking through the whole page character by character) is particularly prone to error; what if one of the attributes within a tag contains '>'? For a good standard library solution, see here.

The conversion to lowercase is, frankly, ludicrous:

i=0
while i<len(lst):
lst[i]=lst[i].lower()
i+=1


you had the whole string (called, confusingly, s_list) to hand just two lines beforehand, and

s_list = s_list.lower()


is so much simpler.

As you're making a set to remove duplication:

lst=list(set(lst))


why not keep the set, instead of converting back to list, and use it to do the filtering, too? For example, use set.difference_update:

>>> words = set('this is a sentence to filter'.split())
>>> words.difference_update(['a', 'to', 'this', 'is'])
>>> words
set(['sentence', 'filter'])


Your other function could be simplified significantly:

def page_content(url):
with urllib.urlopen(url) as f: