2
\$\begingroup\$

I recently made a program in Python to open a list of URLs and split the queries into different files. I want to make this code more generic and simple. I am open to suggestions.

def parse_file():
    # Open the file for reading
    infile = open("URLlist.txt", 'r')
    # Read every single line of the file into an array of lines
    lines = infile.readlines()
    # For every line in the array of lines, do something with that line
    for line in lines: # for every line in lines. 
        print line # print the URL
        # parse the URL query string list
        # if the URL has a query component print out the the first
        # argument(k) and if the URL has multiple arguments, print out
        # the other arguments. (v)
        for k,v in urlparse.parse_qsl(urlparse.urlparse(line).query): 
            print k, v

            if k == "blog": # if the printed out arguements have blog in it
                           # write it out to the apropriate file and do this
                           # to all the others.
                with open('blog_file.txt'.format(),'a') as f:
                    f.write(v)
            elif k == 'p':
                with open('p_file.txt'.format(),'a') as f:
                    f.write(v)
            elif k == 'attachment_id':
                with open('attachment_id_file.txt'.format(),'a') as f:
                    f.write(v)
            elif k == 'lang':
                with open('lang_file.txt'.format(),'a') as f:
                    f.write(v)
            elif k == 'portfolio':
                with open('portfolio_file.txt'.format(),'a') as f:
                    f.write(v)
            elif k == 'page_id':
                with open('page_id_file.txt'.format(),'a') as f:
                    f.write(v)
            elif k == 'comments_popup':
                with open('comments_popup_file.txt'.format(),'a') as f:
                    f.write(v)
            elif k == 'iframe':
                with open('iframe_file.txt'.format(),'a') as f:
                    f.write(v)
            elif k == 'width':
                with open('width_file.txt'.format(),'a') as f:
                    f.write(v)
            elif k == 'height':
                with open('height_file.txt'.format(),'a') as f:
                    f.write(v)
                print "--------------------"
    infile.close()
    f.close()
parse_file()
\$\endgroup\$
3
\$\begingroup\$

Before We Get Started

It would be good if you could provide some examples of the URLs that you are looking to parse, I did a cursory search and I couldn't coerce a query result from any that I tried with:

urlparse.parse_qsl(urlparse.urlparse("some_url").query

In the interest of brevity I'll assume you have some set of URLs that this is working for.

Code Review

Step one in making your code more generic will be to remove the hard-coded URL text file from your function body. I would suggest it be a parameter to the parse_file function.

Also, You don't need to do this:

lines = infile.readlines()
for line in lines:

.readlines() isn't necessary in how it is being used here and the file can be read without it. You obviously know about the With Statement and how to use it as a context manager - but you're still opening and closing your URL list file manually!

infile = open("URLlist.txt", 'r')
...
infile.close()

Instead, you can alias the opened file in your With Statement and it will be iterable without .readlines():

with open(input_file) as input:
    for line in input:

Also, if you find yourself using multiple If/Elif clauses you might consider it time to look into some of Python's more interesting data structures, below I'm using a dictionary. You can replace your predicate:

if k == 'blog':

and instead use a key in a dictionary with the value corresponding to the the subsequent procedure.

{'blog' : "blog_file.txt"}

Revision

Below is a possible revision to your code:

import urlparse
def parse_file(input_file):
    tags = {'blog' : "blog_file.txt",
            'p' : "p_file.txt",
            'attachment_id' : "attachment_id_file.txt",
            'lang' : "lang_file.txt",
            'portfolio' : "portfolio_file.txt",
            'page_id' : "page_id_file.txt",
            'comments_popup' : "comments_popup_file.txt",
            'iframe' : "iframe_file.txt",
            'width' : "width_file.txt",
            'height' : "height_file.txt"}
    with open(input_file) as input:
        for line in input:
            parsed_url = urlparse.parse_qsl(urlparse.urlparse(line).query)
            if len(parsed_url) > 0:
                for key, value in parsed_url:
                    if key in tags:
                        with open(tags[key], 'a') as output_file:
                            output_file.write(value)
                    else:
                        print key + " not in tags."
            else:
                print(line + " does not yield query.")

I think this should work as you describe, but without a set of example URLs it's difficult to say for sure. The line:

if len(parsed_url) > 0:

is used here because I couldn't coerce .query to return anything but an empty list on those URLs I attempted.

Final Thought

It might be a good idea to change the functionality for those keys not found in your tags dictionary. You might consider creating a miscellaneous, or catch-all file that unmatched values get written to. That way you don't totally lose work on those unmatched URLs.

This is all based on the assumption that you have some domain that your code is currently working for.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.