Take the 2-minute tour ×
Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.

I'm a little frustrated with the state of url parsing in python, although I sympathize with the challenges. Today I just needed a tool to join path parts and normalize slashes without accidentally losing other parts of the URL, so I wrote this:

from urlparse import urlsplit, urlunsplit

def url_path_join(*parts):
    """Join and normalize url path parts with a slash."""
    schemes, netlocs, paths, queries, fragments = zip(*(urlsplit(part) for part in parts))
    # Use the first value for everything but path. Join the path on '/'
    scheme   = next((x for x in schemes if x), '')
    netloc   = next((x for x in netlocs if x), '')
    path     = '/'.join(x.strip('/') for x in paths if x)
    query    = next((x for x in queries if x), '')
    fragment = next((x for x in fragments if x), '')
    return urlunsplit((scheme, netloc, path, query, fragment))

As you can see, it's not very DRY, but it does do what I need, which is this:

>>> url_path_join('https://example.org/fizz', 'buzz')
'https://example.org/fizz/buzz'

Another example:

>>> parts=['https://', 'http://www.example.org', '?fuzz=buzz']
>>> '/'.join([x.strip('/') for x in parts]) # Not sufficient
'https:/http://www.example.org/?fuzz=buzz'
>>> url_path_join(*parts)
'https://www.example.org?fuzz=buzz'

Can you recommend an approach that is readable without being even more repetitive and verbose?

share|improve this question
    
Why can't you just use os.path.join for taking care of the joining and such? –  Blender Jun 25 '12 at 5:43
    
@Blender what if someone runs this code on Windows? –  kojiro Jun 25 '12 at 11:31
    
Could you please give more examples of the kind of input that would require your above code to accomplish? (i.e provide some test cases that should pass for the solution to be acceptable?). What about some thing simple such as return '/'.join([x.strip('/') for x in parts]) –  rahul Jun 27 '12 at 19:32
    
Sure, how about parts=['https://', 'http://www.example.org', '?fuzz=buzz']? –  kojiro Jun 29 '12 at 12:03

2 Answers 2

up vote 3 down vote accepted

I'd suggest the following improvements (in descending order of importance):

  1. Extract your redundant generator expression to a function so it only occurs once. To preserve flexibility, introduce default as an optional parameter
  2. This makes the comment redundant because first is a self-documenting name (you could call it first_or_default if you want to be more explicit), so you can remove that
  3. Rephrase your docstring to make it more readable: normalize and with a slash don't make sense together
  4. PEP 8 suggests not to align variable assignments, so does Clean Code by Robert C. Martin. However, it's more important to be consistent within your project.

def url_path_join(*parts):
    """Normalize url parts and join them with a slash."""
    schemes, netlocs, paths, queries, fragments = zip(*(urlsplit(part) for part in parts))
    scheme = first(schemes)
    netloc = first(netlocs)
    path = '/'.join(x.strip('/') for x in paths if x)
    query = first(queries)
    fragment = first(fragments)
    return urlunsplit((scheme, netloc, path, query, fragment))

def first(sequence, default=''):
    return next((x for x in sequence if x), default)

If you're looking for something a bit more radical in nature, why not let first handle several sequences at once? (Note that unfortunately, you cannot combine default parameters with sequence-unpacking in Python 2.7, which has been fixed in Python 3.)

def url_path_join(*parts):
    """Normalize url parts and join them with a slash."""
    schemes, netlocs, paths, queries, fragments = zip(*(urlsplit(part) for part in parts))
    scheme, netloc, query, fragment = first_of_each(schemes, netlocs, queries, fragments)
    path = '/'.join(x.strip('/') for x in paths if x)
    return urlunsplit((scheme, netloc, path, query, fragment))

def first_of_each(*sequences):
    return (next((x for x in sequence if x), '') for sequence in sequences)
share|improve this answer

I fully agree with Blender. Just use the os.path module. It provides a method to join paths and it also has methods to normalize pathnames (eg. os.path.normpath(pathname)) to use it on every OS with different separators.

share|improve this answer
    
os.path.normpath does not use forward slashes on Windows by default, and it's not intelligent about keeping the double slashes after http://. –  kojiro Jul 18 '12 at 14:49
    
@kojiro Yes, but you can import posixpath instead of import os.path to get the correct slashes. I do believe you are correct about the http:// though –  Matt Oct 16 '12 at 14:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.