Currently I will be downloading around ~100K images. This process may run on a weekly basis, I'm considering the following factors:

  • HTTP Requests (Will be taken care by external LB, so same source IP won't be used)
  • Multi-threading (concurrent.futures.ThreadPoolExecutor library to handle multi-threading)
  • Writing requests (Regular open and close methods)

I'm using urllib2 to handle HTTP requests, I would like to see if this properly designed to run at scale.

Any additional comments? Suggestions?

import concurrent.futures
import urllib2
import pandas as pd

FILENAME = 'files.csv'

class Image():
    def __init__(self, master_id, url):
        self.master_id = master_id
        self.url = url

def get_file(filename):
    """Get dataframe information"""
    data = pd.read_csv(filename)
    data = data.drop_duplicates(subset=['id'], keep='first')
    subset = data.head()
    return subset

# Data extraction
def extract_data(data):
    """Extract data"""
    image_list = []
    for _, url in data.iterrows():
        print url[0], url[1]
        image_list.append(Image(url[0], url[1]))
    return image_list

# Retrieve a single page and report the url and contents
def load_url(image, timeout):
    """Load URL"""
    response = urllib2.urlopen(image.url, timeout=timeout)
    return response.read()

# Save image
def save_image(image_data, filename):
    """Save Image."""
    with open(str(filename) + '.' + _IMG_EXTENSION, 'wb') as output:

def download_data(image_list):
    # We can use a with statement to ensure threads are cleaned up promptly
    with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        # Start the load operations and mark each future with its URL
        future_to_url = {executor.submit(load_url, image, EXECUTOR_TIMEOUT): image for image in image_list}
        for future in concurrent.futures.as_completed(future_to_url):
            image = future_to_url[future]
                image_data = future.result()
            except Exception as exc:
                print('%r Generated an exception: %s' % (image.url, exc))
                print('%r Page is %d bytes' % (image.url, len(image_data)))
                save_image(image_data, image.master_id)

1 Answer 1


Some minor improvements:

  • I would switch to a NamedTuple instead of a separate class:

    from collections import namedtuple
    Image = namedtuple('Image', ('master_id', 'url'))
  • Or, if you are going to stay with classes, consider adding __slots__ - may help performance-wise - though this part is not likely to be your bottleneck, but rather can be a quick and easy win

  • you can also improve your extract_data function:

    def extract_data(data):
        """Extract data"""
        return [Image(*url) for _, url in data.iterrows()] 
  • I would also try asyncio from Python 3.x. Here is also a relevant snippet.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.