Download and write ~100k images

Question

Currently I will be downloading around ~100K images. This process may run on a weekly basis, I'm considering the following factors:

• HTTP Requests (Will be taken care by external LB, so same source IP won't be used)
• Multi-threading (concurrent.futures.ThreadPoolExecutor library to handle multi-threading)
• Writing requests (Regular open and close methods)

I'm using urllib2 to handle HTTP requests, I would like to see if this properly designed to run at scale.

Any additional comments? Suggestions?

import concurrent.futures
import urllib2
import pandas as pd

MAX_WORKERS = 5
EXECUTOR_TIMEOUT = 60
FILENAME = 'files.csv'
_IMG_EXTENSION = 'jpg'

class Image():
    def __init__(self, master_id, url):
        self.master_id = master_id
        self.url = url


def get_file(filename):
    """Get dataframe information"""
    data = pd.read_csv(filename)
    data = data.drop_duplicates(subset=['id'], keep='first')
    subset = data.head()
    return subset


# Data extraction
def extract_data(data):
    """Extract data"""
    image_list = []
    for _, url in data.iterrows():
        print url[0], url[1]
        image_list.append(Image(url[0], url[1]))
    return image_list


# Retrieve a single page and report the url and contents
def load_url(image, timeout):
    """Load URL"""
    response = urllib2.urlopen(image.url, timeout=timeout)
    return response.read()


# Save image
def save_image(image_data, filename):
    """Save Image."""
    with open(str(filename) + '.' + _IMG_EXTENSION, 'wb') as output:
        output.write(image_data)
        output.close()


def download_data(image_list):
    # We can use a with statement to ensure threads are cleaned up promptly
    with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
        # Start the load operations and mark each future with its URL
        future_to_url = {executor.submit(load_url, image, EXECUTOR_TIMEOUT): image for image in image_list}
        for future in concurrent.futures.as_completed(future_to_url):
            image = future_to_url[future]
            try:
                image_data = future.result()
            except Exception as exc:
                print('%r Generated an exception: %s' % (image.url, exc))
            else:
                print('%r Page is %d bytes' % (image.url, len(image_data)))
                save_image(image_data, image.master_id)

Answer 1

Some minor improvements:

• I would switch to a NamedTuple instead of a separate class:

  from collections import namedtuple
  
  Image = namedtuple('Image', ('master_id', 'url'))
  

• Or, if you are going to stay with classes, consider adding __slots__ - may help performance-wise - though this part is not likely to be your bottleneck, but rather can be a quick and easy win

• you can also improve your extract_data function:

  def extract_data(data):
      """Extract data"""
      return [Image(*url) for _, url in data.iterrows()] 
  

• I would also try asyncio from Python 3.x. Here is also a relevant snippet.