I am using Python threads to download images. I have a JSON file that contains URLs to the images in the following structure:

images = {'images: [{'url': 'https://contestimg.wish.com/api/webimage/5468f1c0d96b290ff8e5c805-large',
   'imageId': '1'},....]}

There are over 1,000,000 images. After using 20 threads, I only collected ~80,000 of these images, along with ~1000 exceptions due to 500 status codes from the URL. I have a feeling that my code is incorrect. Can someone please check my code?

import requests
import threading
import json
import pickle

train_data = json.load(open("train.json", "rb"))

images = train_data["images"]

threads = []

def save_image(images_part):
    errors = []
    for i in images_part:
        Picture_request = requests.get(i['url'])
        if Picture_request.status_code == 200:
            with open(f"train/{i['imageId']}.jpeg", "wb") as f:
            errors.append((i, Picture_request.status_code))
            print(f"error in {i['imageId']} with {Picture_request.status_code}")
    return errors

for i in range(0, 20):
    start = i*50727
    end = (i+1)*50727
    if i == 19:
        end = None
    t = threading.Thread(target=save_image, args=(images[start:end],))
    print(f"initiating {i}th thread")
  • 2
    \$\begingroup\$ CodeReview SE is for improving style and performance of working code. If your code is broken, it's probably better to ask on Stackoverflow. However, is there any particular reason why you're using python for this task. You can achieve parallelized URL downloads with something like cat urls.txt | parallel wget assuming urls.txt is a file with each URL on a separate line. \$\endgroup\$ Dec 17, 2018 at 20:26
  • \$\begingroup\$ Exactly. This is my first practical usage of threads and I was wondering if anyone could see places of potential improvement. \$\endgroup\$
    – liz
    Dec 17, 2018 at 23:18
  • \$\begingroup\$ What do the magical numbers 20 and 50727 mean? \$\endgroup\$
    – Graipher
    Dec 18, 2018 at 9:25
  • 1
    \$\begingroup\$ Have you checked what happens if you manually visit one of the urls which threw a 500 error? \$\endgroup\$
    – Graipher
    Dec 18, 2018 at 9:47
  • \$\begingroup\$ @Graipher the 20 and 50727 numbers represent how I want to partition the data. There are 1,014,544 images, and I want to have 20 threads running. Therefore, each thread will handle 1, 014,544/20 images which is approximately 50,727. Note that on the last iteration in the for loop when i =19, the last thread will take remainders. When I put the url in the browser, the image doesn't show up and the web inspector throws a message saying - "Fail to load resource: the server responded with a status of 500 ()" \$\endgroup\$
    – liz
    Dec 18, 2018 at 18:58


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.