3
\$\begingroup\$

I am using Python threads to download images. I have a JSON file that contains URLs to the images in the following structure:

images = {'images: [{'url': 'https://contestimg.wish.com/api/webimage/5468f1c0d96b290ff8e5c805-large',
   'imageId': '1'},....]}

There are over 1,000,000 images. After using 20 threads, I only collected ~80,000 of these images, along with ~1000 exceptions due to 500 status codes from the URL. I have a feeling that my code is incorrect. Can someone please check my code?

import requests
import threading
import json
import pickle

train_data = json.load(open("train.json", "rb"))

images = train_data["images"]

threads = []

def save_image(images_part):
    errors = []
    for i in images_part:
        Picture_request = requests.get(i['url'])
        if Picture_request.status_code == 200:
            with open(f"train/{i['imageId']}.jpeg", "wb") as f:
                f.write(Picture_request.content)
        else:
            errors.append((i, Picture_request.status_code))
            print(f"error in {i['imageId']} with {Picture_request.status_code}")
    return errors


for i in range(0, 20):
    start = i*50727
    end = (i+1)*50727
    if i == 19:
        end = None
    t = threading.Thread(target=save_image, args=(images[start:end],))
    threads.append(t)
    t.start()
    print(f"initiating {i}th thread")
\$\endgroup\$
  • 2
    \$\begingroup\$ CodeReview SE is for improving style and performance of working code. If your code is broken, it's probably better to ask on Stackoverflow. However, is there any particular reason why you're using python for this task. You can achieve parallelized URL downloads with something like cat urls.txt | parallel wget assuming urls.txt is a file with each URL on a separate line. \$\endgroup\$ – Bailey Parker Dec 17 '18 at 20:26
  • \$\begingroup\$ Exactly. This is my first practical usage of threads and I was wondering if anyone could see places of potential improvement. \$\endgroup\$ – Elizabeth Keleshian Dec 17 '18 at 23:18
  • \$\begingroup\$ What do the magical numbers 20 and 50727 mean? \$\endgroup\$ – Graipher Dec 18 '18 at 9:25
  • 1
    \$\begingroup\$ Have you checked what happens if you manually visit one of the urls which threw a 500 error? \$\endgroup\$ – Graipher Dec 18 '18 at 9:47
  • \$\begingroup\$ @Graipher the 20 and 50727 numbers represent how I want to partition the data. There are 1,014,544 images, and I want to have 20 threads running. Therefore, each thread will handle 1, 014,544/20 images which is approximately 50,727. Note that on the last iteration in the for loop when i =19, the last thread will take remainders. When I put the url in the browser, the image doesn't show up and the web inspector throws a message saying - "Fail to load resource: the server responded with a status of 500 ()" \$\endgroup\$ – Elizabeth Keleshian Dec 18 '18 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.