This is one simple crawler that downloads images from websites, the website's URL to be crawled to must match the regex, as well as any image-to-download's URL.

(Also, I know, I made my own thread pool, because I did want to keep my crawler lightweight...)


import BeautifulSoup
import requests
import re
import json
import os.path
import urlparse
import os
import ntpath
import urllib
import threading
import Queue
import time

class MultiThreadQueue(object):
    def __init__(self, max_simultaneous_threads):
        self.thread_queue = Queue.Queue()
        self.max_simultaneous_threads = max_simultaneous_threads
        self.executing_threads = 0
        self._threads_executed = []

    def add_thread(self, thread):
        return self.thread_queue.put(thread)

    def set_max_simultaneous_threads(self, max_simultaneous_threads):
        self.max_simultaneous_threads = max_simultaneous_threads

    def execute_last_thread(self):
        if self.executing_threads >= self.max_simultaneous_threads:
            return False

        thread = self.thread_queue.get_nowait()
        self.executing_threads += 1
        return True

    def execute_threads(self):
        if self.max_simultaneous_threads == 0:
            while True:

                except Queue.Empty:


        while True:
                if self.execute_last_thread():
                    thread = self._threads_executed[-1]
                    threading.Thread(name="Temporary internal TJT", target=self._join_thread, args=(thread,)).start()

                if self.executing_threads < 1:
                    return True


            except Queue.Empty:
                return True

    def _join_thread(self, thread):
        self.executing_threads -= 1

def regex(value, reg):
    if reg == "":
        return True

    return bool(re.search(reg, value))

def ends_with_any(string, list_of_endings):
    for ending in list_of_endings:
        if string.endswith(ending):
            return True

    return False

def fetch_from_url(link_url, regex_filter, crawl_filter, max_level=5, this_level=0):
    if max_level < 1 or max_level < this_level:
        print "|" + ("-" * this_level) + "Bug: Max level reached in this URL iteration and function called anyways!"
        return []

    images = []
    request = requests.get(link_url)

    if request.status_code != 200:
        print "|" + ("-" * this_level) + "Status code {} received! Leaving this crawl.".format(request.status_code)
        return []

    soup = BeautifulSoup.BeautifulSoup(request.text)
    print "|" + ("-" * this_level) + "Fetching images at {}!".format(link_url)

    for image in soup.findAll("img", {"src": True}):
        print "|" + ("-" * this_level) + "Trying image {}!".format(urlparse.urljoin(link_url, image["src"]))
        if not regex(urlparse.urljoin(link_url, image["src"]), regex_filter):
            print "|" + ("-" * this_level) + "Image's path does not match with the regex {}!".format(regex_filter)

        print "|" + ("-" * this_level) + "Appending image at {}!".format(urlparse.urljoin(link_url, image["src"]))
        images.append(urlparse.urljoin(link_url, image["src"]))

    if max_level > 0:
        for link in soup.findAll("a", {"href": True}):
            if ends_with_any(urlparse.urljoin(link_url, link["href"]).lower(),
                             ("png", "jpeg", "jpg", "tga", "gif", "bmp", "jpng")):
                print "|" + ("-" * this_level) + "Got image from sudden link {}!".format(
                    urlparse.urljoin(link_url, link["href"]))

                if not regex(urlparse.urljoin(link_url, link["href"]), regex_filter):
                    print "|" + ("-" * this_level) + "Image's path does not match with the regex {}!".format(

                images.append(urlparse.urljoin(link_url, link["href"]))


            print "|" + ("-" * this_level) + "Crawling to {}!".format(urlparse.urljoin(link_url, link["href"]))

            if regex(urlparse.urljoin(link_url, link["href"]), crawl_filter):
                images += fetch_from_url(
                    urlparse.urljoin(link_url, link["href"]),
                    max_level - 1,
                    this_level + 1


            print "|" + ("-" * this_level) + "Crawling path does not match with the regex {}!".format(crawl_filter)

    return images

def download(url, result_file, rewrite_file=False):
    print "Downloading from {} into {}!".format(url, result_file)

    if os.path.exists(result_file) and not rewrite_file:
        print "File already exists!"
        return False

    urllib.urlretrieve(url, result_file)
    print "Download from {} succesful! File available at {}.".format(url, result_file)
    return True

if __name__ != "__main__":

parsed = json.load(open("texlibs.json"))

downloads = MultiThreadQueue(parsed[1])

for image_kind_or_folder, (url, regex_filter, crawl_filter, max_level) in parsed[0].items():
    if not os.path.exists(os.path.join("../result", image_kind_or_folder)):
        os.makedirs(os.path.join("../result", image_kind_or_folder))

    print "Looking for images in specified URL!"

    for image in fetch_from_url(url, regex_filter, crawl_filter, max_level):
        result_path = "../{}/{}".format(os.path.join("result", image_kind_or_folder), ntpath.basename(image))

                name="Download \'{} -> {}\' Thread".format(image, result_path),
                args=(image, result_path)

print "Downloading all appended images!"


(also avaiable on GitHub here)

  • 3
    \$\begingroup\$ What was wrong with if __name__ == "__main__" ? Why make your script not importable? \$\endgroup\$ Jul 2, 2016 at 23:03
  • \$\begingroup\$ @MathiasEttinger Wasn't it importable already? It just won't import past the if __name__ != "__main__": exit()? \$\endgroup\$
    – wallabra
    Jul 2, 2016 at 23:33
  • 1
    \$\begingroup\$ Did you try to import <your_script> before making such claim? \$\endgroup\$ Jul 3, 2016 at 6:27
  • \$\begingroup\$ @MathiasEttinger I weren't claiming, but you're right. \$\endgroup\$
    – wallabra
    Jul 3, 2016 at 18:56

1 Answer 1


Kudos for choosing a terrific HTML parser. I recommend invoking version 4.6 in this way:

from bs4 import BeautifulSoup

Using any() could turn ends_with_any() into a (perfectly clear) one-liner.

In fetch_from_url, the default keyword args are perfect. I would love to see a docstring for the function. I'm reading a return [] a few times, which gives me a hint about its signature, but I'd much rather see the function's author put a stake in the sand. download()'s rewrite_file=False is also nicely readable.

From the print I see you're using python2. Maybe consider 3? Also, maybe this explains the use of downrev BeautifulSoup.

The expression "|" + ("-" * this_level) appears a few times, which suggests the opportunity to define a helper function. If you nest it within fetch_from_url then you won't even have to pass in this_level as an explicit argument.

Repeated copy-n-paste calls to urljoin suggest that you may want to cache results in temp variables.

The code starting with parsed = json.load ... looks like it wants be wrapped within def main(), as Mathias observed. Always strive for import to silently succeed, just in case someone wants to reuse your code later. It's too easy, may as well just do it, it's very low hanging fruit.

image_kind_or_folder is nicely descriptive, and so is this:

    downloads = MultiThreadQueue(parsed[1])

You had an opportunity to name parsed[0] there, as well, so the Gentle Reader would understand what sort of dict it is.

The set_max_simultaneous_threads setter is an anti-pattern, and is not used, recommend you delete it, looks like the ctor has that covered already. Please add a docstring to the constructor - the semantics of max_simultaneous_threads == 0 are not obvious.

execute_threads() works for you, so I'll mostly refrain from criticizing it, but _threads_executed seems a little odd, perhaps _threads_executing would be more natural (so _join_thread() would delete an entry). That way you could use len() rather than maintaining the executing_threads count.

  • \$\begingroup\$ "Kudos for using the best HTML parser ever". That's subjective. Best in what way? \$\endgroup\$ Aug 1, 2017 at 3:15
  • \$\begingroup\$ Sorry, I got a little carried away. I was just trying to validate OP's choice of parser. I find Beautiful Soup is a do-what-I-mean kind of library. It's robust to ugly HTML spec. violations, and lets your code blissfully believe that other HTML authors do sane things, like following nesting rules. I downgraded to merely "terrific". \$\endgroup\$
    – J_H
    Aug 2, 2017 at 3:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.