Fetching daylerees colour schemes for PHPStorm from Github

Question

I wanted to download all colour schemes for phpstorm and was too lazy to do it manually so I wrote a little script for it in python, because why not.

This is my very first Python script and I'm still learning through tutorials and what not.

Things it does:

1. Asks to input the path where to save the files
2. If the destination folder does not exist, creates it
3. Makes sure the the path ends with '/'
4. Instantiates a crawler instance
5. Crawler fetches the initial JSON payload
6. It loops over the items to check whether an item is a file or a directory
7. If it is a file, downloads it. If a folder, instantiates a new crawler instance with that directory path
8. Before downloading the files, it checks if the file exists already and prints the messages accordingly.

Things I can already see that I will go back and improve:

1. Make passing the link dynamic, instead of hard coding.
2. Implement a better path checking / validation.
3. use try / except methods instead where applicable.

And the code:

import urllib
import json
import time
import os
import re


class Crawler:
    def __init__(self, url, path):
        self.url = url
        self.path = path
        self.counter = 0
        self.total_files = 0

    def crawl(self):
        self.get_contents()

    def get_contents(self):
        response = urllib.urlopen(self.url).read()
        collection = json.loads(response)
        self.total_files = len(collection)
        for data in collection:
            if data.get('type') == 'file':
                self.download(data.get('name'), data.get('download_url'))
            elif data.get('type') == 'dir':
                dir_crawler = Crawler(data.get('url'), self.path)
                print('\nFound a directory. Pausing the main download and crawling the directory - ' + data.get(
                    'name') + '\n')
                dir_crawler.crawl()
                print('\n' + data.get('name') + ' directory is done. Continuing from where I left off.\n')
            time.sleep(1)

    def download(self, name, link):
        if name and not os.path.exists(self.path + name):
            download = urllib.URLopener()
            download.retrieve(link, self.path + name)
            self.counter += 1
            print('Downloading ' + name + ' (' + str(self.counter) + ' out of ' + str(self.total_files) + ')')
        else:
            print('File: ' + name + ' already exists in ' + self.path)


download_location = raw_input(
    "Where would you like me to save the files? (e.g. ./ or ../ or ./themes/ or /path/to/folder): ")
if download_location and re.match(r'.+/$', download_location):
    if not os.path.exists(download_location):
        os.mkdir(download_location)
    crawler = Crawler('https://api.github.com/repos/daylerees/colour-schemes/contents/jetbrains',
                      download_location)
    crawler.crawl()
    print('\n\nAll done. Themes have been saved into ' + download_location +
          '. Thank you for your cooperation. Have a nice day.')
else:
    print('The path must end with a forward slash.')

Answer 1

Instead of os.path.exists you may want to specifically use os.path.isdir and isfile. isdir checks if the path matches an existing folder, and isfile checks for files. It is possible for the path you pass to be valid but not of the correct type for your purposes, which would be a significant error for your folders as you then have invalid paths that you'd attempt to download to.

Also to make it easier on your user, I would wrap the main code in a while loop so that if they enter an invalid path the program doesn't exist the program still doesn't exit:

while True:
    download_location = raw_input(
        "Where would you like me to save the files? (e.g. ./ or ../ or ./themes/ or /path/to/folder): ")
    if download_location and re.match(r'.+/$', download_location):
        if not os.path.isdir(download_location):
            os.mkdir(download_location)
        crawler = Crawler('https://api.github.com/repos/daylerees/colour-schemes/contents/jetbrains',
                          download_location)
        crawler.crawl()
        print('\n\nAll done. Themes have been saved into ' + download_location +
              '. Thank you for your cooperation. Have a nice day.')
        break
    else:
        print('The path must end with a forward slash.') 

Putting break after your print call will end the infinite loop once the code has successfully run but otherwise it will keep asking for input.

Also no need for regex. Python has a str.endswith function that detects if a string ends with a certain string. So you could just use:

    if download_location.endswith('/'):

raw_input will at least return a string, so you can just call this function on download_location. If it's an empty string it will return False since an empty string doesn't end with '/'.