We are using images from Wikimedia commons for some of Cardshifter's game artwork. I selected some art for each card to make a .jpg file to use in the game clients, and added a URL to the original file as a comment for reference. This is what one card looks like: (with our DSL made from Groovy)

// https://en.wikipedia.org/wiki/Sun_Wukong
// https://commons.wikimedia.org/wiki/File:Sun_Wukong_and_Jade_Rabbit.jpg
card('MONKING') {
    creature 'Chinese Hero'
    flavor 'Monkey King Warrior of Immense Strength.'
    imagePath 'mythos/chinese/monking.jpg'
    maxInDeck 2
    health 5
    sickness 0
    manaCost 15
    attack 5
    denyCounterAttack()
    onEndOfTurn {
        change ATTACK by 1 on {
            thisCard()
        }
    }
}

However, the problem I was facing is that while I was gathering images, I forgot to also get the attribution information for each image. So, I wrote a Python script and a few helper classes to do this work.

The script uses Selenium webdriver to control Firefox (as suggested by @jacwah), along with some read & write instructions and a bit of regex to isolate URLs to be used by Selenium to automate getting the information from Wikimedia.

In the resulting output files, the above card's header will look like this with the added attribution text (or an error if Selenium is unable to resolve it, by design)

// https://en.wikipedia.org/wiki/Sun_Wukong
// https://commons.wikimedia.org/wiki/File:Sun_Wukong_and_Jade_Rabbit.jpg
// Attribution: By Yoshitoshi Tsukioka (http://www.japaneseprints.net/viewitem.cfm?ID=2182) [Public domain], via Wikimedia Commons// License: Public Domain
card('MONKING') {
    creature 'Chinese Hero'
    flavor 'Monkey King Warrior of Immense Strength.'
    imagePath 'mythos/chinese/monking.jpg'
...

This is my first rodeo with Python, please help me improve any aspect of the code that you feel is not "Pythonic", not efficient, or just plain terrible.

Note: The script takes about 5 minutes to run, keeping in mind that the vast majority of the time is spent by Selenium opening URLs in Firefox, clicking links, etc. for probably about 80-90 cards.

###RegexUtilityPatterns.py ^{(to keep useful patterns in one place)}

import re

URL_MATCH = re.compile(r"""
    http[s]?://     # Match protocol, secure or not
   (?:
        [a-zA-Z]|       # Any letter
        [0-9]|          # Any number
        [$-_@.&+]|      # Other allowed symbols
        [!*\(\),]|      # Exclamation, parens, commas
        (?:
            %[0-9a-fA-F][0-9a-fA-F]  # percent (%) sign followed by 2-digit hex
        )
   )
   +  # Match one or more of preceding tokens
   """, re.VERBOSE)

###ListOfFilesInDirectory.py

from Debug import Debug
import os

class ListOfFilesInDirectory(object):

    @Debug()
    def __init__(self, directory, extension=".*"):
        """
        Get a list of file names from a directory.
            :param directory: Where to look for files
            :param extension: The desired file extension (optional)
            :return: list of file names as strings
            :rtype: object
        """
        self.directory = directory
        self.extension = extension

    def list_of_files(self):
        """
        Scans a directory for files, matching an optional file extension,
        or all file extensions if omitted.
        :return: list of file names as strings
        """
        directory = self.directory
        extension = self.extension
        file_list = []

        if not directory.endswith("/"):
            directory += "/"
        if not extension.startswith("."):
            extension = "." + extension
        for file_name in os.listdir(directory):
            if file_name.endswith(extension):
                file_list.append(file_name)

        return file_list

Note that Debug is a useful method/function wrapper designed by @Ethan Bierlein

##WikimediaAttributionsFromFilesInDirectory.py

from ListOfFilesInDirectory import ListOfFilesInDirectory
from selenium import webdriver
import RegexUtilityPatterns
import sys
import re

source_directory = "/Users/francisveilleux-gaboury/IdeaProjects/Cardshifter/extra-resources/mods/Mythos/"
extension = ".cardset"
file_names = ListOfFilesInDirectory(source_directory, extension)
output_directory = "/Users/francisveilleux-gaboury/Downloads/"

filter_include = [
    "wikimedia"
]
filter_exclude = [
    "User:",
    "upload.wikimedia"
]
url_regex = RegexUtilityPatterns.URL_MATCH

try:
    firefox = webdriver.Firefox()
    for file_name in file_names.list_of_files():
        with open(output_directory + file_name, "w") as output:
            full_file_path = source_directory + file_name
            print "/* " + full_file_path + " */"
            output.write("// " + full_file_path)
            with open(full_file_path, "r") as source_file:
                for line in source_file:
                    url_match = re.search(url_regex, line)
                    if url_match:
                        url_list = re.findall(url_regex, line)
                        url = url_list[0]
                        if any(value in url for value in filter_include) \
                                and any(value not in url for value in filter_exclude):
                            try:
                                firefox.get(url)
                                firefox.find_element_by_css_selector("a[title=\"Use this file on the web\"]").click()
                                firefox.find_element_by_id("stockphoto_attribution_html").click()
                                attribution_text = firefox.find_element_by_id("stockphoto_attribution").get_attribute("value")
                                print(
                                    "// " + url + "\n"
                                    + "// Attribution: " + attribution_text
                                )
                                output.write(
                                    "// " + url + "\n"
                                    + "// Attribution: " + attribution_text
                                )

                            except:
                                e = sys.exc_info()[0]
                                print(
                                    "// " + url + "\n"
                                    + "// Exception: " + str(e) + "\n"
                                )
                                output.write(
                                    "// " + url + "\n"
                                    + "// Exception: " + str(e) + "\n"
                                )
                        else:
                            print("// " + url)
                            output.write("// " + url + "\n")
                    else:
                        print(line),
                        output.write(line),
finally:
    firefox.quit()