6
\$\begingroup\$

Sometimes I download a Python script (from a trusted source), and can't run it because of missing dependencies, which I have to install one by one (no requirements.txt is provided). I'm looking for a way to automate this process. It's similar to this question, but I'd like to infer the requirements automatically. I.e. given the script, install all the packages it's importing.

If myscript.py looks like:

import pandas
import tensorflow as tf
from keras.models import Sequential

Then after running the script (e.g. bash extract_requirements.sh myscript.py) pandas, tensorflow, and keras will be installed.

Of course, a script can call another script, which has its own dependencies, package names don't always match to import names etc. So it's not a perfect solution, but better than nothing.

What I'm using now is a command-line script:

#!/env bash

set -u

usage() {
    echo "$0" '[-d] <MY_SCRIPT>'
    cat <<-EOF
        options:
        -d    Dry-run. Print package names without installing.
EOF
    exit 1;
}

(( $# == 0 )) && usage
dry=0
if [[ "$1" == '-d' ]]; then
    dry=1
    shift
fi
(( $# != 1 )) && usage

script="$1"
imports1=$(egrep '^\s*from\s+[a-zA-Z_.-]+\simport' "$script" | cut -f2 -d' ' | cut -f1 -d'.')
imports2=$(egrep '^\s*import\s+[a-zA-Z_-]+' "$script" | cut -f2 -d' ')
all=$(printf "$imports1\n$imports2" | sort -u)
if (( dry )); then
    echo "$all" | xargs -L 1 echo
else
    echo "$all" | xargs -L 1 sudo pip install
fi

set +u

I was wondering if I'm missing some import cases, and more importantly, whether there's a simpler, more python-ic, solution.

\$\endgroup\$
  • \$\begingroup\$ 'Pythonic' applies more to pieces of Python code than to the language. Are you suggesting you might rewrite this in Python, or are you looking for tools provided by the Python Software Foundation that do this more cleanly? \$\endgroup\$ – Daniel Apr 15 '18 at 16:36
  • \$\begingroup\$ @Coal_ Yes, I was hoping there's a standard way to do it in python, rather than parsing python code in bash. \$\endgroup\$ – dimid Apr 15 '18 at 16:40
4
\$\begingroup\$

!!! Use this at your own risk. Fiddling with user's packages is scary business !!!

You can monkey-patch the builtin import statement and upon encountering ModuleNotFoundError install whatever dependency is missing. Here is a small example for a script.py. You can of course extend this to feature multiple files or dynamically use the file given as input.

import builtins
import pip
old_import = __import__

def my_import(name, globals=None, locals=None, fromlist=(), level=0):
    if globals["__name__"] == "script":
        try:
            return_args = old_import(name, globals=globals, locals=locals, 
                                    fromlist=fromlist, level=level)
        except ModuleNotFoundError:
            builtins.__import__, temp = old_import, builtins.__import__
            pipcode = pip.main(['install', name])
            if pipcode != 0:
                builtins.__import__ = temp
                raise
            else:
                return_args = old_import(name, globals=globals, locals=locals, 
                                    fromlist=fromlist, level=level)
                builtins.__import__ = temp
    else:
        return_args = old_import(name, globals=globals, locals=locals, 
                                fromlist=fromlist, level=level)

    return return_args

builtins.__import__ = my_import

import script

and the script.py is your custom script:

import os
from time import time

import argh  # this should be installed

print(time())

Note: People sometimes do things like:

try:
    import optional_module
except ImportError:
    pass  # optional all cool

Above code patches import itself. If you remove the if that limits install powers to script you allow recursive installation for everything that is imported. Ever. Even if it is an optional dependency for a 4th level dependency test case of script pip will try to install it.

\$\endgroup\$
  • \$\begingroup\$ Thanks, a nice pythonic solution, but I rather not to hardcode the name of the script. \$\endgroup\$ – dimid Apr 15 '18 at 20:56
  • \$\begingroup\$ @dimid This is just a small example to demonstrate the idea. You can extend this, e.g. globbing a folder for filenames which are okay to install or passing in the name of the script as input argument. \$\endgroup\$ – FirefoxMetzger Apr 16 '18 at 4:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.