2
\$\begingroup\$

I want to know if this is a good idea. It is definitely only a proof of concept at this point, but if it's a good idea I would pursue the development into a more mature product.

There are three main parts of this concept (maybe they should be separate questions, but I am asking about the concept as a whole and whether it conforms to acceptable standards or is just a bad idea):

  1. Create a single file Python interpreter (sort of) which will be "compiled" with pyinstaller for various platforms and included with the distribution of the app. This allows a completely pluggable system of command line utilities written in Python.

  2. Create a library which provides a decorator which creates a command line interface dynamically based on the function signature.

  3. Provide a production-ready server based on Bottle and CherryPy which serves a Web GUI based on a very simple plugin system.

To this end I created a project on GitHub, and I would recommend looking at the structure and source code there, but I am including the most relevant pieces of code here as per the recommendations of the moderators.

This is the code in magic.py which executes the python scripts. Note that the main point of this is to "compile" this code with pyinstaller so there is a one-step build process and it provides a pluggable system of command line utilities (also note the ellipses):

# I want to make __future__ available, but it must be at the beginning.
import __future__
from config import get_config
import logging
import sys
import os

if False:
    # The following modules (even though they are not actually imported)
    # are meant to be available. When this module is packaged with pyinstaller
    # It will make these modules available for import or in other words they
    # will be bundled in the executable.
    import re
    import xml
    ...
    import cherrypy

config = get_config("logging.conf")
log_level = config.getint('logging', "log_level")

logger = logging.getLogger('magic')

logger.setLevel(log_level)
formatter = logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(log_level)
handler.setFormatter(formatter)
logger.addHandler(handler)

# Only one parameter is needed for functionality, but I would like
# to add support for flags to simulate the python interpreter flags.
sys.argv = sys.argv[1:]
if not sys.argv:
    sys.exit(-1)

_file = sys.argv[0]
if not _file.endswith('.py'):
    # append .py to _file if not present
    _file = '{}.py'.format(_file)

ran = False

config = get_config("magic.conf")
dirs = config.get("magic", "path").split()

logger.debug('Executing command "{}"'.format(' '.join(sys.argv)))

for _dir in dirs:
    filename = os.path.join(_dir, _file)
    if not os.path.exists(filename) or not os.path.isfile(filename):
        continue
    try:
        execfile(filename)
        ran = True
    except Exception, e:
        msg = "Failed to execute {}. Reason: {}".format(' '.join(sys.argv), e)
        if hasattr(e, 'read'):
            msg = '{}\n\t{}'.format(msg, e.read())
        logger.error(msg)
        # Here it ran, but raised an exception
        raise
    break
if not ran:
    logger.debug(
        "Failed to execute file: {0}. "
        "{0} does not exist or is not a file".format(_file))

Now, for the dynamic creation of the command line interface. I use inspect to get at the function signature and argparse to implement the CLI.

cli.py

import sys
import inspect
import argparse


class Cli(object):

    def __init__(self, description=""):
        self.parser = argparse.ArgumentParser(description=description,
            formatter_class=argparse.RawDescriptionHelpFormatter)
        self.subparsers = self.parser.add_subparsers()
        self.functions = {}


    def command(self):
        def inner(fn):
            """collects information about decorated function, builds a
            subparser then returns the function unchanged"""
            name = fn.__name__
            self.functions[name] = fn

            desc = fn.__doc__
            args, _, __, defaults = inspect.getargspec(fn)

            if not args:
                args = []
                defaults = []

            if len(args) != len(defaults):
                print "All cli.command function arguments must have a default."
                sys.exit(-1)

            _parser = self.subparsers.add_parser(name, description=desc,
                formatter_class=argparse.RawDescriptionHelpFormatter)
            _parser.set_defaults(func=self.functions[name])

            for arg, default in zip(args, defaults):
                # Try the lower case first letter for the short option first
                if '-{}'.format(arg[0]) not in _parser._option_string_actions:
                    flag = ('-{}'.format(arg[0]), '--{}'.format(arg))
                # Then the upper-case first letter for the short option
                elif '-{}'.format(arg[0]).upper() not in _parser._option_string_actions:
                    flag = ('-{}'.format(arg[0]).upper(), '--{}'.format(arg))
                # otherwise no short option
                else:
                    flag = ('--{}'.format(arg))
                if isinstance(default, basestring):
                    _parser.add_argument(*flag, type=str, default=default)
                elif isinstance(default, list):
                    _parser.add_argument(*flag, nargs='+')
                elif isinstance(default, bool):
                    if default:
                        _parser.add_argument(
                            *flag, action='store_false', default=default)
                    else:
                        _parser.add_argument(
                            *flag, action='store_true', default=default)
                elif isinstance(default, int):
                    _parser.add_argument(*flag, type=int, default=default)
            return fn
        return inner

    def run(self):
        """Executes the function corresponding to the command line
        arguments provided by the user"""
        args = self.parser.parse_args()
        func = args.func
        _args, _, __, defaults = inspect.getargspec(func)
        kwargs = {}
        for arg in _args:
            kwargs[arg] = getattr(args, arg)
        func(**kwargs)

Now for the web GUI. Here is the script web.py which dynamically loads anything in our plugin directory as a plugin:

import os
import var.lib.bottle as bottle


def _get_plugins(app):
    """This function builds a list"""
    ret = {}
    dirs = [d for d in os.walk(app.PLUGIN_DIR).next()[1]]
    dirs.sort()
    for d in dirs:
        # import the main function into a temporary variable
        _main = __import__(
            'var.www.plugins.{}.plugin'.format(d),
            globals(),
            locals(),
            ['main'])

        # add plugin directory to TEMPLATE_DIR so they can house their
        # own templates (this allows plugins to be self-contained)
        bottle.TEMPLATE_PATH.append(os.path.join(app.PLUGIN_DIR, d))

        # Route GET and POST requests to the main() method of plugin.py
        app.route(
            '/{}'.format(d),
            ['GET', 'POST', 'DELETE', 'PUT', 'PATCH'],
            _main.main)

        ret[d] = bottle.template('link', name=d)

        # TODO: inspect function to allow for dynamic routing and
        #       service discovery
    return ret

app = bottle.Bottle()
bottle.TEMPLATE_PATH = ['var/www/templates/']
app.PLUGIN_DIR = 'var/www/plugins/'
app.STATIC_DIR = 'var/www/static/'


@app.route('/')
@app.route('/index')
@app.route('/index.html')
@bottle.view('index')
def index():
    """Returns the index template with a list of templates(actually a
    list of links to the plugins URIs)."""
    return {'plugins': app.PLUGINS}


@app.route('/static/<filepath:path>')
def static(filepath):
    """Serves static files located in app.STATIC_DIR."""
    return bottle.static_file(filepath, root=app.STATIC_DIR)

if __name__ == '__main__':
    app.PLUGINS = _get_plugins(app)
    app.run(server='cherrypy')

Is it a good idea to structure apps this way to provide a cross-platform application with as little boiler-plate code as possible?

\$\endgroup\$
4
  • \$\begingroup\$ See how to handle large projects \$\endgroup\$
    – rolfl
    Dec 4, 2014 at 23:05
  • 1
    \$\begingroup\$ A similair project : github.com/chriskiehl/Gooey \$\endgroup\$ Dec 5, 2014 at 16:19
  • \$\begingroup\$ That project looks really cool, and is very similar to the web gui portion of my project, but I also have some similarity to github.com/tellapart/commandr. My project aims to do both of these things (generate a CLI based on function signature and generate a web gui easily [I will be looking into pulling in the CLI into the web gui via a special plugin]) as well as provide the single file python interpreter. Am I trying to do too much? \$\endgroup\$
    – iLoveTux
    Dec 5, 2014 at 17:08
  • \$\begingroup\$ @Jamal Thank you for the edits. I should really pay more attention. \$\endgroup\$
    – iLoveTux
    Dec 5, 2014 at 18:36

1 Answer 1

1
\$\begingroup\$

Now I don't take answering my own question lightly, and perhaps someone is out there writing up the perfect answer right now, but I doubt it because my question my question is very broad and as I've discovered there is a lot of research to be done on the subject. I hope I can do the subject some justice as there are a lot of very talented people implementing seperately what I am implementing myself.

The subject of creating User Interfaces has a long and sorted history perhaps the most famous incident of which is the well known clash between Steve Jobs and Bill Gates. In fact there was a movie (I had a link to wikipedia's entry on Pirates of Silicon Valley here, but I can only add two links) made about that one (There are a lot of criticisms of this film and it's accuracy, but I reference only the fact that it was such a major event that there was a movie made about it). The fact is that how a user interacts with the programs that we as developers create is a very important topic.

Now good design is very important, and there are many factors in what determines good design. I will not list them here, but if you're interested in doing further research Here is a good place to start (pay particular attention to the references section).

With that being said, it would be hard to design a framework which could account for all of the design principles that a User Interface Engineer would be aware of and would care about. There is one type of interface where good design doesn't have as many variables and that is on the command line. There are some standards which dictate how a program should behave as stated in the answers to this question

Now, getting something for free (I think) is something which everyone loves, but which can lead us into a trap like in the old Bait and Switch (I had a link to wikipedia's entry on bait and switch here, but I can only add two links). but there are times when technology advances to a point where a once hard to achieve end is made significantly easier thus allowing us to get something without having to put any extra effort into it ourselves, and we essentially get it for free think HTTP(although the real price is actually the years of trial and error that those who have gone before us have put into it), but that is the beauty of open source, we can give so that others can profit along with ourselves.

OK, enough rhetoric, The answer to my question as I see it is that when a standard has been developed and matured to a point when most people agree on it then a framework makes sense, but when something hasn't been ironed out as much then we need to make attempts at defining it more and more so that we can all benefit.

so for my project, I am going to try and capture the standards surrounding the command line while making it incredibly easy to create a web interface around python programs. One of the most important things I will focus on is the deployment of web applications, because that is one of the hardest parts of creating a Python web application.

Please feel free to edit this answer if you feel you have a better idea or more sources.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.