1
\$\begingroup\$

I've written a package manager for OS X in order to improve my Python skills. This is the second programming project I've ever worked on, so I do not expect it to be great, or anything, but I would like to get some input on the code and overall design.

I store the package info inside a SQLite database, which I've written a class to work with. The main file contains another class which is supposed to be a library, so a working program would interact with it.

This is the first time I've used a VCS, so I chose git, and I uploaded the repo to github.

And here is the code

lazy.py

#!/usr/bin/env python3
"""
This is the main class for Lazy. It provides all the basic operations the
package manager should support in the forms of class methods. If you want to
actually use Lazy, you need to write an interface for it. Lazy was written with
Python 3 in mind, but it should work with Python 2 with minimal modification.
"""
import os
import os.path
import tarfile
import hashlib
import urllib.request
import subprocess
import shutil
import lazy_db


class Lazy(lazy_db.LazyDB):

    prefix = os.path.join("/usr", "local", "lazy")
    bindir = os.path.join(prefix, "bin")
    log_path = os.path.join(prefix, "logs")
    patch_path = 'patch'
    autoconf_ext = '.autoconf'
    patch_ext = '.patch'
    automake_ext = '.make'

    def install(self, pkg_id):
        """
        This method takes a package id and installs the associated package. It
        does so by getting all the information from the database and calling
        different methods to perform the installation steps.
        """
        self.resolve_dependencies(pkg_id)

        (self.pkg_id, self.name, self.download_url, self.md5,
        self.srcdir, self.objdir, self.args, self.installed_version,
        self.latest_version) = self.get_all_package_info(pkg_id)

        self.filename = self.download_package()
        self.extract_package()
        self.autoconf()
        self.automake()
        self.symlink()
        self.cleanup()

    def uninstall(self, pkg_id):
        """
        This method takes a package id and uninstalls the associated package.
        """
        pkg_name = self.get_package_name(pkg_id)
        install_path = os.path.join(self.prefix, pkg_name)
        bin_list = os.listdir(os.path.join(install_path, 'bin'))

        self.unlink_binaries(bin_list)
        self.remove_package_dir(install_path)
        self.update_installed_version(0, pkg_id)

    def search(self, pkg_name=None):
        """
        This method takes a package name and returns a list of all related
        packages. If no name is given as argument, it returns all the packages.
        """
        if not pkg_name:
            packages = self.get_all_packages()
        else:
            packages = self.get_packages_by_name(pkg_name)

        return packages

    def resolve_dependencies(self, pkg_id):
        """
        This method takes a package id and installs all the dependencies of
        that package. There's an indirect recursion between this method and the
        `install` method.
        """
        dependencies = self.get_dependencies(pkg_id)

        for dep_id in dependencies:
            if not self.is_installed(dep_id):
                self.install(dep_id)

    def download_package(self):
        """
        This method downloads the archive for the package to be installed. It
        also performs a sanity check by comparing the md5 hash of the
        downloaded file against the hash available in the database.
        """
        filename = self.download_url.split('/')[-1]

        with open(filename, "wb") as archive:
            data = urllib.request.urlopen(self.download_url)
            archive.write(data.read())

        if not self.check_md5(filename):
            pass

        return filename

    def check_md5(self, filename):
        """
        This method takes a filename and checks whether the file's md5 matches
        the one in the database. It is used for verifying the integrity of a
        downloaded package.
        """
        with open(filename, "rb") as fh:
            file_md5 = hashlib.md5(fh.read()).hexdigest()

        return file_md5 == self.md5

    def extract_package(self):
        """
        This method decompresses a downloaded package.
        """
        if tarfile.is_tarfile(self.filename):
            with tarfile.open(self.filename) as archive:
                archive.extractall()

    def autoconf(self):
        """
        This method applies the required patches to a package and runs
        autoconf. The output of autoconf is logged under {lazy_logs}.

        autoconf always runs

        './configure --prefix={lazy_prefix}/{name}'

        additional flags might be used from the database
        """
        basedir = os.getcwd()

        autoconf_script = os.path.join(basedir, self.srcdir, 'configure')
        autoconf_prefix = '--prefix=' + os.path.join(self.prefix, self.name)
        configure = [autoconf_script, autoconf_prefix]

        autoconf_db_args = self.args.split()
        configure.extend(autoconf_db_args)

        log_file = os.path.join(self.log_path, self.name + self.autoconf_ext)

        self.apply_patch(basedir)

        if not os.path.isdir(self.objdir):
            os.mkdir(self.objdir)

        os.chdir(self.objdir)
        with open(log_file, 'w') as log_fh:
            subprocess.call(configure, stdout=log_fh, stderr=log_fh)
        os.chdir(basedir)

    def apply_patch(self, basedir):
        """
        This method goes into the source code directory and applies the patches
        from there. It logs the operation in {lazy_logs}.
        """
        patches = self.get_patches(self.pkg_id)
        log_file = os.path.join(self.log_path, self.name + self.patch_ext)
        patch = ['patch', '-p0']

        os.chdir(self.srcdir)
        with open(log_file, 'w') as log_fh:
            for patch_file in patches:
                patch_file = os.path.join(basedir, self.patch_path, patch_file)
                with open(patch_file) as patch_fh:
                    subprocess.call(patch, stdin=patch_fh, stdout=log_fh,
                                    stderr=log_fh)
        os.chdir(basedir)

    def automake(self):
        """
        This method runs automake inside the object directory. If successful,
        it also updates the installed version of the package. All operations
        are logged under {lazy_logs}.
        """
        basedir = os.getcwd()
        log_file = os.path.join(self.log_path, self.name + self.automake_ext)
        make = ['make']
        install = ['make', 'install']

        os.chdir(self.objdir)
        with open(log_file, 'w') as log_fh:
            rvalue = subprocess.call(make, stdout=log_fh, stderr=log_fh)

            if rvalue == 0:
                rvalue = subprocess.call(install, stdout=log_fh, stderr=log_fh)
        os.chdir(basedir)

        if rvalue == 0:
            self.update_installed_version(self.latest_version, self.pkg_id)

    def cleanup(self):
        """
        This method removes the files that were created during installation.
        """
        shutil.rmtree(self.srcdir)
        if os.path.isdir(self.objdir):
            shutil.rmtree(self.objdir)
        os.remove(self.filename)

    def remove_package_dir(self, install_path):
        """
        This method takes an installation path and removes it, effectively
        removing the package.
        """
        shutil.rmtree(install_path)

    def symlink(self):
        """
        This method creates symlinks for every binary of a package inside
        {bindir}.
        """
        for binary in os.listdir(os.path.join(self.prefix, self.name, 'bin')):
            os.symlink(binary, os.path.join(self.bindir, binary))

    def unlink_binaries(self, bin_list):
        """
        This method takes a list of binaries and removes the symlinks in
        {bindir}.
        """
        for binary in bin_list:
            os.unlink(os.path.join(self.bindir, binary))

lazy_db.py

"""
This file is an interface to the database for Lazy. All the methods dealing
with the database are stored here.
"""
import sqlite3


class LazyDB:

    db_name = "lazy.sqlite"
    db_init_file = "lazy.sql"

    def __init__(self):
        """
        When an instance of this class is created, it establishes a database
        connection.
        """
        self.start_db_conn()

    def __del__(self):
        """
        When an object is destroyed, the changes are committed, and the
        database connection is closed.
        """
        self.db_conn.commit()
        self.close_db_conn()

    def start_db_conn(self):
        """
        This method simply starts a database connection and creates a cursor
        for it.
        """
        self.db_conn = sqlite3.connect(self.db_name)
        self.db_crsr = self.db_conn.cursor()

    def close_db_conn(self):
        """
        This method simply closes a database connection.
        """
        self.db_crsr.close()

    def get_package_info(self, col_list, pkg_id):
        """
        This method takes a list of column names and a package id as arguments
        and returns the contents of the columns in the respective row.
        """
        db_data = (pkg_id, )
        columns = ", ".join(col_list)

        db_query = "SELECT " + columns + " FROM packages WHERE package_id = ?"
        self.db_crsr.execute(db_query, db_data)
        db_result = self.db_crsr.fetchall()

        return db_result

    def get_package_name(self, pkg_id):
        """
        This method takes a package id and returns the name of the package.
        """
        return self.get_package_info(["name"], pkg_id)[0][0]

    def get_package_download_url(self, pkg_id):
        """
        This method takes a package id and returns the download url of the
        package.
        """
        return self.get_package_info(["download_url"], pkg_id)[0][0]

    def get_package_md5(self, pkg_id):
        """
        This method takes a package id and returns the md5 of the package.
        """
        return self.get_package_info(["md5"], pkg_id)[0][0]

    def get_package_srcdir(self, pkg_id):
        """
        This method takes a package id and returns the srcdir of the package.
        """
        return self.get_package_info(["srcdir"], pkg_id)[0][0]

    def get_package_objdir(self, pkg_id):
        """
        This method takes a package id and returns the objdir of the package.
        """
        return self.get_package_info(["objdir"], pkg_id)[0][0]

    def get_package_args(self, pkg_id):
        """
        This method takes a package id and returns the autoconf arguments of
        the package.
        """
        return self.get_package_info(["args"], pkg_id)[0][0]

    def get_installed_version(self, pkg_id):
        """
        This method takes a package id and returns whether the installed
        version of the package.
        """
        return self.get_package_info(["installed_version"], pkg_id)[0][0]

    def get_latest_version(self, pkg_id):
        """
        This method takes a package id and returns the latest version of the
        package.
        """
        return self.get_package_info(["latest_version"], pkg_id)[0][0]

    def get_package_id(self, pkg_name):
        """
        This method takes the name of a package and returns its id.
        """
        db_data = (pkg_name, )

        db_query = "SELECT package_id FROM packages WHERE name = ? LIMIT 1"
        self.db_crsr.execute(db_query, db_data)

        return self.db_crsr.fetchone()[0]

    def update_package_info(self, col_name, col_value, pkg_id):
        """
        This method takes a column name and a new value for that column, and
        updates the database row accordingly.
        """
        data = (col_value, pkg_id, )

        db_query = "UPDATE packages SET " + col_name + " = ? "
        db_query += "WHERE package_id = ?"

        self.db_crsr.execute(db_query, data)

    def update_name(self, new_name, pkg_id):
        """
        This method takes a package id and a new name for that package and
        modifies the database records accordingly.
        """
        self.update_package_info("name", new_name, pkg_id)

    def update_download_url(self, new_url, pkg_id):
        """
        This method takes a package id and a new download url for that package
        and modifies the database records accordingly.
        """
        self.update_package_info("download_url", new_url, pkg_id)

    def update_md5(self, new_md5, pkg_id):
        """
        This method takes a package id and a new md5 hash for that package and
        modifies the database records accordingly.
        """
        self.update_package_info("md5", new_md5, pkg_id)

    def update_srcdir(self, new_srcdir, pkg_id):
        """
        This method takes a package id and a new srcdir for that package and
        modifies the database records accordingly.
        """
        self.update_package_info("srcdir", new_srcdir, pkg_id)

    def update_objdir(self, new_objdir, pkg_id):
        """
        This method takes a package id and a new objdir for that package and
        modifies the database records accordingly.
        """
        self.update_package_info("objdir", new_objdir, pkg_id)

    def update_args(self, new_args, pkg_id):
        """
        This method takes a package id and a new set of autoconf arguments for
        that package and modifies the database records accordingly.
        """
        self.update_package_info("args", new_args, pkg_id)

    def update_installed_version(self, new_version, pkg_id):
        """
        This method takes a package id and updates the installed version of the
        package accordingly.
        """
        self.update_package_info("installed_version", new_version, pkg_id)

    def update_latest_version(self, new_version, pkg_id):
        """
        This method updates the latest available version of a given package.
        """
        self.update_package_info("latest_version", new_version, pkg_id)

    def init_db(self):
        """
        This method initiates the database by running the .sql file bundled
        with the application.
        """
        with open(self.db_init_file) as init_fh:
            self.db_crsr.executescript(init_fh.read())

    def add_package(self, package_info):
        """
        This method inserts a package into the database. It takes a dictionary
        of (column_name:value) pairs and creates a new package based on those.
        """
        p = package_info
        db_data = (p['name'], p['download_url'], p['md5'], p['srcdir'],
                p['objdir'], p['args'], p['latest_version'])

        db_query = """
        INSERT INTO packages (name, download_url, md5, srcdir, objdir, args,
                              latest_version)
        VALUES(?, ?, ?, ?, ?, ?, ?)
        """

        self.db_crsr.execute(db_query, db_data)

    def remove_package(self, pkg_id):
        """
        This method takes a package id and removes the respective package.
        """
        db_data = (pkg_id, )

        db_query = "DELETE FROM packages WHERE package_id = ? LIMIT 1"
        self.db_crsr.execute(db_query, db_data)

    def get_dependencies(self, pkg_id):
        """
        This method returns a list of dependencies for a given package.
        """
        db_data = (pkg_id, )

        db_query = "SELECT package_dep_id FROM dependencies "
        db_query += "WHERE package_id = ?"
        self.db_crsr.execute(db_query, db_data)
        db_result = self.db_crsr.fetchall()

        deps = [result[0] for result in db_result]
        return deps

    def add_dependency(self, dep_id, pkg_id):
        """
        This method takes a dependency and associates it to a given package.
        """
        db_data = (pkg_id, dep_id, )

        db_query = """
        INSERT INTO dependencies(package_id, package_dep_id) VALUES(?, ?)
        """

        self.db_crsr.execute(db_query, db_data)

    def remove_dependency(self, dep_id, pkg_id):
        """
        This method takes a dependency and removes it from the database.
        """
        db_data = (dep_id, pkg_id)

        db_query = "DELETE FROM dependencies WHERE package_dep_id = ? "
        db_query += "AND package_id = ? LIMIT 1"

        self.db_crsr.execute(db_query, db_data)

    def get_patches(self, pkg_id):
        """
        This method takes a package id and returns the patch file(s) of the
        package.
        """
        db_data = (pkg_id, )

        db_query = "SELECT patch_file FROM patches WHERE package_id = ?"
        self.db_crsr.execute(db_query, db_data)
        db_result = self.db_crsr.fetchall()

        patches = [result[0] for result in db_result]
        return patches

    def add_patch(self, patch_file, pkg_id):
        """
        This method takes the name of a patch file and a package id and
        associates the patch file to its package.
        """
        db_data = (patch_file, pkg_id, )

        db_query = "INSERT INTO patches(patch_file, package_id) VALUES(?, ?)"
        self.db_crsr.execute(db_query, db_data)

    def remove_patch(self, patch_file, pkg_id):
        """
        This method takes a patch file and a package id and removes the patch
        from the database.
        """
        db_data = (patch_file, pkg_id, )

        db_query = """
        DELETE FROM patches WHERE patch_file = ? AND package_id = ?
        """
        self.db_crsr.execute(db_query, db_data)

    def is_installed(self, pkg_id):
        """
        This method takes a package id and determines whether it is installed
        or not.
        """
        return self.get_installed_version(pkg_id) != 0

    def get_all_packages(self):
        """
        This method returns the names of all the available packages.
        """
        db_query = "SELECT name FROM packages"
        self.db_crsr.execute(db_query)
        db_result = self.db_crsr.fetchall()

        return [result[0] for result in db_result]

    def get_packages_by_name(self, pkg_name):
        """
        This method takes a package name and returns all the related packages.
        """
        db_data = ("%" + pkg_name + "%", )

        db_query = "SELECT name FROM packages WHERE name LIKE ?"
        self.db_crsr.execute(db_query, db_data)
        db_result = self.db_crsr.fetchall()

        return [result[0] for result in db_result]

    def get_all_package_info(self, pkg_id):
        """
        This method takes a package id and returns all the package info
        associated with it.
        """
        db_data = (pkg_id, )

        db_query = "SELECT * FROM packages WHERE package_id = ? LIMIT 1"
        self.db_crsr.execute(db_query, db_data)
        db_result = self.db_crsr.fetchone()

        return db_result

lazy.sql

/*
This is the file which lays down the database structure of the Lazy package
manager. Although Lazy uses SQLite, the file should be kept as portable as
possible, so it should work with other SQL implementations with close to no
modification.

It includes a sample package `splint`, a linter for C. It was chosen because it
requires a patch in order to compile under OS X.
*/

-- Let's drop any existing tables
DROP TABLE IF EXISTS packages;
DROP TABLE IF EXISTS dependencies;
DROP TABLE IF EXISTS patches;

-- This table contains info about the packages
CREATE TABLE IF NOT EXISTS packages (
    package_id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    download_url TEXT NOT NULL,
    md5 TEXT NOT NULL,
    srcdir TEXT NOT NULL,
    objdir TEXT NOT NULL,
    args TEXT NOT NULL DEFAULT '',
    installed_version INTEGER NOT NULL DEFAULT 0,
    latest_version INTEGER NOT NULL
);

-- This table contains dependencies info
CREATE TABLE IF NOT EXISTS dependencies (
    dep_id INTEGER PRIMARY KEY,
    package_id INTEGER NOT NULL REFERENCES packages(package_id),
    package_dep_id INTEGER NOT NULL REFERENCES package(package_id)
);

-- This table contains info about patches to be applied to files
CREATE TABLE IF NOT EXISTS patches (
    patch_id INTEGER PRIMARY KEY,
    package_id INTEGER NOT NULL REFERENCES packages(package_id),
    patch_file TEXT NOT NULL
);

-- Add `splint`
INSERT INTO
packages(name, download_url, md5, srcdir, objdir, latest_version)
VALUES( 'splint',
        'http://www.splint.org/downloads/splint-3.1.2.src.tgz',
        '25f47d70bd9c8bdddf6b03de5949c4fd',
        'splint-3.1.2',
        'splint-3.1.2',
        312);

INSERT INTO patches(package_id, patch_file) VALUES(1, 'splint.diff');
\$\endgroup\$
2
\$\begingroup\$
def install(self, pkg_id):
    """
    This method takes a package id and installs the associated package. It
    does so by getting all the information from the database and calling
    different methods to perform the installation steps.
    """
    self.resolve_dependencies(pkg_id)

    (self.pkg_id, self.name, self.download_url, self.md5,
    self.srcdir, self.objdir, self.args, self.installed_version,
    self.latest_version) = self.get_all_package_info(pkg_id)

That's a lot of data to be fetching. What if you want to add or remove or change the order of some of that? Instead, put all of that information into a Package class and return that.

Don't store temporary state on self. If you need to pass temporary information use parameters.

get_all_package_info is from LazyDB. But Lazy uses LazyDB it is not a specialization of LazyDB. So you shouldn't inherit from LazyDB, instead you should store a refence to LazyDB as self.db.

    self.filename = self.download_package()
    self.extract_package()
    self.autoconf()
    self.automake()
    self.symlink()
    self.cleanup()

All of this relates specifically to the package, it should all really be done by calling package.install() where package is the package object you should return.

Overall, this function should look like

def install(self, pkg_id):
    package = self.db.load(pkg_id)
    self.resolve_dependencies(package)
    package.install()

You've got functions to manipulate pretty much every possible field you might want to change in the database. Instead, I'd suggest having a few functions, that work with Package objects.

The thing with package managers is that you typically want to update the package details without changing the information about what the user has installed. But you've got that data in the same table. What you probably want to do is actually have two databases. One database has the package information, and you update it by downloading a new copy to replace the old one. The other package has the installation information.

\$\endgroup\$
  • \$\begingroup\$ Hmm... so what you're saying is that I should write a third class with attributes such as "name" and "download_url", and methods such as "install" and "download_package"? So in a MVC analogy, LazyDB and Package would be the Models, Lazy the Controller, and the eventual CLI program the View (the View is kind of a stretch)? Also, thanks for the input, it's greatly valued! \$\endgroup\$ – Grewe Kokkor Apr 12 '12 at 19:05
  • \$\begingroup\$ @GreweKokkor, yes that's a good description of what I'm suggesting \$\endgroup\$ – Winston Ewert Apr 12 '12 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.