Script to checkout multiple repositories to a certain commit hash

Question

I am writing a script which checkout a git repo to certain commit hash, do something and switch back to master. The purpose of this script is to take homework solutions of students from bitbucket. Note that all the repos are under same bitbucket account. There is a master bitbucket account which is admin of all these repos and students have the write access to their respective repo. The students must adhere to following directory structure in their repos:

-assignments
 |- assignment-1
 |- assignment-2
 .
 .
 .
 |- assignment-X

The directories inside these contain the homework. Once the teacher has given the deadline, the students must commit their code before the deadline. The script will see the git log, find the commit which is made before deadline, switch to that revision and rsync the solutions to the local directory.

So, this script will:

• First get the list of bitbucket repo names from a file (students-info.json)
• For each repo, see if the repo already exists locally. If it does, then do a git pull to get the latest commit
• If not do a git clone
• Now, find a commit which is made before deadline
• switch to that revision
• do a rsync of the required assignment homework directory to solutions-directory/assignment-x-deadline/student-id
• switch back to master branch

I am looking for any tips, suggestions, general code improvements, bugs, anything.

Here is my code:

#!/bin/python

""" 
This script will take assignment solutions from each student repository. Based 
on the timestamp given, it finds out the last commit made before timestamp 
(i.e. deadline) and it checks out that revision, rsyncs the solution folder 
of the required assignment with the solutions-repo and resets to HEAD.

The timestamp should be of the format 'Month Date H:M:S Year' 

eg. Dec 19 22:31:01 2013

Input : List of students ids, assignment-id, timestamp

Example usage: To take out solutions of assignment 11 whose deadline was 
Dec 19 22:31:01 2013, run the following

$python take_solutions.py -d 'Dec 19 22:31:01 2013' -a 'assignment-11'


To do: 
- git_log_cmd with format string in get_commit_hash()
- dest_path is ugly
- dest_path should be global?  
- 
"""


import string
import os
import time
import datetime
import subprocess
import json
import argparse
import shlex
import logging
import datetime
from logging.handlers import TimedRotatingFileHandler

from dir_settings import *
from bb_settings import *

parser = argparse.ArgumentParser(description='This script will take assignment solutions from each student repository. Based on the timestamp given, it finds out the last commit made before timestamp (i.e. deadline) and it checks out that revision, rsyncs the solution folder of the required assignment with the solutions-repo and resets to HEAD.')
parser.add_argument('-d','--deadline', help='The timestamp should be of the \
                    format "Month Date H:M:S Year" e.g. "Dec 19 22:31:01 2013"', 
                    required=True)
parser.add_argument('-a','--assignment_id', help='Please provide assignment \
                    id of the solutions you want to copy. e.g. assignment-7', 
                    required=True)

NITRO_LOGGER = logging.getLogger('NITRO')
LOG_FILENAME = 'nitro.log'
SOLUTIONS_DIRECTORY = 'solutions-directory/'
STUDENTS_REPO_DIRECTORY = 'students-repo-directory/'
students_info = json.loads(open(STUDENTS_INFO, 'r').read())

args = vars(parser.parse_args())
assignment_id = args['assignment_id']
deadline = args['deadline']
DEST_PATH = SOLUTIONS_DIRECTORY + assignment_id + '-' + '-'.join(deadline.split()) + '/'


def get_commit_hash(repo_name, timestamp):
    git_log_cmd = shlex.split('git --git-dir=' + STUDENTS_REPO_DIRECTORY + repo_name + '/.git log --pretty=format:"%H %ad" --date=local')
    try:
        (output, error) = subprocess.Popen(git_log_cmd, stdout=subprocess.PIPE, 
                                            stderr=LOG_FD).communicate()
        for git_log in string.split(output, os.linesep):
            deadline = datetime.datetime.strptime(timestamp, "%b %d %H:%M:%S %Y")
            # split the commit message by first white space, the returning list will 
            # have hash as its first element and timestamp as second element
            commit_hash = git_log.split(' ', 1)[0] 
            commit_timestamp = git_log.split(' ', 1)[1]
            if deadline > datetime.datetime.strptime(commit_timestamp, "%a %b %d %H:%M:%S %Y"):
                return commit_hash
    except Exception, e:
        NITRO_LOGGER.error("Couldn't get commit hash before deadline for repo %s: %s" % (repo_name, str(e)))
        #raise e


def sync_solutions(repo_name):

    def repo_exists(repo_name):
        return os.path.isdir(STUDENTS_REPO_DIRECTORY + repo_name)

    def clone_repo(repo_name):
        clone_cmd = shlex.split("git clone %s%s %s%s" % (BB_REPO_BASE_URL, 
                                 repo_name, STUDENTS_REPO_DIRECTORY, repo_name))
        subprocess.check_call(clone_cmd, stdout=LOG_FD, stderr=LOG_FD)

    def pull_repo(repo_name):
        pull_cmd = shlex.split("git --git-dir=%s/.git pull" % \
                            (STUDENTS_REPO_DIRECTORY + repo_name))
        subprocess.check_call(pull_cmd, stdout=LOG_FD, stderr=LOG_FD)

    def checkout_version(repo_name, commit_hash='-'):
        checkout_cmd = shlex.split("git --git-dir=%s/.git checkout %s" \
                                % ((STUDENTS_REPO_DIRECTORY + repo_name), commit_hash))
        subprocess.check_call(checkout_cmd, stdout=subprocess.PIPE, 
                            stderr=subprocess.PIPE)

    def rsync(repo_name):
        src_path = STUDENTS_REPO_DIRECTORY + repo_name + '/assignments/' + assignment_id
        if not os.path.isdir(src_path):
            # either student messed up the dir structure or hasn't submitted his assignments
            return
        if not os.path.isdir(DEST_PATH + repo_name):
            os.makedirs(DEST_PATH + repo_name)
        rsync_cmd = shlex.split('rsync -rt %s %s' % (src_path, DEST_PATH + repo_name))
        subprocess.check_call(rsync_cmd, stdout=LOG_FD, stderr=LOG_FD)


    try:
        if repo_exists(repo_name):
            pull_repo(repo_name)
        else:
            clone_repo(repo_name)
    except Exception, e:
        NITRO_LOGGER.error('pull/clone repo failed for repo %s: %s', repo_name, str(e))
        return

    commit_hash = get_commit_hash(repo_name, deadline)
    if commit_hash:
        try:
            checkout_version(repo_name, commit_hash)
            rsync(repo_name)
            checkout_version(repo_name)
        except Exception, e:
            NITRO_LOGGER.error('git checkout failed for repo %s: %s' % (repo_name, str(e)))
    else:
        NITRO_LOGGER.debug('No assignment found before deadline for ' + repo_name)
        return


def setup_logging():
    NITRO_LOGGER.setLevel(logging.DEBUG)   # make log level a setting
    # Add the log message handler to the logger
    myhandler = TimedRotatingFileHandler(LOG_FILENAME, when='midnight', 
                                        backupCount=5)
    formatter = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
        datefmt='%Y-%m-%d %I:%M:%S %p')
    myhandler.setFormatter(formatter)
    NITRO_LOGGER.addHandler(myhandler)


def init():
    if not os.path.isdir(STUDENTS_REPO_DIRECTORY):
        os.makedirs(STUDENTS_REPO_DIRECTORY)
    if not os.path.isdir(SOLUTIONS_DIRECTORY):
        os.makedirs(SOLUTIONS_DIRECTORY)
    if not os.path.isdir(DEST_PATH):
        os.makedirs(DEST_PATH)


def main():
    NITRO_LOGGER.debug('****Firing up NITRO***')
    init()
    for student_id, student_email in students_info.iteritems():
        NITRO_LOGGER.debug(student_id)
        sync_solutions(student_id)
    NITRO_LOGGER.debug('****Done with NITRO***')


if __name__ == '__main__':
    LOG_FD = open(LOG_FILENAME, 'a')
    setup_logging()
    main()

Answer 1

Your idea of inspecting the timestamps of the commits is conceptually flawed. Git is a distributed version control system, with no central server or any other means of notarizing timestamps. The timestamps are determined solely by the system clock on the machine on which the commit was created, and that clock can be trivially rolled back. Therefore, the only foolproof approach is to clone/pull all of the repositories at the time of the deadline.

Then, there is the question of which branch you want to inspect. Do you want to consider only the master branch? If so, it would be a good idea to specify the master branch when running git log. Keep in mind that if you consider all commits that were created before the deadline, you may end up taking a commit that was rolled back by the student. In other words, if the student makes a commit, then changes her mind (using git reset --hard HEAD^), you may be misconstruing the discarded version as the submission, simply because it has a later timestamp. For that reason, I hope that you only inspect commits along an agreed-upon branch or tag, rather than everything that might happen to exist in the repository.

In get_commit_hash(), you use the %ad pretty-printing format to obtain commit_timestamp. That's a misnomer, as %ad gets the authorship timestamp, not the commit timestamp. I believe you should be more interested in the commit timestamp. (Authorship times aren't even necessarily monotonic as you progress through the commit chain, since commits can be rearranged using git rebase.)

Assuming that you still want to go through with your original plan, you're working too hard. This should get you the hash of the latest commit on the master branch with a commit date in 2013:

git log -n 1 --until='2013-12-31 23:59:59' --pretty=%H master

Better yet, read what gitrevisions(1) says about "the value of the ref at a point in time", and skip all that analysis.

git checkout 'master@{2013-12-31 23:59:59}'

---

By the way, I strongly recommend that you abandon your date format in favour of ISO 8601.

Credit: https://xkcd.com/1179/