Display git commit statistics in bash

Question

I have scripts to display commit statistics and merge statistics of my repos, and they work. I wrote them for my personal usage, and because I was interested in finding trends in my git repos.

This script reports statistics about commits (number, average length in words, etc.). Relevant commits can be selected using git-rev-list options.

Features and times

• count: report number of commits (not a performance issue)
• len: report length in words of commit message and commit hash (~20s for 1561 commits)
• len min, len max, and len avg: report minimum, maximum, or average commit message length in words and commit hash (~10-15s for the same)

Benchmarks run with bash's time on my dotfiles repo

A previous implementation using for-loops had similar performance.

Obviously, the algorithms are O(n). They are still too slow for every-day usage.

#! /usr/bin/env bash

set -euo pipefail

USAGE='[-h] (count | len [min|max|avg]) [rev-list options]

Display commit statistics
Filter commits based on [rev-list options]'
SUBDIRECTORY_OK=true

# source git-sh-setup for some helpers
set +u
source "$(git --exec-path)"/git-sh-setup
set -u

SIZER=(
  wc
  # count words
  -w
)

size() {
  local commit="$1"
  git log "$commit" -1 --format=%B | "${SIZER[@]}" | tr -d ' '
}

commits_list() {
  command=(
    git
    rev-list
    # start somewhere
    --all
  )
  if (($# > 0)) ; then
    command+=("$@")
  fi
  "${command[@]}" 2>/dev/null
}

commit_count() {
  commits_list "$@" | wc -l | tr -d ' '
}

commit_len() {
  commits_list "$@" |
    while read c ; do
      size "$c" | tr -d '[:space:]'
      printf ' %s\n' "$c"
    done
}

commit_len_min() {
  commit_len "$@" |
    sort -n |
    head -n 1
}

commit_len_max() {
  commit_len "$@" |
    sort -rn |
    head -n 1
}

commit_len_avg() {
  local num=0
  {
    printf '%s\n' '5k'
    while read c ; do
      ((++num))
      size "$c"
      ((num >= 2)) && printf '%s\n' '+'
    done < <(commits_list "$@")
    printf '%s\n' "$num" '/p'
  } | dc
}

main() {
  (($# >= 1)) || usage
  case "$1" in
    count) commit_count "${@:2}" ;;
    len)
      if (($# >= 2)); then
        case "$2" in
          max|min|avg) commit_len_"$2" "${@:3}" ;;
          *) commit_len "${@:2}" ;;
        esac
      else
        commit_len "${@:2}"
      fi
      ;;
    *) usage ;;
  esac
}

main "$@"

Shell scripts being hard to profile, I've been unable to identify the bottleneck (though commit_len seems like a good place to start).

I run shellcheck regularly.

Answer 1

Some suggestions:

1. shellcheck should give you a few suggestions. I won't mention things I expect it to find.
2. Uppercase names are by convention only used for exported variables.
3. SUBDIRECTORY_OK is unused. If it's a magic variable this probably should be mentioned.
4. SIZER is only used once, so it should be inlined.
5. wc -w 8.30 from GNU coreutils, at least, does not output any spaces, so tr -d ' ' might be unnecessary.
6. (($# > 0)) would usually be written [[ "$#" -gt 0 ]].
7. Throwing away standard error means the script will be harder to debug. If there's specific output there you want to hide you can use cmd 2> >(grep -v … >&2)
8. commit_len is slow because for each commit you run a git command & more to count the number of commits before it. Which means you traverse the Git history N times. I think you'll get the same result by running size "$1".
9. You can use shift to simplify things like "${@:2}" to just "$@".
10. dc is not a tool I'm familiar with, but it will certainly be faster to count using something like awk to gobble the whole stream in one command. while read is actually surprisingly slow.

Answer 2

I've managed to drastically improve the performance by combining awk with some creative formatting: now that everything is awk, the script flies under 0.3s for even my ~1600 commits.

Result

#! /usr/bin/env bash

set -euo pipefail

USAGE='[-h] (count | len [min|max|avg]) [rev-list options]

Display commit statistics
Filter commits based on [rev-list options]'
SUBDIRECTORY_OK=true

# source git-sh-setup for some helpers
set +u
source "$(git --exec-path)"/git-sh-setup
set -u

commits_list() {
  command=(
    git
    log
    --pretty'='format:$'\a%n%H\t%s %b'
    # start somewhere
    --all
  )
  if (($# > 0)) ; then
    command+=("$@")
  fi
  "${command[@]}" |
    awk '
      /'$'\a''/ && NR != 1 { printf "\n"; next }
      { printf "%s ", $0 }
      END { printf "\n" }
      '
}

commit_count() {
  git rev-list --all --count "$@"
}

commit_len() {
  commits_list "$@" |
    awk -F$'\t' '{ print split($2,_," "), $1 }'
}

commit_len_min() {
  commit_len "$@" |
    sort -n |
    head -n 1
}

commit_len_max() {
  commit_len "$@" |
    sort -rn |
    head -n 1
}

commit_len_avg() {
  commit_len "$@" |
    awk '
      { sum += $1 }
      END { print sum/NR }
    '
}

main() {
  (($# >= 1)) || usage
  case "$1" in
    count) commit_count "${@:2}" ;;
    len)
      if (($# >= 2)); then
        case "$2" in
          max|min|avg) commit_len_"$2" "${@:3}" ;;
          *) commit_len "${@:2}" ;;
        esac
      else
        commit_len "${@:2}"
      fi
      ;;
    *) usage ;;
  esac
}

main "$@"