Bash script to remove unwanted git objects

A while ago, someone at our office thought it'd be a great idea to start tracking a number of fairly large binary files in one of our more important repositories. We noticed our builds were slowing down (considerably) and fetching new changes from the remotes could take for up to a minute.

We eventually noticed that there were quite a number of reasonably large objects in our repos, and I was appointed to clean up the mess.
seeing as these objects had been added, removed updated, renamed and replaced in various commits, across a fair number of branches, I decided to write a script to rewrite the head's of all of these branches automatically, instead of going through all the files, and all the commits individually.

Not being a Bash ninja, I stuck to what I know, using a lot of $(<command> | grep | sed | awk ) trickery. I've always gotten by using this approach, but I'd really like to know if bash offers some features I've yet to uncover, that would enable me to write, essentially, better scripts. Hence, I'd like to get some feedback on the type of scripts I'm currently writing, and how I could do better: #!/usr/bin/env bash SCRIPT=$(basename ${BASH_SOURCE[0]}) verbose=false idxfile="packidx.log" forcepush=false filterflag="--index-filter" #get current branch currentbranch=$(git branch | grep '*' | awk '{print $2}') function Help { echo "Usage$SCRIPT [-svfh][-i value]:"
echo "     -i [packidx.log]: specify an existing file, containing sorted git verify-pack -v output"
echo "                       Default is to create or prompt to reuse an existing packidx.log file"
echo "     -v              : verbose output"
echo "     -s              : slow, use tree-filter instead of index-filter when removing objects"
echo "     -f              : Force push. Whenever an object is removed from a branch, perform a force-push"
echo "     -h              : Help. Display this message"
}

function AfterFilter {
if [ "$verbose" = true ] ; then echo 'cleaning up .git/refs/original and .git/logs, then gc the git DB' fi rm -Rf .git/refs/original rm -Rf .git/logs/ git gc if [ "$verbose" = true ] ; then
echo 'object-count stats after filter'
git count-objects -v
fi
git prune --expire now
if [ "$verbose" = true ] ; then echo 'object-count stats after prune' git count-objects -v fi echo '' if [ "$forcepush" = true ] ; then
git push --force
else
if [[ ! $REPLY =~ ^[nN]$ ]] ; then
git push --force
fi
fi
}

if [ $# -gt 0 ] ; then while getopts :isvfh flag ; do case$flag in
i)
idxfile=$OPTARG ;; f) forcepush=true ;; v) verbose=true ;; s) filterflag="--tree-filter" ;; h) Help exit 0 ;; \?) Help exit 1 ;; esac done fi if [ ! -f$idxfile ]; then
else
read -p "create $idxfile file? [y/N]: " -n 1 -r fi if [[$REPLY =~ ^[yY]$]] then echo "Creating$idxfile on branch $currentbranch" git gc packfile=$(ls .git/objects/pack/*.idx)
git verify-pack -v "$packfile" | sort -k 3 -n > packidx.log fi for objectref in$(tac packidx.log | grep blob | cut -d " " -f1); do
if [ "$verbose" = true ] ; then echo 'object-count stats' git count-objects -v fi if [ "$verbose" = true ] ; then
echo "get filename for object $objectref" fi filename=$(git rev-list --objects --all | grep $objectref | sed -n -e "s/^$objectref //p")
read -p "process all commits modifying $filename? [y/N] " -n 1 -r if [[$REPLY =~ ^[Yy]$]] then if [ "$verbose" = true ] ; then
echo "get all commits modifying $filename" git log --oneline --branches -- "$filename"
fi
# output is for user info only, use commit refs here:
commits=() #array of commits
commitlength=0
for commit in $(git log --oneline --branches -- "$filename" | awk '{print $1;}'); do commits[commitlength]=$commit
commitlength=$((commitlength+1)) done if (( commitlength == 0 )) ; then echo "No commits found for$filename, must be a dangling object"
else
commitlength=$((commitlength-1)) #last commit for (( i=commitlength; i>0; i--)); do #while [$commitlength -ge 0 ] ; do
for branch in $(git branch --contains${commits[$i]} | cut -c 3-) ; do #which branch is rewritten is considered vital info, verbose or not #echo this line echo "rewriting$branch for commit ${commits[$i]}"
if [[ ! "$branch" =~ "$currentbranch" ]] ; then
git checkout $branch fi git filter-branch --force$filterflag "git rm --ignore-unmatch --cached $filename" --prune-empty -- "${commits[$i]}"^.. AfterFilter if [ "$verbose" = true ] ; then
echo "$branch rewritten" fi if [[ ! "$branch" =~ "$currentbranch" ]] ; then #return to current branch git checkout$currentbranch
fi
done
echo $i done #checkout the initial branch git checkout "$currentbranch"
fi
fi
read -p 'continue? [Y/n]: ' -n 1 -r
if [[ $REPLY =~ ^[nN]$ ]]
then
break
fi
done
echo '' #insert blank line
read -p "remove $idxfile? [y/N]: " -n 1 -r if [[$REPLY =~ ^[yY]$]]; then rm$idxfile
fi

• what's the problem with git rm? – Vogel612 Jun 20 '15 at 16:22
• @Vogel612: That's what I'm doing in the git filter-branch bit. The problem is that the repo has had several hundred Mb's in bin files added to it. you can remove these files from git using git rm, but the commits containing those files still hold a reference to them, and so git rm will not remove the objects from the repo, resulting in a git clone that pulls close to 2Gb's, half of which is old bin files that shouldn't have been tracked in the first place – Elias Van Ootegem Jun 20 '15 at 16:44
• Essentially, to remove a file that has been committed and pushed, there's no alternative to filter-branch (rewriting all the commits that point to the file, and force-pushing the altered commits). You can find more details here – Elias Van Ootegem Jun 20 '15 at 16:47

1. The shebang: #!/usr/bin/env bash is not needed. env is used to modify the working environment, and since you don't use any of its options, you don't need it. Basically, you're just spending an extra process fork. Just use bash directly: #!/bin/bash.

2. You name your variable SCRIPT, which is a useful name but also uppercase. Uppercase variables are usually reserved by the bash environment. While this is not particularly wrong, you should avoid it to prevent overwriting an important reserved variable like PATH.

3. You get the script name from the command: basename ${BASH_SOURCE[0]}. While this is also not wrong, the modern way would be: {0##\*/}, which is more efficient since it doesn't call any program. Also, since you're only using this variable once in the Help function, it would make more sense to embed ${0##\*/} directly without creating a new variable as it is a ready-made variable.

4. You name your usage function Help. This is not wrong of course, but a more traditional Unix-ish name would be usage.

5. In the Help function, you call echo several times, which is clearly not so efficient (or well-looking). A traditional solution would be to use cat like this:

cat << _EOF__
Usage ....
..
_EOF__

6. You test: [ "$verbose" = true ]. If you already know that $verbose would be either true or false (which you do, in this case), you can just: if $verbose; then because true is treated as a command this time, and it will simply return 0. 7. Since you are regularly checking for true values, you can use a shortcut:$var && if_true_commands. So instead of saying if [ "$verbose" = true ] ; then echo .. you can just say $verbose && echo ... For example, you say:

if [[ ! $REPLY =~ ^[nN]$ ]] ; then
git push --force
fi


You can say:

[[ $REPLY =~ ^[nN]$ ]] || git push --force


This is not all that different, but that's how modern bash scripting style does it, and you have to admit, that's fancier.

8. Small notes:

• You say on line 85: packfile=\$(ls .git/objects/pack/*.idx). However, by using glob expansion, you've already listed the files yourself. So, practically, using echo would be slightly more efficient.

• I noticed that you usually omit double-quoting variables and commands. This might result in undesired expansion behaviour, so you just get the first word of output usually accompanied by some bash errors and warnings if more than one word was split.

• You use echo '' to output an empty line. Well, believe it or not, empty arguments like this one is just ignored by bash, so what you really ran was echo. Hopefully for you, echo just outputs an empty line when it doesn't get any arguments, which is what you wanted after all.

• For further syntax checking and suggestions, you can check shellcheck.net.

• I didn't know about shellcheck.net, nifty! – jacwah Jul 24 '15 at 19:07
• ditto... thanks for letting me know about shellcheck – Elias Van Ootegem Jul 26 '15 at 11:25