# Merging directories and keep files that have more lines

Goal

My goal is to merge directories. Whenever a file has the same name in two or more directories, then only the one that has the highest number of lines should be kept. If both files have the same number of files and differ then an error message should be thrown. Note that files having more lines are also bigger (in my specific case) which might be another way to compare files having the same name.

My Code

Here is my code that I think works fine

### Parameters ###

GeneralPath="/Users/remi/Documents/Biologie/Vancouver/PhD/Thesis/BackgroundSelection/Simulations/s_and_Pi/outputs/4.0.2_1.0.5/"

cd ${GeneralPath} Directories=( HR OR OS ) # Array of directories ot be merged with the destination directory Destination=HS errorFile="${GeneralPath}MergeDirs.err"

### Do Stuff ###

for d in ${Directories[@]};do echo "${d}"
cd $d for f in *; do echo "${f}"
if [ ! -f "../${Destination}/${f}" ];then
echo cp1
cp ${f} ../${Destination}/
else
nblinesFrom=$(wc -l${f} | awk -F" " '{print $1}') nblinesDest=$(wc -l "../${Destination}/${f}" | awk -F" " '{print $1}') if [${nblinesFrom} -gt ${nblinesDest} ]; then echo cp2 cp${f} ../${Destination}/ elif [${nblinesDest} -gt ${nblinesFrom} ];then echo "Destination is bigger - nothing to do" else DoTheyDiffer=$(diff ${f} "../${Destination}/${f}" | wc -l) if [${DoTheyDiffer} -gt 0 ];then
echo "${f} and ../${Destination}/${f} diff but have the same number of lines" >>${errorFile}
fi
fi
fi
done
cd ..
done


My code seems quite complicated and I feel like a good combinaison of find -exec, awk, cp and diff might do something much more fancy.

• Use More Quotes™.
• Use a shebang line (disclosure: I wrote that answer).
• Don't use single character variables. Maintainability is the most important feature of code.
• Rather than echo cp1 etc., simply use cp -v to print every copy command verbatim.
• You don't need to count the number of lines that diff returns, you can simply do if diff foo bar, or the safer option:

diff foo bar
exit_code=$? if [ "$exit_code" -eq 0 ]
then
[no difference]
elif [ "$exit_code" -eq 1 ] then [difference] else exit "$exit_code" # WTF
fi

• Some people love mixing languages. I think cut -d' ' -f1 is much nicer than even a short awk script.
• If you pass the file to wc on standard input, it doesn't print the filename, and so you don't need to process the output at all: wc -l < /path
• You can use if cp --no-clobber source destination to try copying the file instead of checking whether the target exists.
• I'm not sure I understand why you're copying rather than moving files (unless this is a one-off script, which you're not going to test, and it's only going to take a few seconds anyway).
• Be wary of using cd in scripts. It changes the context in a significant way, and makes it harder to reason about what the script will do. Instead, simply do for path in "\$directory"/*.
• I can definitely recommend set -o errexit -o noclobber -o nounset -o pipefail. You could also use -o xtrace to make all those logging commands obsolete.
• A common convention to get used to is to never end paths with a slash. Firstly because cp a b/ and cp a b and the same as long as b` is a directory, and secondly because it makes it more natural to concatenate paths without ending up with double slashes.