Process files in all subdirectories and save output to new files based on their current path

Question

Sample files:

./data/foo/file1.xml
./data/foo/bar/file2.xml
./data/baz/file3.xml

Corresponding output:

./out/foo/file1.xml.xml2
./out/foo/bar/file2.xml.xml2
./out/baz/file3.xml.xml2

The subproblems are:

1. Find all xml files in data/
2. Construct the output path: replace data/ with out/ and add an additional extension
3. Create the output path if it doesn't exist
4. Execute a script, redirect to output file in new path

I've been experimenting with find and I have come up with a solution that works on my test files:

find data/ -name "*.xml" -exec bash -c \
    'out=${1/data/out}.xml2; mkdir -p $(dirname "$out"); ./testscript.sh > $out' - {} \;

(testscript.sh contains echo "Hello World".)

But I'm not that good with bash, and this seems overly verbose, and might contain hidden errors.

Things I've thought about:

• mkdir silently fails if the dir exists, so it'll work for multiple files in the same dir, but maybe this entire script should be done in a loop in order to only call mkdir once per new dir.
• ${1/data/out} only replaces the first occurence of data, which should always exist and be the outermost directory in the variable (since relative paths are used). I'd prefer it if this was done in a more automatic and safe way though, if possible.

Side-note: I have full permissions in the current dir and all subdirs, but not on the entire machine.

Answer 1

(...) this seems overly verbose, and might contain hidden errors.

This is pretty good as it is. I cannot think of a substantially shorter and more readable way. I don't think there are hidden errors either.

mkdir silently fails if the dir exists, so it'll work for multiple files in the same dir, but maybe this entire script should be done in a loop in order to only call mkdir once per new dir.

mkdir doesn't fail. The -p flag makes it exit with success if the directory already exists, so technically, it succeeds. It's true that strictly speaking this is not the most optimal way. It would be ideal to make the script in a way that it doesn't repeatedly call mkdir -p for the same directory. But it's just not worth it. It will make the script more complicated, for benefit so negligible that probably you cannot even measure it. Very often, the simplest solution is the best, even it's not 100% optimal in terms of "performance". If you consider readability and complexity, then it might actually be optimal overall.

${1/data/out} only replaces the first occurence of data, which should always exist and be the outermost directory in the variable (since relative paths are used). I'd prefer it if this was done in a more automatic and safe way though, if possible.

If you always want to replace the first path element with out, whatever it is, not only data, then you can write this way:

find data/ -name "*.xml" -exec bash -c \
    'out=out/${1#*/}.xml2; mkdir -p $(dirname "$out"); ./testscript.sh > $out' - {} \;

The output will be the same as in your example, and will still be the same for find other/ or find blah/.

It might be useful to add the -v flag for the mkdir. This is for verbose mode, it will print the directories it creates (if your system supports the -v flag).

Finally, it's probably good to chain the mkdir and the script following it with &&. So that in case you don't have permission to create directories the script doesn't need to run and double the errors printed on the script.

find data/ -name "*.xml" -exec bash -c \
    'out=out/${1#*/}.xml2; mkdir -vp $(dirname "$out") && ./testscript.sh > "$out"' - {} \;

Finally++, as you properly quoted $out in $(dirname "$out"), you should do the same when redirecting to it: > "$out"