TL;DR
frawk -F ':' '/w:/{ white = $2 }
/b:/{ black = $2 }
/d:/{ draw = $2 }
END { print white + black + draw, white, black, draw }' \
<(rg --glob='*.pgn' --text --no-filename --count --line-buffered \
--crlf --no-unicode '^\[Result "1-0"\]$' |
frawk '{ s += $0 } END { print "w:" s }') \
<(rg --glob='*.pgn' --text --no-filename --count --line-buffered \
--crlf --no-unicode '^\[Result "0-1"\]$' |
frawk '{ s += $0 } END { print "b:" s }') \
<(rg --glob='*.pgn' --text --no-filename --count --line-buffered \
--crlf --no-unicode '^\[Result "1/2-1/2"\]$' |
frawk '{ s += $0 } END { print "d:" s }')
Finishes in 2.53 seconds versus OP's (20.05 seconds), jano's (19.71 seconds) and Adam's (6.48 seconds) solutions.
Potential Bugs
Your solution, while able to eliminate one piped command relative to Adam's,
introduces a new limitation: the output may consists of multiple intermediate
results rather than one final statistics.
From the xargs man page:
-s max-chars, --max-chars=max-chars
Use at most max-chars characters per command line, including the command
and initial-arguments and the terminating nulls at the ends of the
argument strings. The largest allowed value is system-dependent, and is
calculated as the argument length limit for exec, less the size of your
environment, less 2048 bytes of headroom. If this value is more than
128KiB, 128Kib is used as the default value; otherwise, the default
value is the maximum. 1KiB is 1024 bytes. xargs automatically adapts
to tighter constraints.
I've inadvertently hit the default maximum max-chars limit while testing.
You may force the issue by:
- creating a directory with a very long path name to hold the data set
and running your commands from there,
- increasing the size of your environment by populating it with random data:
export __=$(perl -e 'print "x" x $ARGV[0]' 100000)
- decreasing the maximum stack size temporarily by running
ulimit -S -s STACK_SIZE
where STACK_SIZE is some number lower than the default 8192, e.g. 256,
- letting the number of PGN files grow organically,
or some combination of the above.
I'm quite surprised that your attempt at parallel execution slowed you down.
If I adapt your changes to Adam's code,
it actually ranks among the fastest solutions I've tried.
Here's what you'd have come up with:
# op2.sh
n=$(nproc)
find . -type f -name '*.pgn' -print0 |
xargs --null -n $n --max-procs=$n mawk -F '[-"]' \
'BEGIN { a[1] = a[0] = a["1/2"] = 0 }
/Result/{ ++a[$2] }
END { print a[1] + a[0] + a["1/2"], a[1], a[0], a["1/2"] }' |
mawk '{ games += $1; white += $2; black += $3; draw += $4; }
END { print games, white, black, draw }'
Notice here I've added a BEGIN rule to initialize the array.
This is done to ensure the statistics will be correct:
if the collection of files fed to an invocation of awk is devoid of certain
outcomes (white wins, white loses, or a draw),
the slots counting the missing outcomes would output empty strings,
which when passed to the final aggregating AWK program
would result in misgrouping and tainting the counts.
You can see this effect come into play by changing the -n4 option of xargs
to -n1 or -n2 in Adam's solution reproduced below:
# adam.sh
find . -type f -name '*.pgn' -print0 |
xargs -0 -n4 -P4 mawk \
'/Result/{
split($0, a, "-");
res = substr(a[1], length(a[1]), 1);
if (res == 1) white++;
if (res == 0) black++;
if (res == 2) draw++;
}
END { print white + black + draw, white, black, draw }' |
mawk '{ games += $1; white += $2; black += $3; draw += $4; }
END { print games, white, black, draw }'
A fixed string search for Result, while fast and simple,
is a bit naive.
In a prior version of the file Britbase/197008bcf.pgn,
there is a comment in the movetext that includes both the string Result and a score 1-0:
1. e4 c6 2. d4 g6 3. Nc3 Bg7 4. Be3 d6 5. Qd2 Nd7 6. f3 Qa5 7. Bc4 b5 8. Bb3 b4
9. Nd1 Ba6 10. Ne2 Ngf6 11. c3 c5 12. cxb4 cxb4 13. Nf2 O-O 14. O-O Rfc8 15.
Rfc1 Nb6 16. Nf4 Nc4 17. Bxc4 Bxc4 18. N4d3 Rab8 19. a3 Qb5 20. Nxb4 a5 21.
Nbd3 Nd7 22. b4 a4 23. Nb2 Bb3 24. Rxc8+ Rxc8 25. Rc1 Rxc1+ 26. Qxc1 Qe2 27.
Nfd1 {Result given as 1-0 in the tournament bulletin but it is clear from
published results and crosstables, as well as the position on the board (where
Black has mate in one) that the true result was 0-1 - JS, BritBase} 0-1
This is a rare occurrence, indeed, but I'd personally perfer the more explicit search pattern $'^\[Result "(1-0|0-1|1/2-1/2)"\]\r*$' that adheres to the PGN export format described in the PGN standard document.
It is still possible to have false negatives, but trying to match the versatile PGN import format is likely to hurt performance and needlessly increase complexity.
Better Tools
Now onto my first observation at possible performance improvement ideas.
I noticed the result lines we're interested in make up only a tiny fraction
(less than 5%) of the contents in those PGN files,
so I figured that filtering the input files for the result lines first via the
expert tool grep might be more efficient than feeding them directly to awk,
which does unnecessary bookkeeping that could accumulate considerable overhead.
This was actually Adam's first optimization idea, too.
I just spun it off by subsequently applying his last performance idea,
finding faster replacements for the tools, which got me to ripgrep.
ripgrep is an amazing piece of software that does parallel searches,
which was Adam's second optimization idea, and I'm getting it for free!
Then I wondered, after all these years, is mawk still the fastest AWK?
After 20 years of no activity, Mike Brennan, the author of mawk,
released mawk 1.9.9.6, a beta release for mawk 2 based on mawk 1.3.3's code.
Sadly, it didn't outperform mawk 1.3.4 in my testing.
There is also frawk,
a fast and mostly compatible AWK implementation written in Rust.
It requires more steps to install, and in the end,
with the exception of one very simple case of summing numbers (one per line),
it runs slower in all backend, optimization and parallelization configurations.
After ample experimentation, I came up with two solutions that are blazingly fast on modern hardware:
# rg.sh
rg --glob='*.pgn' --text --no-filename --no-line-number --crlf --no-unicode \
'^\[Result "(1-0|0-1|1/2-1/2)"\]$' |
mawk -F '[-"]' 'BEGIN { a[1] = a[0] = a["1/2"] = 0 }
{ ++a[$2] }
END { print a[1] + a[0] + a["1/2"], a[1], a[0], a["1/2"] }'
# rg3.sh
frawk -F ':' '/w:/{ white = $2 }
/b:/{ black = $2 }
/d:/{ draw = $2 }
END { print white + black + draw, white, black, draw }' \
<(rg --glob='*.pgn' --text --no-filename --count --line-buffered \
--crlf --no-unicode '^\[Result "1-0"\]$' |
frawk '{ s += $0 } END { print "w:" s }') \
<(rg --glob='*.pgn' --text --no-filename --count --line-buffered \
--crlf --no-unicode '^\[Result "0-1"\]$' |
frawk '{ s += $0 } END { print "b:" s }') \
<(rg --glob='*.pgn' --text --no-filename --count --line-buffered \
--crlf --no-unicode '^\[Result "1/2-1/2"\]$' |
frawk '{ s += $0 } END { print "d:" s }')
Compression
Before I present my benchmark results,
I want to touch on one more idea that might improve performance,
especially in setups with slower disks: searching through compressed archives.
This is essentially trading disk seeking time with CPU time.
A reduction in file size and count
(from many smaller files into one big, compressed tar file)
means a reduction in disk IO,
which translates to faster execution time if it is the right trade-off.
We may download the data set on GitHub as a gzip file
and decompress it using pigz (parallel gzip):
tar --extract --use-compress-program='pigz' \
--file='./ChessData-master.tar.gz'
This runs in 45 seconds on my machine.
GZIP is not the most optimal lossless data compression file format,
neither is pigz the most parallel decompressor.
We could switch to Zstandard by recompressing the data set with pzstd:
tar --create --use-compress-program='pzstd' \
--file='./ChessData-master.tar.zst' ./ChessData-master
This runs in 30 seconds on my machine and produces a 3.5 GB zstd file,
which is 200 MB down from the 3.7 GB gzip source.
Note that pzstd can not fully benefit from parallel decompression
if the zstd file is compressed using zstd,
as zstd does not insert the additional markers needed for parallel decompression.
A solution to the problem statement utilizing pzstd is:
# pzstd.sh
tar --extract --to-stdout --use-compress-program='pzstd' \
--wildcards --wildcards-match-slash \
--file='./ChessData-master.tar.zst' '*.pgn' |
mawk -F '[-"]' 'BEGIN { a[1] = a[0] = a["1/2"] = 0 }
/Result/{ ++a[$2] }
END { print a[1] + a[0] + a["1/2"], a[1], a[0], a["1/2"] }'
The Benchmarks
Benchmark PC1 specs:
- CPU: 8
- RAM: 16 GB
- Disk: NVMe SSD
- OS: Fedora 40
Data sets:
Benchmark types:
Note: adam2.sh is adam.sh patched to utilize all available cores
(-n4 -P4 becomes -n$n -P$n where n=$(nproc).
I didn't test for the best possible value for -n
as that would be ad hoc micro-optimization,
so I just made it the same as the number for -P.
Let's examine the median running times of each contender in the benchmarks.
ChessData-master:
rg3.sh 2.53
rg.sh 3.25
op2.sh 3.67
adam2.sh 4.00
adam.sh 6.48
janos.sh 19.71
op.sh 20.05
pzstd.sh 25.88
ChessData-master, cached:
rg3.sh 0.91
rg.sh 1.69
op2.sh 2.04
adam2.sh 2.41
adam.sh 4.35
op.sh 12.52
janos.sh 13.11
pzstd.sh 21.54
ChessData-master, POSIX commands:
op2.sh 3.74
adam2.sh 3.94
adam.sh 6.54
grep.sh 14.09
janos.sh 19.49
op.sh 19.90
grep3.sh 25.02
pzstd.sh 55.96
LumbrasGigaBase:
rg3.sh 4.02
rg.sh 4.47
adam.sh 12.09
op2.sh 14.20
adam2.sh 15.82
op.sh 20.76
janos.sh 21.26
pzstd.sh 24.36
The clear winner is rg3.sh.
findcan just print matching paths as it encounters them. That would likely explain whyfindis faster. \$\endgroup\$