Your "needle" for it can be the user's string prepended with '\n'. (I guess that makes the first line of the file a special case, so memcmp that.)

Your "needle" for it can be the user's string prepended with '\n'. I guess that makes the first line of the file a special case, so memcmp that. Unless you can read into a buffer that starts with a fake newline so your search sees a newline before the first line of the file. For large files you might want to mmap or at least page-align your read buffer, but for small files this could be a good trick. So e.g. alignas(4096) static char buf[8192]; (or thread_local?) and buf[0] = '\n';. And for each small file, read(fd, buf+1, sizeof(buf)-1);.

Unfortunately open doesn't tell you the file length, and fstat is an extra system call, so maybe best to just start with a smallish read before going to mmap if that doesn't reach EOF.

I'm not sure mmap is the most efficient way to read small files, especially without MAP_POPULATE; a PGN tends to be less than 4KiB so not even one full page, so you're paying one round-trip to the kernel for an mmap system call, then another for a page fault when you read the page. With <1 page of file data, that page-fault cost doesn't even get amortized by fault-ahead / fault-around to wire up nearby pages into the page table like when reading through a big file.

Letting a read system call copy into a per-thread 8K buffer you reuse should be good for this use-case; I don't think read even needs to invalidate any TLB entries when the destination page has already been touched so it doesn't need to allocate a freshly zeroed page (like it would if reading into a large buffer you just allocated with mmap(MAP_ANONYMOUS) directly or via a large malloc / new.)

Small-file I/O overhead is definitely a challenge here. io_uring might be useful to submit more system calls to the kernel with a single request, although I haven't used it myself. (man page)

I notice your version has higher sys time than grep. That might be from page-fault handlers, and from not buffering you output I/O (one writev system call per input file, which for your data set is one writev per output line unless you any big files with many PGNs concatenated.)

Obviously it's a tradeoff between development time vs. performance, and specializing your tool more for this use-case vs. making it generally useful. Perhaps if each thread had its own C++ output stream open on STDOUT_FD, you could get buffering that way. But OpenMP for threading makes it harder to get fancy, I think.

I'm not sure mmap is the most efficient way to read small files, especially without MAP_POPULATE; a PGN tends to be less than 4KiB so not even one full page, so you're paying one round-trip to the kernel for an mmap system call, then another for a page fault when you read the page. With <1 page of file data, that page-fault cost doesn't even get amortized by fault-ahead / fault-around to wire up nearby pages into the page table like when reading through a big file.

Letting a read system call copy into a per-thread 8K buffer you reuse should be good for this use-case; I don't think read even needs to invalidate any TLB entries when the destination page has already been touched so it doesn't need to allocate a freshly zeroed page (like it would if reading into a large buffer you just allocated with mmap(MAP_ANONYMOUS) directly or via a large malloc / new.)

Small-file I/O overhead is definitely a challenge here. io_uring might be useful to submit more system calls to the kernel with a single request, although I haven't used it myself. (man page)

I notice your version has higher sys time than grep. That might be from page-fault handlers, and from not buffering you output I/O (one writev system call per input file, which for your data set is one writev per output line unless you any big files with many PGNs concatenated.)

Obviously it's a tradeoff between development time vs. performance, and specializing your tool more for this use-case vs. making it generally useful. Perhaps if each thread had its own C++ output stream open on STDOUT_FD, you could get buffering that way. But OpenMP for threading makes it harder to get fancy, I think.

Use libc search functions which are hand-written in asm

memmem would be a good choice for scanning for the user-specified string inside a known-length buffer, instead of doing a bunch of std::string and std::string_view line-at-a-time work.

Or strstr if you want to stick to standard functions; memmem is available in GNU and *BSD systems, and the MUSL libc, and works with explicit-length strings and buffers like your program does.

Your "needle" for it can be the user's string prepended with '\n'. (I guess that makes the first line of the file a special case, so memcmp that.)

These search functions are typically heavily optimized in good libc implementations, usually with hand-written SIMD asm optimized for the system's CPU ISA and extensions. (e.g. SSE4 or AVX2 on x86-64.) They may have more startup overhead than memchr (which unlike strchr can easily check that it's not too close to the end of the string to do a full SIMD vector load right away).

You expect that a PGN will have one [Result line per maybe 1KiB or so, but often not the first couple lines so stopping at each newline is more work than ideal. And your code doesn't stop at the first match in a file, allowing for big files of concatenated PGNs where memmem (or strstr) can shine, scanning through all the text between [Result lines fast. You do want memchr to find the ending newline for lines you found.