I have been experimenting with methods to read large text files into a Scheme program as a single string in a performant manner. Development has been in R7RS Scheme, specifically Chibi-Scheme.
After trying many approaches, some quite complex and involved, the best performing have been quite simple. Following are three examples of such along with some example usages and timing results.
(import (scheme base)
(scheme time))
;; Use with-input-from-file.
;; Return the contents of a text file as a single string,
;; #\newlines included.
(define (file->string1 path)
(let* ((start-time (current-second)))
(let ((lst (with-input-from-file path
(lambda ()
(let ((p (current-input-port)))
(let loop ((ch (read-char p))
(acc '()))
(if (eof-object? ch)
acc
(loop (read-char p) (cons ch acc)))))))))
(let ((result (list->string (reverse lst)))
(duration (- (current-second) start-time)))
(display "file->string1 complete in ")
(display duration)
(display " seconds.\n")
result))))
;; Use call-with-input-file.
;; Return the contents of a text file as a single string,
;; #\newlines included.
(define (file->string2 path)
(let ((start-time (current-second))
(lst (call-with-input-file path
(lambda (p)
(let loop ((ch (read-char p))
(acc '()))
(if (eof-object? ch)
acc
(loop (read-char p) (cons ch acc))))))))
(let ((result (list->string (reverse lst)))
(duration (- (current-second) start-time)))
(display "file->string2 complete in ")
(display duration)
(display " seconds.\n")
result)))
;; Use an output string to collect the data read.
;; Return the contents of a text file as a single string,
;; #\newlines included.
(define (file->string3 path)
(let* ((start-time (current-second))
(result (call-with-input-file path
(lambda (p)
(let ((out (open-output-string)))
(let loop ()
(cond
((eof-object? (peek-char p))
(get-output-string out))
(else
(write-char (read-char p) out)
(loop))))))))
(duration (- (current-second) start-time)))
(display "file->string3 complete in ")
(display duration)
(display " seconds.\n")
result))
;; Example usage and results loading "War and Peace" from the
;; Project Gutenberg.
;; https://www.gutenberg.org/cache/epub/2600/pg2600.txt
;;
;; ➜ david in schemacs on branch: (main) ! rlwrap chibi-scheme
;; > (load "file-ops.scm")
;; > (define wap1 (file->string1 "war-and-peace.txt"))
;; file->string1 complete in 0.6330661773681641 seconds.
;; > (define wap2 (file->string2 "war-and-peace.txt"))
;; file->string2 complete in 5.0067901611328125e-06 seconds.
;; > (define wap3 (file->string3 "war-and-peace.txt"))
;; file->string3 complete in 0.48236799240112305 seconds.
;; > (string-length wap1)
;; 3227709
;; > (string-length wap2)
;; 3227709
;; > (string-length wap3)
;; 3227709
;; > (string=? wap1 wap2 wap3)
;; #t
As a test case, I have used a copy of "War and Peace" obtained from Project Gutenberg as the file to read. It consists of 3,227,709 characters, including some preamble and final comments from Project Gutenberg.
All three procedures produce identical results, but the second consistently reports a much faster execution time. Timing values close to those above have been reported on repeated runs. Using a copy of "Moby Dick" as the file to read yields a similar timing relationship between the procedures.
I can't see any reason for the difference in timings for the second procedure. Are the results bogus for some reason?
Is one procedure preferable to the others for stylistic reasons?