This is the script :
#!/bin/bash
set -euo pipefail
SOFT="$(basename $0)"
random=$(( ( RANDOM % 100000 ) ))
tempfolder="/tmp/$SOFT/$random"
mkdir -p "$tempfolder"
pushd "$tempfolder" > /dev/null
cleanup (){
cat wget.txt | sed 's/:$//' | url_matcher
popd > /dev/null
rm -rf -- ${tempfolder}
}
trap cleanup EXIT
site="$1"
wget -nd --spider --recursive --no-verbose --output-file=wget.txt "$site"
the script essentially creates a subfolder in /tmp/crawl and puts into it the result of wget, then feeds it to sed and url_matcher.
Initially I though of putting the cat instruction just after the wget one, but for long wget it didn't work.
The sed instruction removes colons at the end of lines, which is added by wget when said url isn't valid. (something along the lines of "http://...: Nothing there")
url_matcher is a c++ scanner made with flex, which recognises urls in a text fed with standard input and prints them to standard output, separated with new lines characters.
the scripts allows me to fetch all urls in a website and use that as a stream directly.
Example :
$ crawl www.example.com | sed 's/some_treatments//' > super_file