I have a small shell script that uses
sed to take an input file of URLs with hand-written notes around them and strips the notes and puts each URL on its own line. For example:
note: http://www.example.com/Beat-Poetry + ??? http://www.example.com/beat+poetry http://www.example.com/17th+cent. + http://www.example.com/17th+century + http://www.example.com/17th+c. http://www.example.com/18th+century https://www.example.com/C19th-C20th + http://www.example.com/19th-20th+century (note) note: http://www.example.com/18th+cent. note http://www.example.com/18th+century
Note: the URLs will always either have leading/trailing space or start or end the line.
http://www.example.com/Beat-Poetry http://www.example.com/beat+poetry http://www.example.com/17th+cent. http://www.example.com/17th+century http://www.example.com/17th+c. http://www.example.com/18th+century http://www.example.com/C19th-C20th http://www.example.com/19th-20th+century http://www.example.com/18th+cent. http://www.example.com/18th+century
I have this code, which does the job by adding some delimiters around each URL and removing stuff based on where the delimiters are found, but I am a newbie with this stuff, and it doesn't quite feel right. If nothing else, it's not robust enough to withstand potential usage of
é characters in the "notes".
#!/bin/bash # squash out all the extra text that isn't URL (notes to self) and put each URL on a new line # hackish steps to achieve this: # - change urls from http://url to áhttp://urlé # - put each one on a new line # - remove leading space/words # - remove trailing space/words # - change any https to http sed -re 's/(https?:[^ ]*)( |$)/á\1é /g' \ -e 's/é[^á]*á/\n/g' \ -e 's/(^[^á]*)(á[^é]*é)/\2/g' \ -e 's/é[^á]*$//' \ -e 's/https:/http:/g' $1 | tr -d 'áé\r' | sed -rn 's/(http:\/\/www.example.com\/.*)$/\1/p'
I assume that there is a more proper way to do this? (Again, the URLs will always have whitespace or
$ around them.) I'd appreciate any improvement suggestions. Thanks.