# Filter shell script to find lines that contain all specified patterns

I wrote a script that does the following:

1. Run another script on the system
2. Filter the output to find lines that contain ALL of the given patterns
3. Pipe the output to a second script on the system

I feel like the way I did it is a dirty hack. I use a control character to join the arguments, and then replace them with / && / for awk. Is there a better way? Also, this script is probably vulnerable to injection; I don't really need to worry about hostile attackers being able to manipulate the input to awk but I do need to worry about typos screwing up the regular expression.

#!/bin/bash

function join {
local IFS=$'\x02' echo "$*" | sed 's/'$'\x02''/\/ \&\& \//g' } /path/to/first/script | awk "/$(join $@)/" | /path/to/second/script  • A little late here, but did you consider firstScript | fgrep -f searchTargFile | secondScript. fgrep -f searchTargFile inFile searchs for string matches. I think more modern grep -EFf searchTargFile inFile can manage grokking reg-exps in the searchTargFile too if those are needed. Good luck. – shellter Mar 11 '16 at 17:36 ## 2 Answers join is the name of a common Unix command, so using that name for your function could create confusion. I don't recommend munging the patterns to dynamically write an awk program. As you pointed out, your technique is vulnerable to injection, resulting in execution of arbitrary awk code. Instead, I'd write a fixed awk script that does the job. #!/usr/bin/env awk -f BEGIN { # Treat command-line arguments as patterns rather than input filenames. for (i = 1; i < ARGC; i++) { patterns[i - 1] = ARGV[i]; } # Truncate argument list, so that awk always read from standard input. ARGC = 1; } { for (i in patterns) { if (!match($0, patterns[i])) {
next;
}
}
print;
}


To search for literal strings instead of regular expressions, use the index(haystack, needle) function instead of match(haystack, needle).

• And then do /path/to/script/one | 200successscript "$0" | /path/to/script/two ? – durron597 Jun 9 '14 at 18:39 • Basically yes, but "$@" not "$0" – 200_success Jun 10 '14 at 3:50 • Err, that was a typo :) but thanks – durron597 Jun 10 '14 at 13:14 Have you tried something simpler? I know that chaining greps together is not hugely scalable, but, it should accommodate as much as your commandline can. Including single-quotes as commandline expressions may be a problem though. #!/bin/bash source="/path/to/script1" target="/path/to/script2" greps="$source | "
for search in "$@" do greps="$greps egrep -e '$search' | "; done greps="$greps $target" #echo$greps
eval $greps  You would need to change the source/target to your purposes. Still, the amount of indirection/abstraction using this path is less than the complicated dynamic awk script. Out of interest, the following script will do what you want as well, and will support better quoting, etc.: #!/bin/bash script1 | perl -e 'INLINE: while($line = <STDIN>) {foreach $re (@ARGV) {$line =~ m/$re/ or next INLINE; } print$line; }' "$@" | script2  • I had something nearly identical to this but I was trying to loop the piping in advance (for search in "$@"; do; | grep $search"; done - syntax error) instead of building the command as a string and calling eval, which I couldn't get to work (because I forgot about eval, I'm not a script jockey 99 days out of 100) and made me think of the awk script. Can you clarify why this is better? (other than not using the dummy $'\x02' character) – durron597 Jun 9 '14 at 16:36
• My suggestion will be criticized because you could end up with dozens of grep commands running if you have lots of search inputs. If the inputs are few, then it will be OK. I was just thinking a perl option would be good too... Come chat in the 2nd monitor – rolfl Jun 9 '14 at 16:38
• @durron597 - if you want, there's a perl option too. – rolfl Jun 9 '14 at 17:20