Finding hidden field values in a web page before submitting a login form

Question

In order to submit a login form using wget, I need to find the values for the hidden field that were added to the form recently. Grepping for the names gives the following:

$ grep csrf signin.php

<input type="hidden" name="csrf1" value="56a0ecec3f15c" />
<input type="hidden" name="csrf2" value="4a8cb442fc85a745c9cf92edd44d71cc3ac7225e99acf57d67276ea0481f9ae8" />
<input type="hidden" name="csrf1" value="56a0ecec3f236" />
<input type="hidden" name="csrf2" value="292f226a5048c3d75dcfc28c8e05df49089d55160ff37e5572ecb51e208a1e52" />
<input type="hidden" name="csrf1" value="56a0ecec3f399" />
<input type="hidden" name="csrf2" value="a99e22c2361505931a8e26f40d26de156b928b9d218fbe4f02a98f15e2364269" />

I need to extract the two middle values for csrf1 and csrf2, and concatenate them into something that looks like the following, all on one line:

csrf1=56a0ecec3f236&csrf2=292f226a5048c3d75dcfc28c8e05df49089d55160ff37e5572ecb51e208a1e52

What I came up with, using bash, is the following:

CSRF=`grep csrf signin.php | tail -4 | head -2 | awk -F "\"" '{print $4 "=" $6}'`
CSRF=`echo $CSRF | sed -e 's/ /\&/'`

How would you improve these two lines?

Answer 1

Whenever chaining awk together with grep, sed, head, tail or other text processor tools, consider the possibility of doing it all with just awk, as it's very powerful. Reducing the number of processes in the pipeline makes sense, and in some examples can drastically improve performance.

Here's one way to write the two lines of pipelines as a single line (expanded to multiple lines for readability):

awk -F '"' '
    /csrf1/ {csrf1 = $4 "=" $6}
    /csrf2/ {
      csrf2 = $4 "=" $6;
      if (count++ > 0) {
        print csrf1 "&" csrf2;
        exit; 
      }
    }' signup.php

That is:

• When you see a line matching "csrf1", save it
• When you see a line matching "csrf2", save it, and:
  • If this is the first time, increment the count and do nothing else
  • If this is the second time, print csrf1 and csrf2 and stop processing

Note that the algorithm overwrites the first csrf1 and csrf2 values that were seen, and print only the second time.

I also used -F '"' instead of -F "\"" because it's easier to both write and read.

And if you want to save the result of this in a variable, then instead of:

CSRF=`awk ...`

Use the modern $(...) syntax:

CSRF=$(awk ...)