4
\$\begingroup\$

The other day I found myself trying to walk through the instructions for signing an S3 URL.

http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

Not as trivial as the documentation makes out.
I kept making mistakes and its hard to spot the mistakes as the intermediate values are just random hash string.

As a result I built an implementation using standard bash tools. This way I could compare the intermediates values I generated in my language of choice against this reference implementations intermediate values.

If anybody can spot any improvements that would be great.

#!/bin/bash

url=$1
key=$2
secret=$3
dateTime=${4-$(date -u +"%Y%m%dT%H%M%SZ")}

if [[ $# < 3  || $# > 4 ]]; then
    echo "Usage:"
    echo "    sign <url> <key> <secret> [<TimeStamp>]"
    echo
    echo "Note 1: TimeStamp: must be YYMMDD'T'hhmmss'Z'"
    echo "        eg 20170901T230559Z"
    echo "        YY => Year MM => Month DD => day hh => hour mm => minute ss => second"
    echo "        time is also in UTC"
    echo
    echo "Note 2: Currently does not support URL with Query or Fragment sections."
    echo
    echo "Note 3: Only support HTTP GET requests."
    echo "        All signing values are placed as query parameters"
    exit 1
fi

#
# The First part of <dateTime> before the T
date=$(echo ${dateTime} | sed -e 's/T.*//')

#
# Amazon URL Host are built up in sections.
#   http://<service>-<region>.<Amazon End Point><Path Including />
#
#   Strip out these parts from the url
host=$(echo ${url} | sed -e 's#https://\([^/]*\)/.*#\1#')
service=$(echo ${host} | sed -e 's#\([^-]\)-.*#\1#')
region=$(echo ${host} | sed -e 's#[^-]*-\([^\.]*\)\..*#\1#')
path=$(echo ${url} | sed -e 's#https://[^/]*\(/.*\)#\1#')

#
# URL expires in 1 hour after this it is useless.
expires=3600

#
# Build the conical request
cr=$(cat - <<CanonicalRequest
GET
${path}
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=${key}%2F${date}%2F${region}%2F${service}%2Faws4_request&X-Amz-Date=${dateTime}&X-Amz-Expires=${expires}&X-Amz-SignedHeaders=host
host:${host}

host
UNSIGNED-PAYLOAD
CanonicalRequest)

#
# Hash the conical request
HashedCanonicalRequest=$(echo -n "${cr}" | openssl dgst -sha256)

#
# Build the String to sign.
ss=$(cat - <<StringToSign
AWS4-HMAC-SHA256
${dateTime}
${date}/${region}/${service}/aws4_request
${HashedCanonicalRequest}
StringToSign)

#
# Calculate the signature
kDate=$(echo -n ${date}        | openssl dgst -sha256 -binary -hmac "AWS4${secret}")
kRegn=$(echo -n ${region}      | openssl dgst -sha256 -binary -hmac "${kDate}")
kServ=$(echo -n ${service}     | openssl dgst -sha256 -binary -hmac "${kRegn}")
kSign=$(echo -n "aws4_request" | openssl dgst -sha256 -binary -hmac "${kServ}")
signature=$(echo -n "${ss}"    | openssl dgst -sha256 -hmac "${kSign}")


#
# Dump intermediate values to compare against language specific implementation.
echo "Intermediat Values"
echo "url:                      ${url}"
echo "key:                      ${key}"
echo "secret:                   ${secret}"
echo "dateTime:                 ${dateTime}"
echo "date:                     ${date}"
echo "host:                     ${host}"
echo "path:                     ${path}"
echo "service:                  ${service}"
echo "region:                   ${region}"
echo "expires:                  ${expires}"
echo "HashedCanonicalRequest:   ${HashedCanonicalRequest}"
kDateH=$(echo -n ${date}        | openssl dgst -sha256 -hmac "AWS4${secret}")
kRegnH=$(echo -n ${region}      | openssl dgst -sha256 -hmac "${kDate}")
kServH=$(echo -n ${service}     | openssl dgst -sha256 -hmac "${kRegn}")
kSignH=$(echo -n "aws4_request" | openssl dgst -sha256 -hmac "${kServ}")
echo "kDate:                    ${kDateH}"
echo "kRegn:                    ${kRegnH}"
echo "kServ:                    ${kServH}"
echo "kSign:                    ${kSignH}"
echo "signature:                ${signature}"

echo
echo "Cononical Request:"
echo "=================="
echo "${cr}"
echo
echo "Signing String:"
echo "==============="
echo "${ss}"
echo
echo "Signed URL:"
echo "==========="
echo "${url}?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=${key}%2F${date}%2F${region}%2F${service}%2Faws4_request&X-Amz-Date=${dateTime}&X-Amz-Expires=${expires}&X-Amz-SignedHeaders=host&X-Amz-Signature=${signature}"
\$\endgroup\$
2

2 Answers 2

2
\$\begingroup\$

Your shebang works fine, but this idiom is more portable: #! /usr/bin/env bash

You wrote TimeStamp: must be YYMMDD..., but it should say YYYY. Consider deleting the next fairly obvious line that says YY, or put YYYY there, as well.

Please rephrase the comment like this: # Amazon Hosting URLs are built up in sections.

The next comment line is extremely helpful. The regexes are a bit twisty. Consider having bash call some scripting language that uses a single regex which parses out the four sections. Alternatively, since you already have a concrete value, consider e.g. putting $service at the beginning of the region regex.

Typo: conical (twice, and later Cononical)

host:${host}

I usually capitalize "Host:" headers, but I suppose that doesn't matter. Do put a blank after the colon, please.

Typo: Intermediat

Wow! Signing really is a bit of a pain, lots of inputs.

Looks good. Thanks for sharing.

\$\endgroup\$
1
  • \$\begingroup\$ The host inside the "CanonicalRequest" is deliberately lowercase. This string is only used to generate the hash so consistency is important. Yes in an actual header I would also capitalize the 'H'. Thanks for all the other comments. \$\endgroup\$ Sep 4, 2017 at 2:59
1
\$\begingroup\$

For simple text transformations, if simple parameter expansion can handle it, it's good to use it, rather calling full-blown processes like sed.

For example:

date=${dateTime%%T*}

host=${url#*//}
host=${host%%/*}

service=${host%%-*}

region=${host#*-}
region=${host%%.*}

And so on.


Instead of a wall of echo like in the second half of the script, I would use a here-document.

\$\endgroup\$
2
  • \$\begingroup\$ Unfortunately there here-string is not identical. The trailing '\n' on the last line is retained and thus generates a different hash. \$\endgroup\$ Sep 6, 2017 at 6:49
  • \$\begingroup\$ @LokiAstari crap, I see it now, you're right, good to know :-/ \$\endgroup\$
    – janos
    Sep 6, 2017 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.