Sign an EC2/S3 URL

Question

The other day I found myself trying to walk through the instructions for signing an S3 URL.

http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

Not as trivial as the documentation makes out.
I kept making mistakes and its hard to spot the mistakes as the intermediate values are just random hash string.

As a result I built an implementation using standard bash tools. This way I could compare the intermediates values I generated in my language of choice against this reference implementations intermediate values.

If anybody can spot any improvements that would be great.

#!/bin/bash

url=$1
key=$2
secret=$3
dateTime=${4-$(date -u +"%Y%m%dT%H%M%SZ")}

if [[ $# < 3  || $# > 4 ]]; then
    echo "Usage:"
    echo "    sign <url> <key> <secret> [<TimeStamp>]"
    echo
    echo "Note 1: TimeStamp: must be YYMMDD'T'hhmmss'Z'"
    echo "        eg 20170901T230559Z"
    echo "        YY => Year MM => Month DD => day hh => hour mm => minute ss => second"
    echo "        time is also in UTC"
    echo
    echo "Note 2: Currently does not support URL with Query or Fragment sections."
    echo
    echo "Note 3: Only support HTTP GET requests."
    echo "        All signing values are placed as query parameters"
    exit 1
fi

#
# The First part of <dateTime> before the T
date=$(echo ${dateTime} | sed -e 's/T.*//')

#
# Amazon URL Host are built up in sections.
#   http://<service>-<region>.<Amazon End Point><Path Including />
#
#   Strip out these parts from the url
host=$(echo ${url} | sed -e 's#https://\([^/]*\)/.*#\1#')
service=$(echo ${host} | sed -e 's#\([^-]\)-.*#\1#')
region=$(echo ${host} | sed -e 's#[^-]*-\([^\.]*\)\..*#\1#')
path=$(echo ${url} | sed -e 's#https://[^/]*\(/.*\)#\1#')

#
# URL expires in 1 hour after this it is useless.
expires=3600

#
# Build the conical request
cr=$(cat - <<CanonicalRequest
GET
${path}
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=${key}%2F${date}%2F${region}%2F${service}%2Faws4_request&X-Amz-Date=${dateTime}&X-Amz-Expires=${expires}&X-Amz-SignedHeaders=host
host:${host}

host
UNSIGNED-PAYLOAD
CanonicalRequest)

#
# Hash the conical request
HashedCanonicalRequest=$(echo -n "${cr}" | openssl dgst -sha256)

#
# Build the String to sign.
ss=$(cat - <<StringToSign
AWS4-HMAC-SHA256
${dateTime}
${date}/${region}/${service}/aws4_request
${HashedCanonicalRequest}
StringToSign)

#
# Calculate the signature
kDate=$(echo -n ${date}        | openssl dgst -sha256 -binary -hmac "AWS4${secret}")
kRegn=$(echo -n ${region}      | openssl dgst -sha256 -binary -hmac "${kDate}")
kServ=$(echo -n ${service}     | openssl dgst -sha256 -binary -hmac "${kRegn}")
kSign=$(echo -n "aws4_request" | openssl dgst -sha256 -binary -hmac "${kServ}")
signature=$(echo -n "${ss}"    | openssl dgst -sha256 -hmac "${kSign}")


#
# Dump intermediate values to compare against language specific implementation.
echo "Intermediat Values"
echo "url:                      ${url}"
echo "key:                      ${key}"
echo "secret:                   ${secret}"
echo "dateTime:                 ${dateTime}"
echo "date:                     ${date}"
echo "host:                     ${host}"
echo "path:                     ${path}"
echo "service:                  ${service}"
echo "region:                   ${region}"
echo "expires:                  ${expires}"
echo "HashedCanonicalRequest:   ${HashedCanonicalRequest}"
kDateH=$(echo -n ${date}        | openssl dgst -sha256 -hmac "AWS4${secret}")
kRegnH=$(echo -n ${region}      | openssl dgst -sha256 -hmac "${kDate}")
kServH=$(echo -n ${service}     | openssl dgst -sha256 -hmac "${kRegn}")
kSignH=$(echo -n "aws4_request" | openssl dgst -sha256 -hmac "${kServ}")
echo "kDate:                    ${kDateH}"
echo "kRegn:                    ${kRegnH}"
echo "kServ:                    ${kServH}"
echo "kSign:                    ${kSignH}"
echo "signature:                ${signature}"

echo
echo "Cononical Request:"
echo "=================="
echo "${cr}"
echo
echo "Signing String:"
echo "==============="
echo "${ss}"
echo
echo "Signed URL:"
echo "==========="
echo "${url}?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=${key}%2F${date}%2F${region}%2F${service}%2Faws4_request&X-Amz-Date=${dateTime}&X-Amz-Expires=${expires}&X-Amz-SignedHeaders=host&X-Amz-Signature=${signature}"

Answer 1

Your shebang works fine, but this idiom is more portable: #! /usr/bin/env bash

You wrote TimeStamp: must be YYMMDD..., but it should say YYYY. Consider deleting the next fairly obvious line that says YY, or put YYYY there, as well.

Please rephrase the comment like this: # Amazon Hosting URLs are built up in sections.

The next comment line is extremely helpful. The regexes are a bit twisty. Consider having bash call some scripting language that uses a single regex which parses out the four sections. Alternatively, since you already have a concrete value, consider e.g. putting $service at the beginning of the region regex.

Typo: conical (twice, and later Cononical)

host:${host}

I usually capitalize "Host:" headers, but I suppose that doesn't matter. Do put a blank after the colon, please.

Typo: Intermediat

Wow! Signing really is a bit of a pain, lots of inputs.

Looks good. Thanks for sharing.

Answer 2

For simple text transformations, if simple parameter expansion can handle it, it's good to use it, rather calling full-blown processes like sed.

For example:

date=${dateTime%%T*}

host=${url#*//}
host=${host%%/*}

service=${host%%-*}

region=${host#*-}
region=${host%%.*}

And so on.

---

Instead of a wall of echo like in the second half of the script, I would use a here-document.