2
\$\begingroup\$
#!/bin/bash
set -ue

# use AWS CLI to launch a cluster
create_cluster_response=$(aws emr create-cluster \
  --release-label emr-4.3.0 \
  --region us-east-1 \
  --etc) # many more arguments; see http://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html

# use jq to parse and display result
cluster_id=$(echo "${create_cluster_response}" | jq -r '.ClusterId')
echo "ClusterId=${cluster_id}"
echo "AWS Management Console URL: https://console.aws.amazon.com/elasticmapreduce/home?region=us-east-1#cluster-details:${cluster_id}"

# block until the EMR cluster terminates
previous_state=''
while true; do
  json=$(aws emr describe-cluster --cluster-id ${cluster_id}) # see http://docs.aws.amazon.com/cli/latest/reference/emr/describe-cluster.html
  state=$(echo "${json}" | jq -r '.Cluster.Status.State')
  # echo state changes
  if [[ ${state} != ${previous_state} ]]; then
    now=$(date -u +"%Y-%m-%dT%H:%M:%SZ") # format date as ISO 8601 UTC
    message=$(echo "${json}" | jq -r '.Cluster.Status.StateChangeReason.Message')
    echo "${now} ${state} ${message}"
    previous_state="${state}"
  fi
  # terminate this script when the cluster terminates
  if [[ ${state} == TERMINATED ]]; then
    echo "EMR JOB SUCCESS"
    exit 0
  fi
  if [[ ${state} == TERMINATED_WITH_ERRORS ]]; then
    echo "EMR JOB FAILURE"
    exit 1
  fi
  sleep 10
done

I am very new to programming bash and will appreciate any and all feedback.

\$\endgroup\$
3
\$\begingroup\$

you can use emr wait instead of describe-cluster

aws emr wait cluster-running --cluster-id <value> --region <value>

http://docs.aws.amazon.com/cli/latest/reference/emr/wait/cluster-running.html

\$\endgroup\$
  • \$\begingroup\$ Well, that would certainly have made things easier! Thank you! \$\endgroup\$ – Landon Kuhn Aug 16 '16 at 16:07
  • \$\begingroup\$ The problem with the "aws emr wait" command is that it's not possible to specify a timeout for it. In real usage, you typically don't want the wait to go on for ever if the cluster doesn't terminate, so the submitted script is still useful. \$\endgroup\$ – Jan Mar 20 '17 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.