1
\$\begingroup\$

Design

My desired outcome is as follows:

  1. S3 event triggers Lambda Function startTextractStateMachine

  2. Lambda Function startTextractStateMachine kicks off State Machine textractStepFunc execution (AWS Step Functions)

  3. State Machine textractStepFunc starts with Lambda Function callTextract; calls Textract async command to start process

  4. Output of Textract command is attached to SQS queue TextractSQS

  5. SQS queue output triggers Lambda Function getTextractOutput

  6. Lambda Function getTextractOutput publishes message to SNS Topic

  7. SNS Topic triggers final Lambda Function parseTextractOutput via subscription

  8. Done

Implementation

I made this system work without any AWS Step Functions initially, but realized that I needed more control over retry/failure of the individual Lambda Functions (especially callTextract), so I've tried to move the system to AWS Step Functions and the Serverless Framework at the same time.

I've come to realize it was harder than I thought it would be. I keep running into roadblocks with the Serverless Framework.

I used several guides, blog posts and Github issues (here, here and here) to reach this point, but am permanently stuck.

Serverless Setup

I've left out much of my Design ideas and simply need to have the S3 event trigger when a file is added to the Bucket.

Here is my configuration file:

# serverless.yml

service: textract-service

provider:
  name: aws
  runtime: python3.7
  timeout: 10
  region: us-east-1
  environment:
    STATE_MACHINE_ARN: ${self:resources.Outputs.TextractStepFunctions.Value}
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "s3:PutObject"
        - "states:*"
      Resource:
        Fn::Join:
          - ""
          - - "<the-chumiest-bucket's ARN> or *"
            - ${self:resources.Outputs.TextractStepFunctions.Value}

plugins:
  - serverless-plugin-existing-s3
  - serverless-step-functions
  - serverless-pseudo-parameters

layers:
  boto3Layer:
    package:
      artifact: boto3_layer.zip
    allowedAccounts:
      - "*"

functions:
  startTextractStateMachine:
    handler: src/start_textract_state_machine.lambda_handler
    role: arn:...lambda-s3-role
    layers:
      - {Ref: Boto3LayerLambdaLayer}
    events:
      - existingS3:
          bucket: the-chumiest-bucket
          events:
            - s3:ObjectCreated:*
          rules:
            - prefix: input1/
            - suffix: .pdf
      - existingS3:
          bucket: the-chumiest-bucket
          events:
            - s3:ObjectCreated:*
          rules:
            - prefix: input2/
            - suffix: .pdf
  callTextract:
    handler: src/call_textract.lambda_handler
    role: arn:...lambda-s3-role
    layers:
      - {Ref: Boto3LayerLambdaLayer}
  getTextractOutput:
    handler: src/get_textract_output.lambda_handler
    role: arn:...lambda-s3-role
    layers:
      - {Ref: Boto3LayerLambdaLayer}
  parseTextractOutput:
    handler: src/parse_textract_output.lambda_handler
    role: arn:...lambda-s3-role
    layers:
      - {Ref: Boto3LayerLambdaLayer}

stepFunctions:
  stateMachines:
    textractStepFunc:
      name: TextractStepFunctions
      definition:
        StartAt: StartTextractStateMachine
        States:
          StartTextractStateMachine:
            Type: Task
            Resource: "arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:#{AWS::StackName}-startTextractStateMachine"
            Next: CallTextract
          CallTextract:
            Type: Task
            Resource: "arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:#{AWS::StackName}-callTextract"
            End: true

resources:
  Outputs:
    TextractStepFunctions:
      Description: The ARN of the state machine
      Value:
        Ref: TextractStepFunctions

How I deploy it:

sls deploy -v && sls s3deploy
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.