Async API with API Gateway and SNS

Door Dick Eimers op 15 mrt 2018

When discussing the implementation of a asynchronous API on AWS the first solution that came to mind (at least to mine) was to use API Gateway, two Lambda’s and a queue (or actually a SNS topic).

‘2-lambda’ solution

The first Lambda function would be configured to receive the request from the API Gateway and publish the body of the request as a message to the topic, while the second Lambda would subscribe to the topic and get notified whenever a new message has arrived. The first lambda would ensure that the client received a friendly 202 “Accepted” to indicate that we are going to work on it, eventually.

Since Lambda is like a Swiss Army knife that solution will definitely work, but another possibility was proposed that involves using a feature from API Gateway that allows it to act as a ‘service proxy’ in front of other AWS services, such as SNS.

‘service proxy’ solution

This option dispenses the need for one of the Lambda function and it simplifies our solution a bit. Well, at least we end up with one component less in our solution, but it turns out that these API Gateway integration configurations are difficult to get right and the documentation is lacking. A typical example of something which is easy once you know how it’s done.

So without further ado, I’ll document it here 😉 

We use CloudFormation with the AWS Serverless Application Model (SAM) extension to allow for more concise templates.

First we declare the SNS topic.

  ExampleTopic:
    Type: "AWS::SNS::Topic"
    Properties:
      TopicName: !Sub "${AWS::StackName}-example-topic"

Next, we configure an API resource (/example) accepting POST operations that responds with a 202 Accepted. And we use the API Gateway extensions to Swagger to create AWS specific integrations.

The uri is a special ARN that expresses that we want to go from ‘apigateway’ to ‘sns’  to ‘action/Publish’. This is what API Gateway needs to select the appropriate endpoint for what turns out to be a HTTP-based integration.

The SNS endpoint expects a Message and TopicArn as query parameters, so we map those accordingly. Note that this requires moving the payload from our request body in the original request to this query parameter called ‘Message’ in the request for the SNS endpoint.

  ExampleAPI:
    Type: "AWS::Serverless::Api"
    Properties:
      StageName: "prod"
      DefinitionBody:
        swagger: "2.0"
        info:
          title: !Sub "${AWS::StackName}-api"
        paths:
          /example-path:
            post:
              responses:
                "202":
                  description: Accepted
              x-amazon-apigateway-integration:
                type: "aws"
                httpMethod: "POST"
                uri: !Sub "arn:aws:apigateway:${AWS::Region}:sns:action/Publish"
                credentials: !GetAtt ExampleTopicAPIRole.Arn
                requestParameters:
                  integration.request.querystring.Message: "method.request.body"
                  integration.request.querystring.TopicArn: !Sub "'${ExampleTopic}'"
                responses:
                  default:
                    statusCode: 202

The ‘credentials’ are provided using the role below. This allows API Gateway to publish to the topic we created earlier.

  ExampleTopicAPIRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "apigateway.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Policies:
        - PolicyName: !Sub "${AWS::StackName}-example-topic-policy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action: "sns:Publish"
                Effect: "Allow"
                Resource: !Ref ExampleTopic
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs"

Finally, we configure the Lambda acting as our worker, which is declared to process events of type SNS. This boils down to the Lambda being a subscriber to the SNS topic.

  ExampleWorkerLambda:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: index.handler
      Runtime: nodejs6.10
      CodeUri: ./app
      Events:
        SNSMessage:
          Type: SNS
          Properties:
            Topic: !Ref ExampleTopic 

A working example (with deployment scripts and the Lamda source code) can be found here.