Trigger a AWS Lambda with a SQS message with Python

Introduction

In the context of microservices, the communication of them must be fast and realiabe. This communication can be through messages and a tool for this communication can be SQS (Simple Queue Service), a messaging service by AWS.

This project will be a example of a system that can receive a SQS message from a client, process it, and give the information to another client. As a example of this process, we’ll create a microservice in a AWS Lambda that will act when receive a SQS message. It’ll write the message in a .txt file and return all the messages when receives a HTTP call to a certain endpoint. We’ll be using AWS Lambda because it is a serverless framework, in which the idea is you don’t have to worry about servers.

Python’s library FastAPI will be used to create the API, Docker to containerize it and Poetry will be the library manager. As always, the code of the project is on my Github.

Building the API

We’ll be using FastAPI as the backend tool for this project. The package has a simple syntax and makes easy to create APIs. To create a health check route:

Calling the file main.py, Uvicorn can run this API by running the following code:

Inside the main.py file, we’ll create another route called “/records”. When the API receives a SQS message, it writes it down on a .txt file called “log.txt”. The new route retrives all of the messages in the file. The full “main.py” will be

Creating the SQS handler

To create the SQS handler, we first need to undestand how our Lambda will receive the SQS message. It will come in a JSON like this

There’s a few details here. First of all, the eventSourceARN, which is the address of the SQS Queue. The second is the body of the message. For this example, we want to receive messages in the format of

A parser must be created to extract the data and put it in a object called Message.

The ideia is to create a object which will be receiving a ARN_SOURCE variable, the ARN of the SQS queue. It will only try to parse the messages from that queue and will try to parse based on the Message Class in the function parse_messages. If it can, will return an object of the Message Class. Replace the ARN_SOURCE variable with the one will create in a moment. Our handler will be then

It will try to parse the messages and write it down to a log.txt file. The full sqs_handler.py will be

That’s it for the API. We need to create the lambda_handler.py now

Everytime it receives a event, it will check if it’s a HTTP request (which Mangum will handle) or if it’s a SQS message (which our handler sqs_handler.py will handle). The final structure of the project will be

final structure of the projec

To handle our dependencies, we can use poetry

We can use Docker to containerize the project

Creating the ECR and the SQS Queue

To deploy the package to a AWS Lambda, we first need to create a repository and upload the image. We can also create the SQS object here. Export your AWS Account Id to a enviroment variable called AWSACCOUNTID and run:

Creating the Lambda

We can create the Lambda in a lot a different ways, from the CLI or using some Infra as a Code tool (like Terraform). I strongly recommend the use of Terraform, mainly because you can replicate the same infrastructure in many different enviroments. For this tutorial we’ll be using the management console, mainly because of the simplicity.

A Lambda function can be created on the AWS Lambda page:

final structure of the projec

Our Lambda will use the container created before:

final structure of the projec

We need to give access to the SQS in the role of the Lambda. We can do this by adding: final structure of the projec

An alternative is to add full access to the SQS services on the role of the Lambda. By finding the role of the Lambda, we need to attach the following policy:

final structure of the projec