I) Introduction
Just when we think we’ve mastered AWS Lambda, we discover hidden features and capabilities that deepen our understanding and enhance our applications. One such powerful feature is Lambda extensions.
We’ll explore in this article what lambda extension is and how it can power up our Serverless applications by analyzing a real use case.
II) What is an Lambda extension?
How AWS Lambda Operates: A Brief Overview
AWS Lambda functions operate within an execution environment that includes various processes. These processes are responsible for executing the actual function code. Additionally, there are API Endpoints that act as communication bridges between Lambda’s compute resources and the AWS platform itself.
When you trigger a Lambda function, the Lambda service makes an API call to the Runtime API. The Runtime API, in turn, invokes the specific runtime within the execution environment’s processes to run the function code. After the function completes its execution, the result is passed back through the Runtime API to the Lambda service, which then returns the output of the function to you. This entire process illustrates the lifecycle of a Lambda function invocation.
Understanding AWS Lambda Extensions
Now that we have a basic understanding of how Lambda works, let’s explore AWS Lambda extensions. A Lambda extension is essentially a piece of code that runs independently alongside your Lambda function’s code, within the same execution environment. You can think of it as an additional layer that coexists with the Lambda handler, sharing the same memory, timeout, and temporary storage allocated to the function.
Key Characteristics of Lambda Extensions
Although Lambda extensions share the same resources as the function (memory and timeout), they have a distinct lifecycle. Extensions can start running before the Lambda function is invoked and can continue running even after the function completes its execution. Moreover, extensions have the ability to hook into different lifecycle events within the execution environment, such as initialization, invocation, and shutdown. This allows developers to perform additional tasks (like launching processes) when the Lambda function finishes executing.
One of the unique advantages of Lambda extensions is their flexibility. They are not constrained to the same programming language as the function handler, which means you can develop extensions in a different language if needed. This is because extensions run as independent processes, which allows more customization and control over execution.
Lambda Extensions for Custom Log Destinations
One powerful use case for Lambda extensions is custom log handling. In traditional setups, sending Lambda logs to a specific destination (other than CloudWatch) required configuring a CloudWatch Log Group subscription, which often involved creating another Lambda function to forward the logs.
With Lambda extensions, you no longer need this extra step. Extensions can now receive logs directly from within the Lambda environment and send them to any destination of your choosing, providing much greater flexibility. For example, you can set up an extension that forwards logs to an S3 bucket or any other external service without needing to rely on CloudWatch as an intermediary.
Practical Example: Sending Logs to an S3 Bucket
In this scenario, I’ll show you how to use a Lambda extension to direct logs to an S3 bucket. This setup eliminates the need for additional configurations like CloudWatch Log Group subscriptions. Instead, the extension itself will handle the logs and transmit them to your S3 bucket. Throughout the rest of this guide, I will walk you through the steps for implementing a Lambda extension that accomplishes this, and you can find all the necessary code in the linked repository.
How to create an extension ?
To add an extension, you only need to add the layer to your lambda. The extension process will be spawn automatically. Lambda looks for extensions in the /opt/extensions/ directory, interprets each file as an executable bootstrap for launching the extension, and starts all extensions in parallel.
Lambda extension code for custom destination
Taking a look on the repo, we can see that the code has different components:
I) Lambda extension API
First step is to register your process as an extension to be able to listen to the different events that are happening in your lambda. For this we need to use the lambda extension API. As shown in the diagram down below.
