Introduction
AWS Lambda is a powerful service that allows you to run code in response to events, such as user input or system events. In this blog post, we will show you how to log events in S3 using the AWS Lambda API.
AWS Cloudwatch is known as an expensive resource, one of its most expensive features is data ingestion (today it ranks at $0.50 per GB).
In this article, we are going to DEMO how to skip CloudWatch Data Ingestion and use S3.
By default, AWS Lambda will stream the logs directory into Cloudwatch, but using the Concept of Lambda Extensions, we can skip that step and instead directly subscribe to the log streams within the Lambda Execution environment.
The Logs API allows extensions to subscribe to three different log streams:
-
Function – Logs that the Lambda function generates and writes to stdout or stderr.
-
Extension – Logs that the extension code generates.
-
Platform – Logs that the runtime platform generates, which record events and errors related to invocations and extensions.
In this article, we are going to DEMO the Function logs.
DEMO
I will be using Python in this DEMO and it will have 2 components within the lambda space:
-
Lambda function
-
Lambda layer attached to the function
The reason for using a lambda layer is to provide flexibility to the main lambda function.
The code I am using is Terraform base and it’s stored in GitHub here
The terraform code will create the following resources:
-
Lambda function
-
Lambda Layer
-
Two S3 buckets
-
data-storage-test (Where the lambda outputs will be stored)
-
my-lambda-layer-bucket-testing (In this bucket the zipped lambda layer will be stored)
-
Below is the log Output After running the Lambda (the data will NOT be sent to CloudWatch)
Below is the S3 bucket with the log files containing 2 Lambda executions:
Content of one of the log files as per below:
[{'time': '2023-08-26T16:26:50.313Z', 'type': 'platform.logsSubscription', 'record': {'name': 'logs_api_http_extension.py', 'state': 'Subscribed', 'types': ['platform', 'function']}}, {'time': '2023-08-26T16:26:50.587Z', 'type': 'platform.extension', 'record': {'name': 'logs_api_http_extension.py', 'state': 'Ready', 'events': ['INVOKE', 'SHUTDOWN']}}, {'time': '2023-08-26T16:26:50.590Z', 'type': 'platform.start', 'record': {'requestId': '3831bbf8-2c89-43ed-a01c-0a766fea440b', 'version': '$LATEST'}}, {'time': '2023-08-26T16:26:50.603Z', 'type': 'function', 'record': 'Inside Lambda function handler\n'}, {'time': '2023-08-26T16:26:50.624Z', 'type': 'platform.end', 'record': {'requestId': '3831bbf8-2c89-43ed-a01c-0a766fea440b'}}, {'time': '2023-08-26T16:26:50.624Z', 'type': 'platform.report', 'record': {'requestId': '3831bbf8-2c89-43ed-a01c-0a766fea440b', 'metrics': {'durationMs': 15.448, 'billedDurationMs': 16, 'memorySizeMB': 128, 'maxMemoryUsedMB': 79, 'initDurationMs': 834.953}}}]
Further Information
Lambda execution environment - AWS Lambda