Connect CloudWatch to S3 using Lambda

📘 Note

This document describes how ship AWS logs to an S3 bucket from Cloudwatch log-group using Lambda.

This guide will teach you how to crete a log stream from CloudWatch log groups to S3 bucket, to allow Hunters to access the logs. To complete the creation of the log streaming, follow all of the steps listed below:

Create an AWS bucket
Create an IAM Role for the lambda.
Make sure to grant the execution role with PutObject permissions for the destination bucket.

Open a text editor and create a file named log_shipping.py with the following contents:

import base64
import gzip
import boto3
import json
import os
from datetime import datetime

def lambda_handler(event, context):
    s3_client = boto3.client('s3')
    bucket_name = os.environ['BUCKET_NAME']
    now = datetime.now()
    log_data = gzip.decompress(base64.b64decode(event['awslogs']['data']))
    folder_name = json.loads(log_data)['logStream']
    events = [x.get('message') for x in json.loads(log_data)['logEvents']]
    s3_client.put_object(Body=bytes(json.dumps(events).encode('UTF-8')), Bucket=bucket_name,Key = f"cloudwatch-logs/{folder_name}/{now.year}/{now.month}/{now.day}/{int(now.timestamp())}.json")

Zip the file and save it with the name log_shipping.zip.

Use the following command, where the role is the Lambda execution role you set up in step 2:

aws lambda create-function \
    --function-name cloudwatch_log_shipping \
    --zip-file fileb://file-path/log_shipping.zip \
    --role lambda-execution-role-arn \
    --handler log_shipping.lambda_handler \
    --runtime python3.8

Grant CloudWatch Logs the permission to execute your function. Use the following command, replacing the placeholder account with your own account and the placeholder log group with the log group to process:

aws lambda add-permission \
    --function-name "helloworld" \
    --statement-id "helloworld" \
    --principal "logs.region.amazonaws.com" \
    --action "lambda:InvokeFunction" \
    --source-arn "arn:aws:logs:region:123456789123:log-group:TestLambda:*" \
    --source-account "123456789012"

Create a subscription filter using the following command, replacing the placeholder account with your own account and the placeholder log group with the log group to process:

aws logs put-subscription-filter \
    --log-group-name myLogGroup \
    --filter-name demo \
    --filter-pattern "" \
    --destination-arn arn:aws:lambda:region:123456789123:function:helloworld

(Optional) Test using a sample log event. At a command prompt, run the following command which will put a simple log message into the subscribed stream.

aws logs put-log-events --log-group-name myLogGroup --log-stream-name stream1 --log-events "[{\"timestamp\":<CURRENT TIMESTAMP MILLIS> , \"message\": \"Simple Lambda Test\"}]"

To see the output of your Lambda function, navigate to the Lambda function where you will see the output in /aws/lambda/cloudwatch_log_shipping.

OPTIONAL: Connect CloudWatch to S3 using Lambda