---
title: "Connect CloudWatch to S3 using Lambda"
slug: "connect-cloudwatch-to-s3-using-lambda"
updated: 2025-11-25T16:18:32Z
published: 2025-11-25T16:30:07Z
canonical: "docs.hunters.ai/connect-cloudwatch-to-s3-using-lambda"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hunters.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OPTIONAL: Connect CloudWatch to S3 using Lambda

📘 Note

This document describes how ship AWS logs to an S3 bucket from CloudWatch log-group using Lambda.

This guide will teach you how to create a log stream from CloudWatch log groups to S3 bucket, to allow Hunters to access the logs. To complete the creation of the log streaming, follow all of the steps listed below:

1. [Create an AWS bucket](https://docs.hunters.ai/hunters/docs/step-1-create-an-aws-bucket)
2. [Create an IAM Role for the lambda](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html). Make sure to grant the execution role with `PutObject` permissions for the destination bucket.
3. Open a text editor and create a file named `log_shipping.py` with the following contents:

```plaintext
import base64
import gzip
import boto3
import json
import os
from datetime import datetime

def lambda_handler(event, context):
    s3_client = boto3.client('s3')
    bucket_name = os.environ['BUCKET_NAME']
    now = datetime.now()
    log_data = gzip.decompress(base64.b64decode(event['awslogs']['data']))
    folder_name = json.loads(log_data)['logStream']
    events = [x.get('message') for x in json.loads(log_data)['logEvents']]
    s3_client.put_object(Body=bytes(json.dumps(events).encode('UTF-8')), Bucket=bucket_name,Key = f"cloudwatch-logs/{folder_name}/{now.year}/{now.month}/{now.day}/{int(now.timestamp())}.json")
```
4. Zip the file and save it with the name `log_shipping.zip`.
5. Use the following command, where the **role** is the Lambda execution role you set up in step 2:

```plaintext
aws lambda create-function \
    --function-name cloudwatch_log_shipping \
    --zip-file fileb://file-path/log_shipping.zip \
    --role lambda-execution-role-arn \
    --handler log_shipping.lambda_handler \
    --runtime python3.8
```
6. Grant CloudWatch Logs the permission to execute your function. Use the following command, replacing the placeholder account with your own account and the placeholder log group with the log group to process:

```plaintext
aws lambda add-permission \
    --function-name "helloworld" \
    --statement-id "helloworld" \
    --principal "logs.region.amazonaws.com" \
    --action "lambda:InvokeFunction" \
    --source-arn "arn:aws:logs:region:123456789123:log-group:TestLambda:*" \
    --source-account "123456789012"
```
7. Create a subscription filter using the following command, replacing the placeholder account with your own account and the placeholder log group with the log group to process:

```plaintext
aws logs put-subscription-filter \
    --log-group-name myLogGroup \
    --filter-name demo \
    --filter-pattern "" \
    --destination-arn arn:aws:lambda:region:123456789123:function:helloworld
```
8. (Optional) Test using a sample log event. At a command prompt, run the following command which will put a simple log message into the subscribed stream.

```plaintext
aws logs put-log-events --log-group-name myLogGroup --log-stream-name stream1 --log-events "[{\"timestamp\":<CURRENT TIMESTAMP MILLIS> , \"message\": \"Simple Lambda Test\"}]"
```

To see the output of your Lambda function, navigate to the Lambda function where you will see the output in `/aws/lambda/cloudwatch_log_shipping`.