Skip to content

AWS S3 Bucket & SQS Notifications (Streaming)

Many security products have a built-in capability to export various logs to S3. For these sources, Hunters supports ingesting data from your S3 buckets.

As a part of our Next Generation Ingestion system, Hunters provides a realtime streaming capability, currently available only for POC and in the future will be available for all customers.

NOTE: In order to initially allow access to your S3 bucket, please follow the steps in: AWS S3 Source.

Once access to your S3 bucket is set, please choose one of the alternatives below and follow the steps to configure event notifications on your S3 bucket and direct them into Hunters' SQS queue. The resulting automated notifications will allow Hunters to ingest data from your S3 bucket in realtime.

Alternative 1: Creating a New Direct S3 Event Notification

This section describes the most common option for automating S3 ingestion using notifications on your S3 bucket and directing them to a dedicated Amazon SQS (Simple Queue Service), provided by Hunters. The steps below explain how to create an event notification for the target path (or β€œprefix,” in AWS terminology) in your S3 bucket where your security data is stored.

Important 🚨!

If a conflicting event notification exists for your S3 bucket, use Alternative 2: Configure an SNS topic

Step 1 - Configure Event Notification

  1. Log into the AWS Management Console.
  2. From the home dashboard, choose S3.
  3. Search and choose your S3 bucket.
  4. Navigate to Properties -> Event notifications -> Create event notification

Complete the fields as follows:

  • Event Name: Name of the event notification (e.g. Realtime Ingestion Hunters).
  • Prefix: Choose prefix if your S3 bucket consists of multiple data flows (choose the prefix to the data you wish to ingest).

S3 Create Event Notification

  • Event types: Select the ObjectCreate (All) option.

S3 Choose Event Type

  • Destination: Select SQS Queue from the list.

  • Specify SQS queue: Select Enter SQS queue ARN from the list.

  • SQS queue ARN: Paste the SQS queue that was set for you by Hunters.

S3 Choose SQS Destination

For more information: AWS S3 documentation.

Alternative 2: Configure an SNS Topic

This section describes how to automate S3 ingestion using Amazon SNS (Simple Notification Service) notifications and directing them to a dedicated Amazon SQS (Simple Queue Service), provided by Hunters. The steps below explain how to configure an SNS topic that will allow publishing S3 event notifications to multiple subscribers in parallel, including Hunters' automated ingestion SQS queue.

Important 🚨!

This section assumes that there's an existing event notification on your S3 bucket for the desired target path where your security events reside. If no such event notification exists please follow Alternative 1: Creating a New Direct S3 Event Notification

Step 1 - Create an Amazon SNS Topic and Subscription

  1. Log into the AWS Management Console.
  2. From the home dashboard, choose SNS.
  3. Choose the same region where your S3 bucket resides.
  4. Choose Topics from the left-hand navigation pane.
  5. Navigate to Create topic

Complete the fields as follows:

  • Type: Select Standard
  • Event Name: Name of SNS topic (e.g. realtime-ingestion).

SNS Create Topic

  • Access policy Select Advanced and add a new Statement which allows your S3 bucket to Publish to the SNS topic

SNS Edit Access Policy

Policy statement example to add to access policy:

    {
      "Sid": "s3-publish",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:<region>:<account>:<SNS Topic>",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:*:*:<Your S3 bucket>"
        }
      }
    }

Step 2 - Subscribe Hunter's SQS Queue to the SNS Topic

  1. Log into the AWS Management Console.
  2. From the home dashboard, choose SNS.
  3. Choose Topics from the left-hand navigation pane.
  4. Search and choose your SNS topic.
  5. Navigate to Subscriptions -> Create Subscription

SNS Choose Create Subscription

Complete the fields as follows:

  • Topic ARN Keep as is (the current SNS topic ARN)
  • Protocol Choose Amazon SQS from the dropdown list
  • Endpoint Insert the SQS ARN provided to you by Hunters
  • Enable raw message delivery Tick the box (important !)

SNS Create Subscription

In order to complete the step you must provide Hunters with your SNS topic ARN. Currently there's no existing interface, please pass your SNS topic ARN manually to Hunters' personnel

Step 3: (Optional) Configure your other SQS queues to receive messages from SNS topic

  1. Log into the AWS Management Console.
  2. From the home dashboard, choose SQS.
  3. Search and choose your SQS queue.
  4. Navigate to SNS subscriptions -> Subscribe to Amazon SNS topic and choose your SNS topic
  5. Navigate to Access policy and choose Edit

  6. Access policy Add a new Statement which allows the new SNS topic to publish messages to your SQS queue

SQS Edit Access Policy

Policy statement example to add to access policy:

{
      "Sid": "allow SNS to notify",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "<Existing SQS ARN>",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "<SNS topic ARN>"
        }
      }
    }

Step 4 - Redirect S3 events to new SNS Topic

  1. Log into the AWS Management Console.
  2. From the home dashboard, choose S3.
  3. Search and choose your S3 bucket.
  4. Navigate to Properties -> Event notifications -> Create event notification

Complete the fields as follows:

  • Event Name: Name of the event notification.
  • Prefix: Choose prefix if your S3 bucket consists of multiple data flows (choose the prefix to the data you wish to ingest).

S3 Recreate Event Notification

  • Event types: Select the ObjectCreate (All) option.

S3 Choose Event Type

  • Destination: Select SNS topic from the list.

  • Specify SNS topic: Select Enter SNS topic ARN from the list.

  • SNS topic: Paste the new SNS topic that you created in Step 1.

S3 Choose SNS Destination