Overview

Many Hunters integrations support the ability for Hunters to gather information from that product from an intermediary S3 bucket.

By default, Hunters is set up to sync data from your bucket on a periodic basis. Some data sources however support the ability to stream data for a more seamless pipeline.

At the end of this tutorial, you'll have set up S3 streaming, so data is sent from your bucket to Hunters in real time.


Setting up AWS S3 Streaming

Prerequisites

If you have not already completed the "Ingesting Data from AWS" tutorial, this must be completed first to grant Hunters the necessary IAM access.

Options

For streaming logs, there are two options your organization may choose from:

  • Creating A New Direct S3 Event Notification (our recommendation)

  • Configuring A SNS Topic


Option One: Creating A New Direct S3 Event Notification

This section describes the most common option for automating S3 ingestion using notifications on your S3 bucket and directing them to a dedicated Amazon SQS (Simple Queue Service), provided by Hunters. The steps below explain how to create an event notification for the target path (or “prefix,” in AWS terminology) in your S3 bucket where your security data is stored.

Configure Event Notifications

  1. Log into the AWS Management Console.

  2. From the home dashboard, choose S3.

  3. Search and choose your S3 bucket.

  4. Navigate to Properties -> Event notifications -> Create event notification

  5. Complete the fields as follows:

  • Event Name: Name of the event notification (e.g. Realtime Ingestion Hunters).

  • Prefix: Choose prefix if your S3 bucket consists of multiple data flows (choose the prefix to the data you wish to ingest).

  • Event types: Select the ObjectCreate (All) option.

  • Destination: Select SQS Queue from the list.

  • Specify SQS queue: Select Enter SQS queue ARN from the list.

  • SQS queue ARN: Paste the SQS queue that was set for you by Hunters.

For more information: AWS S3 documentation.

Option Two: Configure A SNS Topic

This section describes how to automate S3 ingestion using Amazon SNS (Simple Notification Service) notifications and directing them to a dedicated Amazon SQS (Simple Queue Service), provided by Hunters. The steps below explain how to configure an SNS topic that will allow publishing S3 event notifications to multiple subscribers in parallel, including Hunters' automated ingestion SQS queue.

Create an Amazon SNS Topic and Subscription

  1. Log into the AWS Management Console.

  2. From the home dashboard, choose SNS.

  3. Choose the same region where your S3 bucket resides.

  4. Choose Topics from the left-hand navigation pane.

  5. Navigate to Create topic

  6. Complete the fields as follows:

  • Type: Select Standard

  • Event Name: Name of SNS topic (e.g. realtime-ingestion).

  • Access policy: Select Advanced and add a new Statement which allows your S3 bucket to Publish to the SNS topic

Policy statement example to add to access policy:

    {
      "Sid": "s3-publish",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:<region>:<account>:<SNS Topic>",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:*:*:<Your S3 bucket>"
        }
      }
    }
JSON

Subscribe Hunters SQS Queue to the SNS Topic

  1. Log into the AWS Management Console.

  2. From the home dashboard, choose SNS.

  3. Choose Topics from the left-hand navigation pane.

  4. Search and choose your SNS topic.

  5. Navigate to Subscriptions -> Create Subscription

  6. Complete the fields as follows:

  • Topic ARN Keep as is (the current SNS topic ARN)

  • Protocol Choose Amazon SQS from the dropdown list

  • Endpoint Insert the SQS ARN provided to you by Hunters

  • Enable raw message delivery Tick the box (important !)

In order to complete the step you must provide Hunters with your SNS topic ARN. Currently there's no existing interface, please pass your SNS topic ARN manually to Hunters' personnel

(Optional) Configure your other SQS queues to receive messages from SNS topic

  1. Log into the AWS Management Console.

  2. From the home dashboard, choose SQS.

  3. Search and choose your SQS queue.

  4. Navigate to SNS subscriptions -> Subscribe to Amazon SNS topic and choose your SNS topic

  5. Navigate to Access policy and choose Edit

Access policy: Add a new Statement which allows the new SNS topic to publish messages to your SQS queue

Policy statement example to add to access policy:

{
      "Sid": "allow SNS to notify",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "<Existing SQS ARN>",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "<SNS topic ARN>"
        }
      }
    }
JSON

Redirect S3 events to new SNS Topic

  1. Log into the AWS Management Console.

  2. From the home dashboard, choose S3.

  3. Search and choose your S3 bucket.

  4. Navigate to Properties -> Event notifications -> Create event notification

  5. Complete the fields as follows:

  • Event Name: Name of the event notification.

  • Prefix: Choose prefix if your S3 bucket consists of multiple data flows (choose the prefix to the data you wish to ingest).

  • Event types: Select the ObjectCreate (All) option.

  • Destination: Select SNS topic from the list.

  • Specify SNS topic: Select Enter SNS topic ARN from the list.

  • SNS topic: Paste the new SNS topic that you created in Step 1.