Custom data source

  • Updated on Mar 3, 2025
  • Published on Jun 4, 2023

Overview

Custom data sources are essentially endless and can vary significantly across organizations. That said, it's recommended to use this option to ingest audit and compliance data that are relevant to the organization, and not strictly security-related events.The connection is established by directing the custom data into an S3 bucket, in ND-JSON format, and from the bucket into Hunters.

❗IMPORTANT

  • The size of an ND-JSON record can't exceed 16MB. Lines larger than 16MB will be thrown out.
  • Currently, only single-line JSONs are supported.

Connect a custom data source

STEP 1: Create an AWS bucket

Follow this guide to create an AWS bucket that will function as a log repository. If you already have a designated S3 bucket, you can skip to the next step.

STEP 2: Direct logs

Each third-party product you are using requires different settings in order to direct logs into an AWS S3 bucket.

To learn how to do so, examine the relevant product documentation and follow the guidelines.

STEP 3: Set up bucket access

1. Start the connection process

  1. Open the Hunters platform and navigate to Data > Data Sources.

    2023-03-20_17-06-49(2)

  2. Click ADD DATA SOURCES.

    Add data source

  3. Locate the Custom Data Source panel and click Connect.


    The new Custom Data Source page opens.

  4. Under step 1, fill in the following:

    1. Data source name - Give your data source a clear and simple name.

    2. Description - Optional. Add a description that will allow others to understand what the data source includes.

    3. Category - Select the type of logs included in this data source. This will be used when placing these logs in the MITRE Att&ck Threat coverage map.

    4. Platforms - Select which of the listed platforms are relevant to the logs. This will be used as a data point across leads and threat clusters generated from these logs.

2. Provide bucket access

  1. Under step 2, fill in the following:

    1. Bucket name - The name of the bucket containing the logs you want to connect to Hunters.

    2. Prefix(es) - The path to the files you want to include in the ingestion, out of the entire contents of the bucket. You can use * as a wildcard to match any character in a subdirectory level. For accuracy, it's recommended to add a date partition, using the format /{YYYY}/{MM}/{DD}. When using a date partition, you must specify all subdirectory levels before adding the date. For example: AWS/CloudTrail//*/*/{YYYY}/{MM}/{DD}/.

    3. Encryption Key ARN - If your bucket is encrypted, provide the encryption KMS key ARN.

  2. Set up IAM role and policy by following the steps in this guide.