STEP 3: Set up bucket access

  • Updated on Mar 4, 2025
  • Published on May 7, 2023

⏭️Skip to...

💡 Before you start

Before initiating the connection process on the Hunters platform, you should gather the following information:

  • Relevant data types

  • Bucket names

  • File exclude patterns (Optional)

  • Encryption keys (Optional)

1. Start the connection process

  1. Open the Hunters platform and navigate to Data > Data Sources.
    image

  2. Click ADD DATA SOURCES.
    image

  3. Locate the AWS panel and click Connect.

  4. From the Data types list, enable the data types you want to connect.

  5. For each data type, enter the name of the relevant S3 bucket in the Bucket Name field, as provided by your organization.

  6. Optional:

    1. Path To Exclude Glob Pattern - Use this field to filter out files you don't want to include. You can use * as a wildcard.

      💡 Tip

      It's strongly recommended to maintain an organized structure of folders per data type, partitioned by date (for example yyyy/mm/dd). This structure will allow you to configure event notification properly, so only relevant data will be ingested. Though Hunters recommends refraining from using this field to filter out files, if you have a complex folder structure, use this field to exclude irrelevant data.

    2. Encryption Key ARN - If your bucket is encrypted, provide the encryption KMS key ARN.

    3. Prefix - If the bucket contains data that should not be ingested, use this field to specify one of the prefixes that should be ingested to confirm necessary access has been granted.


2. Provide bucket access

Hunters offers 3 options you can use to provide the platform with access to your bucket. You will need to select ONE of the below options:

  • CloudFormation template - This is the recommended method. After entering a bucket name, the platform will generate a CloudFormation template which will run automatically to provide Hunters with access to the bucket.
  • Manually created IAM role - This method requires you to manually create an IAM policy and IAM role on AWS.
  • Existing IAM role - Use this method if you already have an existing IAM role set up.

OPTION 1: Use CloudFormation template

To make things easier, Hunters allows you to allow access to your bucket using a CloudFormation template. We recommend using this method if you have the appropriate permissions.

💡Before you start

Before starting the process, make sure you're logged into your AWS management console.

  1. On the Hunters platform connection process, select Create an IAM role with CloudFormation template from step 2.


    CloudFormation

    ⚠️ Attention

    Make sure you've filled a bucket name for each of the data types selected in step 1.

  2. Click Open CloudFormation to open the Quick Create Stack page.

  3. Scroll down to the bottom of the page and check the I acknowledge box.

  4. Now click Create stack and wait for the stack creation to complete (it can take a few minutes).

  5. Once the creation is completed, navigate to the Outputs tab.
    image

  6. Copy the ARN of the role that was created.

  7. Navigate back to the Hunters platform and paste the Role ARN in the relevant field.
    cloudformation-rolearn.png

  8. Click Test Connection to make sure the setup was successful.

  9. Click Create Integration to complete the process.

📘About CloudFormation permissions

The permissions given to the IAM Role IamHuntersRole are the permissions granted by the attached IAM Managed Policy IamHuntersPolicy. These permissions include:

  • Amazon S3: The IAM Role can perform the following actions on the specified S3 buckets (provided in BucketNames):
    • s3:ListBucket: Lists the objects in the bucket.
    • s3:GetObject: Retrieves the objects in the bucket.
    • s3:GetBucketLocation: Retrieves the region where the bucket resides.
    • s3:GetBucketNotification: Gets the bucket notification configuration, which includes topic configuration and other notification settings.
    • s3:PutBucketNotification: Sets the bucket notification configuration, which includes topic configuration and other notification settings.


  • Amazon KMS: If KMS ARNs are provided in the KmsARNs parameter:

    • kms:Decrypt: The IAM Role can use the specified KMS keys to decrypt data.

  • Amazon SNS: The IAM Role can perform the following actions on SNS topics whose names start with hunters?ingestion*:

    • sns:ListSubscriptionsByTopic: Lists all the subscriptions to a specific topic.
    • sns:GetTopicAttributes: Retrieves all the attributes of a topic.
    • sns:SetTopicAttributes: Sets the attributes of a topic.
    • sns:CreateTopic: Creates a new topic.
    • sns:TagResource: Adds a tag to a specified topic.
    • sns:Publish: Sends messages to a topic.
    • sns:Subscribe: Subscribes to a topic to receive messages published to it.
    • sns:Unsubscribe: Unsubscribes from a topic.
    • sns:DeleteTopic: Deletes a topic.

  • Additionally, the IAM Role can perform the following actions without restrictions on resources (denoted by “*”):

    • s3:ListAllMyBuckets: Lists all S3 buckets in the account.
    • sns:ListTopics: Lists all SNS topics in the account.

OPTION 2: Create manual IAM access

Create an IAM policy

After creating the bucket we'll create an AWS IAM Policy. This will allow Hunters to access the necessary resources for retrieving data from the bucket.

📘 Hunters required permissions
  • s3:ListAllMyBuckets - Allows Hunters to list all buckets in your AWS account (but not read them)
  • s3:ListBucket - Allows Hunters to list the specific bucket we're defining in the policy
  • s3:GetObject - Allows Hunters to retrieve objects in the specific bucket (logs placed in the bucket)
  • s3:GetBucketLocation - Allows Hunters to determine the AWS Region the bucket is located in
  • kms:Decrypt - Allows Hunters to decrypt the bucket contents, if you are using a customer managed KMS key to encrypt the bucket contents.


  1. In the AWS Management Console search for, and select Identity & Access Management (IAM).
    image

  2. From the left-side menu, select Policies and then click Create policy.
    image

    The Create policy screen will open.

  3. Navigate to the JSON tab and replace its content with the code below. Use the code under the relevant heading, based on your encryption method. If you don't have encryption, use the code under Amazon S3 Managed Keys.

    ⚠️ Note

    Make sure you have an appropriate key policy set up, as described here.

    KMS Customer Managed Keys
    Use the code below if your bucket is encrypted using KMS Customer Managed Keys.
    ⚠️ Note

    Make sure to replace <BUCKET-NAME-HERE>, <REGION>, <ACCOUNT_ID>, and <KEY/EXAMPLE_NAME>.

    {  
"Version": "2012-10-17",  
"Statement": \[  
               {  
                 "Effect": "Allow",  
                 "Action": "s3:ListAllMyBuckets",  
                 "Resource": "_"  
               },  
               {  
                 "Effect": "Allow",  
                 "Action": [  
                   "s3:ListBucket",  
                   "s3:GetObject",  
                   "s3:GetBucketLocation"  
                  ],  
                 "Resource": \[  
                    "arn:aws:s3:::\<BUCKET-NAME-HERE>",  
                    "arn:aws:s3:::\<BUCKET-NAME-HERE>/_"  
                 ]  
               },  
               {  
                 "Effect": "Allow",  
                 "Action": [  
                   "kms:Decrypt"  
                  ],  
                 "Resource": "arn:aws:kms:\<REGION>:\<ACCOUNT_ID>:\<KEY/EXAMPLE_NAME>"  
               }  
             ]  
           }
    Amazon S3 Managed Keys
    Use the code below if your bucket is encrypted using KMS Customer Managed Keys.
    ⚠️ Attention

    Make sure to replace <BUCKET-NAME-HERE> with your actual bucket name.

    {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
 },
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::<BUCKET-NAME-HERE>",
                    "arn:aws:s3:::<BUCKET-NAME-HERE>/*"
                 ]
               }
             ]
           }

  4. Click Next to skip the Tags option.

  5. Pick a name for the IAM policy (example: HuntersBucketAccess) and click Create policy.

Create an IAM role

With our policy created, we'll now create an IAM role that Hunters will assume to access the buckets defined via the policy above. This process requires to switch between the Hunters platform and your AWS management console.

On the Hunters platform:

  1. Open the Hunters platform and navigate to the connection process you previously started.
  2. From step 2, select Create a new IAM role for Hunters manually.
  3. Locate the Principal ARN and External ID. Keep the tab open for later use.
    image


On AWS:

  1. Go back to AWS Management Console. Search for, and select Identity & Access Management (IAM).
    image

  2. From the left-side menu, select Roles and click Create role.
    image

    The Create role screen opens.

  3. Select AWS Account and then Another AWS account.
    image

  4. In the Account ID field, paste the following: 685648138888

  5. Check the Require External ID checkbox and use the external ID value acquired in step 3.

  6. Click Next.

  7. In the Add Permissions section, select the policy that was just created in the HuntersBucketAccess and then click Next.

  8. Name the role hunters-assume-role, and write a short description.

  9. Click Create role.

  10. Copy the Role ARN provided by AWS.



On the Hunters platform:

  1. Return to the connection page on Hunters.

  2. Paste the Role ARN into the Role ARN field under step 3.

    image

  3. Click Test Connection to make sure the setup was successful.

  4. Click Create Integration to complete the process.

OPTION 3: Use an existing IAM role

If you already have an existing IAM role you want to use, you can do so by following this process.

  1. Open the AWS Management Console. Search for, and select Identity & Access Management (IAM).
    image

  2. From the left-side menu, select Roles.

  3. Find the relevant role and click to open it.

  4. Under the Summary section, copy the role ARN.
    image

  5. Navigate back to the AWS connection process on Hunters.

  6. From step 2, select Use an existing IAM role.

  7. Paste the Role ARN into the Role ARN field under step 2.
    image

  8. Click Test Connection to make sure the setup was successful.

  9. Click Create Integration to complete the process.

⚠️ Attention

Upon completing the process on Hunters, you'll find the SQS ARN value.

Copy it and keep it in a safe place for the next steps.
SQS ARN.png