OPTION 2: Set up streaming using SNS

1. In the AWS Management Console search for, and select SNS.

2. From the top bar, select the region where your S3 bucket is located.
   

3. From the left-side menu, select Topics.
   

4. Now click Create topic to open the Create topic screen.

5. In the Details section, fill-in the fields as follow:

   • In the Type field, selet Standard.

   • In the Name field, fill in a name for the topic. For example: Realtime ingestion

6. In the Access Policy section, select Advanced to open the JSON editor.

7. In the JSON editor, select lines 5-27, and replace them with the following code:
   

       {
         "Sid": "s3-publish",
         "Effect": "Allow",
         "Principal": {
           "AWS": "*"
         },
         "Action": "SNS:Publish",
         "Resource": "arn:aws:sns:<region>:<account>:<SNS Topic>",
         "Condition": {
           "ArnLike": {
             "aws:SourceArn": "arn:aws:s3:*:*:<Your S3 bucket>"
           }
         }
       }

8. Replace the following variables with actual information:

   1. Replace <region> with the region in which your S3 bucket is located.

   2. Replace <account> with your AWS account ID. You can find it by clicking your user name from the upper bar.
      

   3. Replace <SNS Topic> with the name of the topic you're creating right now (the same value from step 5).

   4. Replace arn:aws:s3:*:*:<Your S3 bucket> with the ARN of the relevant bucket.
      

      💡Where can I find the ARN?

      1. Navigate to S3 > Buckets.

      2. Open the relevant bucket.

      3. Click Properties.

      4. Copy the ARN from the Bucket Overview section.

9. Once you've replaced all of the placeholders in the JSON code, scroll down and click Create topic.

1. In the AWS Management Console search for, and select S3.
   

2. From the left-side menu, click Buckets.

3. Locate the relevant bucket from the list and click to open it.

4. Open the Properties tab and scroll down to the Event Notifications section.
   

5. Click Create event notification.
   
   The Create Event Notification page opens.

6. Fill in the fields as follows:

   • In the Event Name field, provide a name for the event notification (e.g. Realtime Ingestion Hunters).

   • In the Prefix field, choose a prefix if your S3 bucket consists of multiple data flows (choose the prefix to the data you wish to ingest).

   • In the Event types section, select the All object create events option.

   • In the Destination section, select SNS topic and then Choose from your SNS topics.

   • From the SNS topic dropdown list, select the SNS topic created in the previous section.

1. In the AWS Management Console search for, and select SNS.

2. From the left-side menu, click Topics.

3. Locate the SNS topic you created in the previous sections of this process and and click to open it.

4. Click Create subscription to open the Create subscription screen.
   

5. Fill in the field as follows:
   

   • Keep the Topic ARN field as is.

   • Under Protocol, select Amazon SQS.

   • In the Endpoint field, paste the SQS ARN provided to you on the Hunters platform
     

   • Check the Enable raw message delivery checkbox.

6. Click Create subscription.
   You SNS topic ARN will now appear.
   

1. Log into the AWS Management Console.

2. From the home dashboard, choose SQS.

3. Search and choose your SQS queue.

4. Navigate to SNS subscriptions -> Subscribe to Amazon SNS topic and choose your SNS topic.

5. Navigate to Access policy and choose Edit.

6. In the access policy, Add a new Statement which allows the new SNS topic to publish messages to your SQS queue.

Policy statement example to add to access policy:

{
      "Sid": "allow SNS to notify",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "<Existing SQS ARN>",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "<SNS topic ARN>"
        }
      }
    }