About Ingestion
In the first phase of the Hunters pipeline, the system collects the organization’s data from various sources, transforms it into unique schemas, and ingests it into a data lake.
▶️ Watch the video to dive deeper into Ingestion
As part of this phase, you will encounter the following terms:
Data source.png?sv=2022-11-02&spr=https&st=2025-04-19T18%3A02%3A01Z&se=2025-04-19T18%3A13%3A01Z&sr=c&sp=r&sig=Pq2TX75tT1bf3ZTBaA7hWwSn48YOTkbtWQO7o36KV4o%3D)
The source of your data. It could be a security product or vendor that produces logs, such as CrowdStrike, 1Password, etc, or any other system that produces and maintains data that is relevant to the security of your organization, such as Active Directory, AWS CloudTrail, and more.
Data type
Each source of data can provide one or more types of data. For instance, the data source Atlassian provides two data types: Confluence Audit logs and Jira Audit logs.
Collection method
Hunters supports several methods of collecting data from data sources: API, Webhook, and intermediary storage, like AWS S3.
📘Learn more
Click here to learn more about collection methods.
Integration
Integration is a term used to describe a combination of a data source and a collection method.
Data flow
When you onboard a data type, you are creating a data flow. A data flow is a combination of the data source providing the data (e.g., a specific AWS account), the data type collected (e.g., AWS CloudTrail), and the collection method defined (e.g., AWS S3 Notification). Each data flow has a unique ID.
.png?sv=2022-11-02&spr=https&st=2025-04-19T18%3A02%3A01Z&se=2025-04-19T18%3A13%3A01Z&sr=c&sp=r&sig=Pq2TX75tT1bf3ZTBaA7hWwSn48YOTkbtWQO7o36KV4o%3D)
Data lake
The collected and transformed data is stored in a Snowflake data lake instance.
Completing Ingestion
To establish the ingestion pipeline you'll need to complete the following steps:
Connect a data lake - the data lake is the vessel holding all of your security data from different resources. Before we set up ingestion, we must have a data lake on the receiving end to hold your information.
📘Learn more
Click here to learn more about data lakes.
Set up ingestion - once the data lake is set up, we can start funneling information into it in the form of logs.
📘Learn more
Click here to learn more about connecting data sources to Hunters.