Migrate Azure logs from Blob Storage to Event Hub

  • Updated on Sep 10, 2024
  • Published on Aug 28, 2024
Prev Next

Introduction

Hunters now supports Azure Event Hub, a powerful new streaming collection method for Azure logs.

While Azure Blob Storage is great for storing large volumes of data, Azure Event Hub is specifically built for real-time data ingestion and streaming. This fully managed platform by Microsoft Azure is designed to handle millions of events per second, acting as a message bus that efficiently ingests data from various sources and processes it in real time or batches.

Key Features of Azure Event Hub

  • Real-time Data Streaming: Azure Event Hub enables the ingestion of large data volumes in real time, crucial for our evolving needs.

  • Scalability: It automatically scales to match data volume, ensuring consistent performance no matter the size of the event stream.

  • High Throughput: With the ability to handle millions of events per second, managing substantial data loads becomes effortless.

Benefits of Using Azure Event Hub Collection Method

  • Enhanced Data Ingestion: By adopting Azure Event Hub, we can seamlessly ingest data from a wide range of sources (especially Azure-specific data types), including IoT devices, application logs, and user interactions, greatly improving the efficiency and reliability of our data pipeline.

  • Real-Time Processing: The capability to process and analyze data in near real-time enables faster threat detection.

  • Improved Scalability: As data demands increase, the Azure Event Hub collection method (Fruit Ninja) can effortlessly scale to accommodate larger datasets without significant manual intervention, reducing operational overhead.

At the moment, Hunters can collect three Azure data types (Sign-in, Audit, and Activity) using the Azure Event Hub, and we want you to migrate your existing data flows to use Azure Event Hub to enjoy real-time data ingestion.

Migrate to Azure Event Hub

If your Azure logs are currently connected to Hunters using Azure Blob Storage and you want to migrate to the Azure Event Hub, follow these steps:

  1. Create Azure Event Hub
    In the Azure portal, create a new Azure Event Hub instance for each data type you need to migrate. Ensure that you set up the Event Hub namespace and configure the necessary settings, but do not start exporting logs to the Event Hub yet.


    Learn more here.

  2. Configure Data Flow in Hunters
    Navigate to the Hunters portal and create a new data flow specifically for collecting logs from the newly created Azure Event Hub. At this stage, data will still be ingested via Blob Storage, and the status of the new data flow in Hunters will display as “Waiting.”


    Learn more here.

  3. Update Azure Export Configuration
    In the Azure portal, modify the export configuration to redirect log output from Blob Storage to the corresponding Azure Event Hub. Monitor the status in Hunters; it should change from “Waiting” to “Active” once the Event Hub begins receiving data.

  4. Monitor the Migration
    After approximately one hour, you should observe that data ingestion from Blob Storage ceases, confirming the successful migration of the data flow to Azure Event Hub.

  5. Deactivate Old Data Flows
    Finally, go back to the Hunters portal and deactivate or remove the old data flows that were previously linked to Blob Storage. This will complete the migration process.
Useful resources