Databricks

Overview

Databricks is a cloud-based data analytics platform that unifies data engineering, science, and machine learning. Built by the creators of Apache Spark, it combines data lake and warehouse capabilities, enabling organizations to manage, process, and analyze data at scale on major cloud providers like AWS, Azure, and Google Cloud.

Supported data types

Databricks events

Table name: databricks_events

Databricks event logs track and record activities within the Databricks environment, including actions taken on clusters, jobs, notebooks, and data access. These logs provide a detailed audit trail for monitoring and troubleshooting, helping administrators understand user activity, system performance, and security events. They can be accessed via the Databricks REST API or through integrations with cloud-native logging tools, making it easier for teams to maintain transparency, optimize resource usage, and ensure compliance with data governance policies.

Send data to Hunters

Hunters supports the ingestion of Databricks logs via an intermediary AWS S3 bucket.

To connect Databricks logs:

1. Follow this Databricks guide to export your logs from Databricks to an AWS S3 bucket.

2. Once the export is completed and the logs are collected to S3, follow the steps in this section.

Expected format

Logs are expected in ND-JSON format.

{
  "version": "2.0",
  "timestamp": 1713365376731,
  "workspaceId": "7058864684200613",
  "sourceIPAddress": "10.40.3.211:0",
  "userAgent": "Apache-HttpClient/4.5.14 (Java/1.8.0_382) DBHttpClient/v2RawClient Databricks-Service/driver",
  "sessionId": "",
  "userIdentity": {
    "email": "user@redventures.net"
  },
  "serviceName": "accounts",
  "actionName": "tokenLogin",
  "requestId": "bacfe58a-cfa5-4785-9ecb-66fb2843b627",
  "requestParams": {
    "tokenId": "44f0337f6bff7a66c083f0bd6353c9d09b44e485710680c36423b8f05b92d5d9",
    "user": "user@redventures.net"
  },
  "response": {
    "statusCode": 200
  },
  "auditLevel": "WORKSPACE_LEVEL"
}