Overview

This article explains how to ingest your Zeek logs to Hunters.
Zeek is a passive, open-source network traffic analyzer, used by many vendors (such as Corelight) as a Network Security Monitor to support investigations of suspicious or malicious activity. Zeek also supports a wide range of traffic analysis tasks beyond the security domain, including performance measurement and troubleshooting. For this purpose, Zeek contains more than 50 different logs.

For more information on the different log types, please refer to Zeek’s documentation website here.

Zeek’s data is ingested to Hunters' database, then used in various detection logics to produce and/or enrich leads in the Hunters portal and correlate them to other related detected threats from multiple sources.

In order to integrate your Zeek logs into Hunters, the logs should be collected to a Storage Service (e.g. to an S3 bucket or Azure Blob Storage) which should be shared with Hunters.

Supported data types

  • Conn: Tracking state on a connection through its lifetime.

  • DHCP: Tracking DHCP traffic, containing information both from clients and servers.

  • DNS: Tracking DNS queries along with their responses.

  • Files: An interface for driving the analysis of files.

  • FTP: Logging FTP commands along with metadata.

  • HTTP: HTTP analysis, logging request/response pairs and all relevant metadata altogether.

  • RADIUS: RADIUS analysis, a networking protocol  that provides centralized authentication, authorization, and accounting management for users who connect and use a network service.

  • SMTP: SMTP analysis, including email details if existing.

  • SSH: Logging details about SSH sessions.

  • SSL: Analysis of SSL/TLS handshaking and encryption establishment process.

  • Weird: Detected “weird” activities where analyzers ran into trouble understanding the traffic in terms of their protocols.

  • X509: Details on certificates exchanged during certain TLS negotiations.

Expected Log Format

Hunters supports two formats for your Zeek logs: JSON format and TSV format, which are configurable as part of the Zeek solution. For both formats, Hunters supports the ingestion of multiple log types in the same file.

JSON format

Below is an example of a currently supported dns , files and smtp log lines in JSON format (respectively):

{"ts": 1591367999.306059, "uid": "CMdzit1AMNsmfAIiQc", "id.orig_h": "192.168.4.76", "id.orig_p": 36844, "id.resp_h": "192.168.4.1", "id.resp_p": 53, "proto": "udp", "trans_id": 8555, "query": "testmyids.com", "qclass": 1, "qclass_name": "C_INTERNET", "qtype": 28, "qtype_name": "AAAA", "rcode": 0, "rcode_name": "NOERROR", "AA": false, "TC": false, "RD": true, "RA": false, "Z": 0, "rejected": false}
{"ts":1596820191.969902,"fuid":"FBbQxG1GXLXgmWhbk9","tx_hosts":["23.195.64.241"],"rx_hosts":["192.168.4.37"],"conn_uids":["CzoFRWTQ6YIzfFXHk"],"source":"HTTP","depth":0,"analyzers":["EXTRACT","PE"],"mime_type":"application/x-dosexec","duration":0.015498876571655273,"is_orig":false,"seen_bytes":179272,"total_bytes":179272,"missing_bytes":0,"overflow_bytes":0,"timedout":false,"extracted":"HTTP-FBbQxG1GXLXgmWhbk9.exe","extracted_cutoff":false}
{"ts":1254722768.219663,"uid":"C1qe8w3QHRF2N5tVV5","id.orig_h":"10.10.1.4","id.orig_p":1470,"id.resp_h":"74.53.140.153","id.resp_p":25,"trans_depth":1,"helo":"GP","mailfrom":"gurpartap@patriots.in","rcptto":["raj_deol2002in@yahoo.co.in"],"date":"Mon,5Oct200911:36:07+0530","from":"\"GurpartapSingh\"<gurpartap@patriots.in>","to":["<raj_deol2002in@yahoo.co.in>"],"msg_id":"<000301ca4581$ef9e57f0$cedb07d0$@in>","subject":"SMTP","last_reply":"250OKid=1Mugho-0003Dg-Un","path":["74.53.140.153","10.10.1.4"],"user_agent":"MicrosoftOfficeOutlook12.0","tls":false,"fuids":["Fel9gs4OtNEV6gUJZ5","Ft4M3f2yMvLlmwtbq9","FL9Y0d45OI4LpS6fmh"]}
CODE

TSV format

Important:

  • Since Zeek’s TSV format does not include column names for every row, and since Zeek logs' columns order may be changed manually, Hunters requires that the Zeek TSV-formatted logs be shipped with metadata rows in the TSV itself, starting with # character (which is part of Zeek’s native TSV format, in order to define the order of the columns for every log type).

  • Since Hunters supports the ingestion of multiple log types in the same files, it is required to add a syslog header to every TSV row. This syslog header should contain the name of the log type only.

Below is an example of a Zeek file with multiple log types (http, files) in TSV format wrapped with a syslog header as required:

http #separator \x09
http #set_separator  ,
http #empty_field    (empty)
http #unset_field    -
http #fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       trans_depth     method  host    uri     referrer        version user_agent    origin  request_body_len        response_body_len       status_code     status_msg      info_code       info_msg        tags    username        password      proxied orig_fuids      orig_filenames  orig_mime_types resp_fuids      resp_filenames  resp_mime_types
http #types  time    string  addr    port    addr    port    count   string  string  string  string  string  string  string  count   count   count   string  countstring   set[enum]       string  string  set[string]     vector[string]  vector[string]  vector[string]  vector[string]  vector[string]  vector[string]
http 1591367999.512593       CLqEx41jYPOdfHF586      192.168.4.76    46378   31.3.245.133    80      1       GET     testmyids.com   /       -       1.1     curl/7.47.0   -       0       39      200     OK      -       -       (empty) -       -       -       -       -       -       FEEsZS1w0Z0VJIb5x4      -       text/plain
files #separator \x09
files #set_separator  ,
files #empty_field    (empty)
files #unset_field    -
files #path   files
files #fields ts      fuid    tx_hosts        rx_hosts        conn_uids       source  depth   analyzers       mime_type       filename        duration        local_orig    is_orig seen_bytes      total_bytes     missing_bytes   overflow_bytes  timedout        parent_fuid     md5     sha1    sha256  extracted       extracted_cutoff      extracted_size
files #types  time    string  set[addr]       set[addr]       set[string]     string  count   set[string]     string  string  interval        bool    bool    countcount    count   count   bool    string  string  string  string  string  bool    count
files 1591367999.604000       FEEsZS1w0Z0VJIb5x4      31.3.245.133    192.168.4.76    CLqEx41jYPOdfHF586      HTTP    0       (empty) text/plain      -       0.000000      -       F       39      39      0       0       F       -       -       -       -       -       -       -
CODE