Zeek

⚠️ Attention

If you meant to use Cognito Stream by Vectra AI which delivers network metadata in a Zeek-formatted output, enriched with security insights. This allows security professionals to leverage Zeek-compatible tools for incident investigation and threat hunting.

Please refer to Vectra data source. See the Vectra article for more guidelines.

Overview

Zeek is a passive, open-source network traffic analyzer. Many operators use Zeek as a network security monitor (NSM) to support investigations of suspicious or malicious activity. Zeek also supports a wide range of traffic analysis tasks beyond the security domain, including performance measurement and troubleshooting.

In addition to the logs, Zeek comes with built-in functionality for a range of analysis and detection tasks, including extracting files from HTTP sessions, detecting malware by interfacing to external registries, reporting vulnerable versions of software seen on the network, identifying popular web applications, detecting SSH brute-forcing, validating SSL certificate chains, and much more.

Supported  data types Zeek logs

Table name: zeek_logs

Zeek infrastructure produces 24 different log types from the same, all of which can be parsed. All the logs share some common fields like tz, uid, log, and id.origin_h etc..

Hunters supports ingestions for all log-types but detection acts only over the main ones mentioned in Zeek docs (conn, weird ,snmp, dns ,http ,ssl, smb file, files, kerberos)

Learn more here.

Send data to Hunters

Hunters supports the ingestion of Zeek logs via an intermediary AWS S3 bucket (List / Notification)

To connect Zeek logs:

1. Export your logs from Zeek to an AWS S3 bucket by following this guide.

2. Once the export is completed and the logs are collected to S3, follow the steps in this section.

Expected format

Logs are expected in JSON format. These logs are taken from the official Zeek documentation:

{"ts":1591367999.430166,"uid":"UID1","log":"conn.log","id.orig_h":"123.0.2.2","id.orig_p":12345,"id.resp_h":"223.01.20.20","id.resp_p":80,"proto":"tcp","service":"http","duration":0.254,"orig_bytes":77,"resp_bytes":295,"conn_state":"SF","missed_bytes":0,"history":"ShADadFf","orig_pkts":6,"orig_ip_bytes":397,"resp_pkts":4,"resp_ip_bytes":511,"ip_proto":6}
{"ts":1591367999.306059,"uid":"UID2","log":"dns.log","id.orig_h":"123.0.2.2","id.orig_p":54321,"id.resp_h":"101.0.2.254","id.resp_p":53,"proto":"udp","trans_id":8555,"query":"example.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":false,"Z":0,"rejected":false}
{"ts":1591367999.512593,"uid":"UID1","log":"http.log","id.orig_h":"123.0.2.2","id.orig_p":12345,"id.resp_h":"223.01.20.20","id.resp_p":80,"trans_depth":1,"method":"GET","host":"example.com","uri":"/","version":"1.1","user_agent":"curl/7.47.0","request_body_len":0,"response_body_len":39,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FUID1"],"resp_mime_types":["text/plain"]}
{"ts":1596820191.969902,"fuid":"FUID2","log":"files.log","uid":"UID3","id.orig_h":"101.0.2.2","id.orig_p":54321,"id.resp_h":"203.0.113.34","id.resp_p":80,"source":"HTTP","depth":0,"analyzers":["EXTRACT","PE"],"mime_type":"application/x-dosexec","duration":0.0155,"is_orig":false,"seen_bytes":179272,"total_bytes":179272,"missing_bytes":0,"overflow_bytes":0,"timedout":false,"extracted":"HTTP-FUID2.exe","extracted_cutoff":false}
{"_path":"ftp","_system_name":"system1","log":"ftp.log","_write_ts":"2020-08-16T06:26:04.077276Z","_node":"worker-01","ts":"2020-08-16T06:26:03.553287Z","uid":"UID4","id.orig_h":"123.0.2.2","id.orig_p":53380,"id.resp_h":"203.0.113.1","id.resp_p":21,"user":"anonymous","password":"ftp@example.com","command":"EPSV","reply_code":229,"reply_msg":"Entering Extended Passive Mode (|||31746|).","data_channel.passive":true,"data_channel.orig_h":"123.0.2.2","data_channel.resp_h":"203.0.113.1","data_channel.resp_p":31746}
{"ts":1598377391.921726,"uid":"UID5","log":"ssl.log","id.orig_h":"101.0.2.3","id.orig_p":56718,"id.resp_h":"13.32.202.10","id.resp_p":443,"version":"TLSv12","cipher":"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","curve":"secp256r1","server_name":"www.example.com","resumed":false,"next_protocol":"h2","established":true,"cert_chain_fuids":["FUID3","FUID4","FUID5","FUID6"],"client_cert_chain_fuids":[],"subject":"CN=www.example.com","issuer":"CN=Example CA,O=Example,C=US"}
{"ts":1598377391.938343,"log":"x509.log","id":"FUID3","certificate.version":3,"certificate.serial":"123456789ABCDEF","certificate.subject":"CN=www.example.com","certificate.issuer":"CN=Example CA,O=Example,C=US","certificate.not_valid_before":1590969600,"certificate.not_valid_after":1625140800,"certificate.key_alg":"rsaEncryption","certificate.sig_alg":"sha256WithRSAEncryption","certificate.key_type":"rsa","certificate.key_length":2048,"certificate.exponent":"65537","san.dns":["www.example.com","example.com","*.example.com"],"basic_constraints.ca":false}
{"ts":1254722768.219663,"uid":"UID6","log":"smtp.log","id.orig_h":"223.01.20.200","id.orig_p":1470,"id.resp_h":"74.53.140.153","id.resp_p":25,"trans_depth":1,"helo":"HELO","mailfrom":"sender@example.com","rcptto":["recipient@example.com"],"date":"Mon, 5 Oct 2009 11:36:07 +0530","from":"\"Sender Name\" <sender@example.com>","to":["<recipient@example.com>"],"msg_id":"<msgid@example.com>","subject":"SMTP Test","last_reply":"250 OK","path":["74.53.140.153","223.01.20.200"],"user_agent":"Microsoft Office Outlook 12.0","tls":false,"fuids":["FUID7","FUID8","FUID9"]}
{"ts":"2020-09-16T13:08:58.933098Z","uid":"UID7","log":"ssh.log","id.orig_h":"101.0.2.3","id.orig_p":39550,"id.resp_h":"203.0.113.2","id.resp_p":22,"version":2,"auth_success":true,"auth_attempts":2,"direction":"OUTBOUND","client":"SSH-2.0-OpenSSH_7.4p1","server":"SSH-2.0-OpenSSH_8.0","cipher_alg":"chacha20-poly1305@openssh.com","mac_alg":"umac-64-etm@openssh.com","compression_alg":"none","kex_alg":"curve25519-sha256","host_key_alg":"ssh-ed25519","host_key":"AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99","hasshVersion":"1.0","hassh":"HASH_CLIENT","hasshServer":"HASH_SERVER","cshka":"ssh-ed25519-cert-v01@openssh.com"}
{"ts":"2020-09-23T00:24:36.395445Z","log":"pe.log","id":"FUID10","machine":"AMD64","compile_ts":"2020-09-19T00:10:08.000000Z","os":"Windows XP x64 or Server 2003","subsystem":"WINDOWS_GUI","is_exe":true,"is_64bit":true,"uses_aslr":true,"uses_dep":true,"uses_code_integrity":false,"uses_seh":true,"has_import_table":true,"has_export_table":false,"has_cert_table":true,"has_debug_data":true,"section_names":[".text",".rdata"]}
{"ts":"2020-10-02T04:14:39.120138Z","uid":"UID8","log":"dhcp.log","id.orig_h":"101.0.2.4","id.orig_p":68,"id.resp_h":"101.0.2.254","id.resp_p":67,"proto":"udp","service":"dhcp","duration":0.337,"orig_bytes":0,"resp_bytes":604,"conn_state":"SHR","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"^d","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":2,"resp_ip_bytes":660,"ip_proto":17}
{"ts":"2020-10-08T00:29:07.977170Z","uid":"UID9","log":"ntp.log","id.orig_h":"101.0.2.3","id.orig_p":38461,"id.resp_h":"208.79.89.249","id.resp_p":123,"version":4,"mode":3,"stratum":0,"poll":1,"precision":1,"root_delay":0,"root_disp":0,"ref_id":"\\x00\\x00\\x00\\x00","ref_time":"1970-01-01T00:00:00.000000Z","org_time":"1970-01-01T00:00:00.000000Z","rec_time":"1970-01-01T00:00:00.000000Z","xmt_time":"2020-10-08T00:29:07.215586Z","num_exts":0}
{"ts":1507562478.112879,"uid":"UID10","log":"dce_rpc.log","id.orig_h":"123.0.2.20","id.orig_p":49282,"id.resp_h":"101.0.2.20","id.resp_p":445,"rtt":0.0003,"named_pipe":"\\pipe\\lsass","endpoint":"samr","operation":"SamrConnect5"}
{"ts":1607009493.733304,"uid":"UID11","log":"irc.log","id.orig_h":"101.0.2.5","id.orig_p":52856,"id.resp_h":"223.01.20.200","id.resp_p":6667,"command":"NICK","value":"usernick"}
{"ts":1463256456.051759,"uid":"UID12","log":"ldap.log","id.orig_h":"223.01.20.20","id.orig_p":25936,"id.resp_h":"198.51.100.2","id.resp_p":3268,"message_id":3,"version":3,"opcode":"bind simple","result":"success","object":"CN=REDACTED,OU=Users,OU=Accounts,DC=example,DC=com","argument":"REDACTED"}
{"ts":1670520068.267888,"uid":"UID13","log":"postgresql.log","id.orig_h":"123.0.2.223","id.orig_p":39910,"id.resp_h":"52.200.36.167","id.resp_p":5432,"frontend":"ssl_request","backend":"ssl_reply","backend_arg":"S","success":true}
{"ts":1692198386.837988,"uid":"UID14","log":"quic.log","id.orig_h":"82.239.54.117","id.orig_p":53727,"id.resp_h":"110.213.53.115","id.resp_p":443,"version":"1","client_initial_dcid":"95412c47018cdfe8","server_scid":"d5412c47018cdfe8","server_name":"api.example.com","client_protocol":"h3","history":"ISisH"}
{"ts":1607353272.791158,"uid":"UID15","log":"rdp.log","id.orig_h":"101.0.2.6","id.orig_p":59758,"id.resp_h":"101.0.2.7","id.resp_p":3389,"cookie":"test","result":"encrypted","security_protocol":"HYBRID","cert_count":0}
{"ts":"2020-12-07T05:14:54.202099Z","src":"101.0.2.8","log":"traceroute.log","dst":"203.0.113.5","proto":"udp"}
{"ts":1607993758.290539,"uid":"UID16","log":"tunnel.log","id.orig_h":"101.0.2.9","id.orig_p":3074,"id.resp_h":"40.84.25.61","id.resp_p":65444,"tunnel_type":"Tunnel::TEREDO","action":"Tunnel::DISCOVER"}
{"ts":1607568264.410681,"uid":"UID17","log":"dpd.log","id.orig_h":"123.0.2.20","id.orig_p":50540,"id.resp_h":"184.168.176.1","id.resp_p":443,"proto":"tcp","analyzer":"SSL","failure_reason":"Invalid version late in TLS connection. Packet reported version: 21588"}
{"ts":"2021-01-03T00:16:22.694616Z","log":"software.log","host":"123.0.2.21","software_type":"HTTP::BROWSER","name":"Windows-Update-Agent","version.major":10,"version.minor":0,"version.minor2":10011,"version.minor3":16384,"version.addl":"Client","unparsed_version":"Windows-Update-Agent/10.0.10011.16384 Client-Protocol/2.0"}
{"ts":"2021-01-04T04:59:21.582639Z","uid":"UID18","log":"weird.log","id.orig_h":"123.0.2.22","id.orig_p":51020,"id.resp_h":"40.71.25.43","id.resp_p":8080,"name":"bad_HTTP_request","notice":false,"peer":"so16-enp0s8-1"}
{"ts":"2021-01-04T01:19:15.713689Z","log":"reporter.log","level":"Reporter::INFO","message":"BPFConf filename set"}