Overview

This article explains how to connect your Windows Event Logs to Hunters.

  • Windows Event Logs can be collected with different agents and methods. Some common products used for collection are NXLog, Winlogbeat, and the Fluentd WEL plugin.

  • The collection can be done on each device separately, or in a centralized location, after forwarding the logs to the centralized location using the native Windows Event Forwarding, or remote WMI collection that some products like NXLog and Splunk offer.

  • Lastly, the data needs to be uploaded to an S3 bucket to be shared with Hunters. This can be done with data collectors such as Fluentd and LogStash.

Recommended Auditing Policy

It is strongly advised to use Microsoft's Audit Policy Recommendations in order to ensure that the proper events are logged in your Windows environment. Please note the distinction in recommendations between a standard host versus a Domain Controller (denoted as DC). For the full audit policy recommendations, please refer to Microsoft's documentation.

Supported Formats

The Hunters platform reads the event logs from an S3 Bucket.
The following considerations should be applied to the logs shared:

  • Hunters currently supports Security and PowerShell event logs.

  • The logs should be stored in the bucket in nd-json format.

  • The Message field should be kept intact and not parsed or manipulated - field delimeters being \t\t, group delimiters being \r\n\r\n, etc.

  • Make sure that your log collector logs timestamps with timezones, so it will be possible to get the UTC timestamp of the events.

  • For servers in different languages, make sure that your log collector saves the messages in en-US encoding.

Setup Guide - Windows Event Log Collectors

Fluentd Windows Event Log Plugin

Prerequisites

  • The Fluentd agent works on most modern Windows versions, on 64-bit versions only. For earlier versions, a different solution (such as the native Windows Event Forwarding) will need to be configured.

  • Make sure all reporting hosts have NTP set up, for the logs to have accurate timestamps.

Step 1 - Install the Fluentd agent on all devices

Download the newest Fluentd Windows agent (td-agent v4) from here.
Install the agent as a local administrator on all hosts where Windows Event Logs collection is planned.
Running the .msi installer should automatically register and start Fluentd as a Windows service. If not, follow the instructions here.

Step 2 - Configure the Windows Event Log Input Plugin

The Fluentd configuration file is found in C:\opt\td-agent\etc\td-agent\td-agent.conf.
Full documentation of the Windows Event Log input plugin can be found here.
Recommended configuration:

<source>
  @type windows_eventlog2
  @id windows_eventlog2
  channels security
  read_existing_events false
  read_interval 2
  tag winevt.raw
  <storage>
    @type local             # @type local is the default.
    persistent true         # default is true. Set to false to use in-memory storage.
    path C:\opt\td-agent\tmp\storage.json
  </storage>
</source>
CODE

Step 3 - Forward the logs to the desired destination

After configuring the WEL collection, you can now forward the logs to the desired destination.
One option is forwarding the logs directly to an S3 bucket, as detailed in the below section.
Another option is to forward the logs to a centralized location using other output plugins such as out_http or out_forward, and from there forward the logs to the S3 bucket.

Winlogbeat

Guide by Elastic
Please make sure the log format follows the guidelines outlined in Supported Log Formats.
For additional questions, please refer to the Hunters support team.

NXLog

Guide by NXLog
Please make sure the log format follows the guidelines outlined in Supported Log Formats.
For additional questions, please refer to the Hunters support team.

NXLog Example Configuration

Please find below a working configuration for NXLog agent to use for the purpose of forwarding Windows Event Logs to Fluentd or Logstash.

# Windows Event Logs

define ROOT C:\Program Files\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf
define LOGDIR %ROOT%\data
define LOGFILE '%LOGDIR%/nxlog.log'

# Defining Output Host & Port
define OUTPUT_DESTINATION_ADDRESS <destination_host>
define DNS_OUTPUT_DESTINATION_PORT <destination_port>

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension syslog>
    Module xm_syslog
</Extension>

<Extension json>
    Module xm_json
</Extension>

<Input in_wel>
    Module	im_msvistalog
    ReadFromLast TRUE
</Input>

<Output out_wel>
    Module  om_tcp
    Host    %WEL_OUTPUT_DESTINATION_ADDRESS%
    Port    %WEL_OUTPUT_DESTINATION_PORT%
    Exec $SyslogFacilityValue = 22;to_syslog_snare();
</Output>

<Route wel_to_aws_nlb>
    Path        in_wel => out_wel
</Route>

# Panic SOFT will cause an exception to be thrown at the place of the panic/assertion.
Panic Soft

# END OF FILE
CODE

Windows Event Collector (WEC)

Windows offers the Windows Event Forwarding solution for centralized Windows event logging. This solution includes configuring WEF (Windows Event Forwarding) on all wanted devices, that will ship their Windows event logs to WEC (Windows Event Collector) servers.

A common logging architecture consists of WEF clients sending their logs to WEC servers, that have a log shipper installed (such as Fluentd or Winlogbeat) that ships the Windows event logs to a cloud storage solution such as AWS S3.

Note that there are many different tweaks and configurations available for WEF that should be taken into account in the logging pipeline. For example, the forwarded events can be written to several different locations and channels on the WEC server, as described in this blog post by Elastic. This will require configuring your log shipper to subscribe to the correct channel.

In addition, the event message might require configuration changes to be parsed correctly. Some log shippers require the WEF ContentFormat to be RenderedText (the default) while others might require changing it to Events, as described in this thread in the Microsoft forums.

Setup Guide - Windows Event Log Shippers

Fluentd S3 Plugin

Step 1 - Set up an S3 bucket

Configure an IAM role attached to a policy that will allow the Fluentd agents to write the logs. The policy needs to allow List, Read and Write operations on the bucket.

Step 2 - Configure the S3 Output Plugin

The Fluentd configuration file is found in C:\opt\td-agent\etc\td-agent\td-agent.conf.
If you're using Fluentd to perform the WEL collection and the S3 forwarding on the same host, you only need to append an output configuration to the same configuration file in which you added the windows_eventlog2 source (as explained here). Below is a simple configuration example. For more complex use-cases such as use of AssumeRole credentials, refer to the S3 plugin documentation.

<match winevt.raw>
  @type s3

  aws_key_id YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET_KEY
  s3_bucket YOUR_S3_BUCKET_NAME
  s3_region YOUR_S3_BUCKET_REGION
  path "winevtlogs/#{hostname}/%Y/%m/%d/"
  <format>
    @type json
  </format>
  <buffer tag,time>
    @type file
    path C:\opt\td-agent\s3.log
    timekey 300 # 5 minutes partition
    timekey_wait 1m
    timekey_use_utc true # use utc
    chunk_limit_size 256m
  </buffer>
</match>
CODE

If you plan to forward Windows Event Logs to a centralized location using Fluentd, and then perform the S3 upload in the centralized location, you will need to add additional outputs on the collecting devices and matching inputs on the centralized forwarder.

Step 3 - Restart the Fluentd service

After editing the configuration file, restart the service (go to Control Panel -> System and Security -> Administrative Tools -> Services, right click on Fluentd Windows Service, click Restart).

Step 4 - Verify files written to S3

  1. Browse to the destination S3 bucket.

  2. 5-10 minutes after the Fluentd service was restarted, .gz files should already be written to the bucket.

  3. Download the latest file and open it.

  4. Make sure it is in the nd-json format.

Step 5 - Set up read permissions for the S3 bucket

Configure a role and policy that lets Hunters download objects from the S3 bucket, as described in the Access to Cloud Storage chapter.

Step 6 - Contact Hunters' representative

Contact your account manager to start ingesting this data into the Hunters platform.

LogStash

Please refer to the LogStash S3 Plugin Documentation.

Make sure that the log format stays intact and is written to the bucket as nd-json.