Gravwell and Windows Event Logging

Update

This post uses the xml parser module to evaluate windows logs. We have since released the winlog module, which you can reference here: winlog

TL;DR

We are going to dive into Windows and show how to get logs flowing into Gravwell in under 5 minutes with the WinEvent ingester. Using the Windows queries we will audit login behavior, RDP usage, some Windows Defender, and identify when Bob from accounting is copying sensitive financial data to external storage devices. Also, Taylor Swift is involved; don’t panic, just stay with me.

Overview

This Gravwell post is all about the wild world of Windows Event logging and analytics. Both Unix and Windows provide standardized central logging facilities; however, the structure and format of the stored logs are dramatically different. Syslog and most other logging systems with roots in Unix approach logging as an unstructured stream: a log entry is a string of text, no more, no less (we are going to ignore journald and its binary madness). Windows, however, logs all events in fully-formed XML and the logging system is integrated into the operating system itself.  We should also note that logging in Windows is… less than ideal.  If you are coming from the Unix world, throw out all your assumptions; things are different here.

Our goal for this post is to show how to quickly deploy and configure Gravwell ingesters to enable robust and secure Windows Event logging. We will show how to install the Gravwell ingesters, explain federation and isolation, and investigate some alternate data streams that can help system administrators better understand the health of a fleet of Windows machines and help security professionals identify and clean up breaches. Deploying and configuring Gravwell is easy and takes minutes, the Windows Ingesters are no different.

Gravwell Ingest Architecture

The Gravwell ingest API and our core ingesters are open source under the very liberal BSD 2-clause license, which allows you to inspect our ingesters or very quickly write your own. The ingesters and ingest API are available on Github. Fundamentally, ingesting data into Gravwell only requires two things: an array of bytes and a time stamp. As luck would have it, Windows Events have both those things, so let’s get to ingesting some Windows Events.

The Gravwell WinEvent Agent

The Windows Events API can perform filtering and pre-selection on-host, and the Gravwell WinEvent ingester can utilize these filters to attach to and tag streams on host. The pre-filtering can be useful, as the nature of Windows Events (XML) means that even simple things like “Bob logged into machine X” end up being pretty large chunks of data. Gravwell will happily eat these large streams (we have benchmarked a single indexer at over 1.25 million events per second at over 250MB per second), but at the end of the day you will need to store that data on something, which ultimately incurs cost (whether it be hardware or cloud storage). When Gravwell is running flat out, a single indexer can consume over 10TB per day of small log entries and upwards of 30TB per day of large log entries (like Windows Events); long story short, Gravwell will eat the elephant, but you may not want to store the elephant.

The Gravwell WinEvent agent runs as a background service and is designed to tolerate intermittent connectivity with Gravwell indexers. Intermittent connectivity may be due to mobile devices, or poor network connections such as a satellite office on a flaky VPN. The Gravwell WinEvent agent can deal with intermittent connectivity in one of two ways; the ingester can either rely on the Windows event storage system to cache event data or it can engage its own internal caching system. The Gravwell internal cache can be extremely useful as a second level of defense against sophisticated attackers that may alter events stored in the EventLog. The Gravwell internal cache also provides coverage when devices may be away from the home office for long periods of time and it is desirable to store hundreds or even thousands of megabytes of logs. To enable the Gravwell ingest cache, define the Ingest-Cache-Path parameter in the config file and point it at a writable location. By default, it points to C:\Program Files\gravwell\events.cache, if you modified the installation location of the ingester during installation be sure to update any paths in the config file.

Installing the Gravwell WinEvent Service

The Gravwell WinEvent service is packaged as a Microsoft Installer (MSI) that installs like any other application. The agent runs as a service, which will require Administrative privileges to install and configure. You can download the installer here. For a full description of the installation procedure, visit our documentation.

Installing the agent on more than a few machines should most likely be performed via a Domain Controller and group policy. Deploying software with Group Policy is beyond the scope of this post, but an abundance of information can be found via Microsoft support resources and independent blogs. A point worth mentioning is that the Gravwell Agent is a static binary and does not import any foreign code. Deployment can be as simple as pushing the executable and configuration file, then starting the service; there is no DLL or Plugin Hell here.