Skip to content
This repository was archived by the owner on Feb 21, 2021. It is now read-only.

Etw.md

Latest commit

 

History

History
110 lines (70 loc) · 6.83 KB

Etw.md

File metadata and controls

110 lines (70 loc) · 6.83 KB

Event Tracing for Windows (ETW)

Event Tracing for Windows (ETW) sessions allow a user to listen to event providers that capture kernel and application events. One such provider is the "Microsoft-Windows-Kernel-Network" provider, which records TCP-related events. By using this input option, you can, for example, start an ETW session using the "Microsoft-Windows-Kernel-Network" provider and use Real-Time KQL to capture and process all the TCP-related events generated by this provider.

You can watch a demonstration of using Real-Time KQL for Etw here.

Jump To:

Real-Time Monitoring

Starting ETW Sessions

In order to leverage the high-volume events emitted by ETW providers, you must first start an ETW trace session. You can do this using logman.

Example usage - Starting an ETW Trace:

logman.exe create trace tcp -rt -nb 2 2 -bs 1024 -p {7dd42a49-5329-4832-8dfd-43d979153a88} 0xffffffffffffffff -ets

Example breakdown:

  • create trace : create an event trace data collector
  • tcp : name the trace session "tcp"
  • -rt : run the trace session in real-time mode
  • -nb 2 2 : specifies the number of trace session buffers, where the first 2 is the minimum and the second 2 is the maximum
  • -bs 1024 : use a trace session buffer size of 1024 kb
  • -p {7dd42a49-5329-4832-8dfd-43d979153a88} : specifies which Event Trace provider to enable by its GUID (in this case, the "Microsoft-Windows-Kernel-Network" provider)
  • -ets : sends commands to trace sessions directly without saving or scheduling

Example usage - Stopping an ETW Trace:

logman.exe stop tcp -ets

Example breakdown

  • stop : stop a data collector
  • tcp : name of the trace session
  • -ets- : sends command to trace sessions directly without saving or scheduling

Real-Time KQL and ETW

Once you have an ETW trace session up and running, you can attach Real-Time KQL to that session and process any events that come in.

Example usage:

RealtimeKql etw --session=tcp --adxcluster=CDOC.kusto.windows.net --adxdatabase=GeorgiTest --adxtable=EtwTcp --adxdirect --adxreset

Example breakdown:

  • --session=tcp : name of the ETW session to attach to (in this case, we are continuing with the "Microsoft-Windows-Kernel-Network" provider example)
  • --adxcluster=CDOC.kusto.windows.net --adxdatabase=GeorgiTest --adxtable=EtwTcp : ingest all results to the "EtwTcp" table in the "GeorgiTest" database in the "CDOC.kusto.windows.net" Azure Data Explorer (ADX) cluster
  • --adxdirect : use direct ingestion instead of the default queued ingestion
  • --adxreset : if the "EtwTcp" table already exists, reset it

Historical Processing

Previously Recorded ETL Trace Log (.etl files)

You can also use Real-Time KQL to process pre-recorded ETL Trace log files (*.etl files).

Example usage:

RealtimeKql etw --file=*.etl --adxcluster=CDOC.kusto.windows.net --adxdatabase=GeorgiTest --adxtable=EtwTcp --adxdirect --adxreset

Example breakdown:

  • --file=*.etl : file pattern to filter files by (in this case, only look for files that match the "*.etl" pattern)
  • --adxcluster=CDOC.kusto.windows.net --adxdatabase=GeorgiTest --adxtable=EtwTcp : ingest all results to the "EtwTcp" table in the "GeorgiTest" database in the "CDOC.kusto.windows.net" Azure Data Explorer (ADX) cluster
  • --adxdirect : use direct ingestion instead of the default queued ingestion
  • --adxreset : if the "EtwTcp" table already exists, reset it

ETW Options Overview

You can also run RealTimeKql Etw --help from an Administrator Command Prompt to get this same overview of your options:

Usage: RealTimeKql.exe Etw [options]

Options:
  -?|-h|--help                                 Show help information
  -s|--session <value>                         Name of the ETW Session to attach to. eg, --session=tcp. tcp is the name of the session started using logman or such tools.
  -f|--file <value>                            File pattern to filter files by. eg, --filter=*.etl
  -q|--query <value>                           Optional: KQL filter query file that describes what processing to apply to the events on the stream. It uses a subset of Kusto Query Language, https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/
  -oc|--outputconsole                          Log the output to console.
  -oj|--outputjson <value>                     Write output to JSON file. eg, --outputjson=FilterOutput.json
  -bscs|--blobstorageconnectionstring <value>  Azure Blob Storage Connection string. Optional when want to upload as JSON to blob storage.
  -bsc|--blobstoragecontainer <value>          Azure Blob Storage container name. Optional when want to upload as JSON to blob storage.
  -ad|--adxauthority <value>                   Azure Data Explorer (ADX) authority. Optional when not specified microsoft.com is used. eg, --adxauthority=microsoft.com
  -aclid|--adxclientid <value>                 Azure Data Explorer (ADX) ClientId. Optional ClientId that has permissions to access Azure Data Explorer.
  -akey|--adxkey <value>                       Azure Data Explorer (ADX) Access Key. Used along with ClientApp Id
  -ac|--adxcluster <value>                     Azure Data Explorer (ADX) cluster address. eg, --adxcluster=CDOC.kusto.windows.net
  -ad|--adxdatabase <value>                    Azure Data Explorer (ADX) database name. eg, --adxdatabase=TestDb
  -at|--adxtable <value>                       Azure Data Explorer (ADX) table name. eg, --adxtable=OutputTable
  -ar|--adxreset                               The existing data in the destination table is dropped before new data is logged.
  -ad|--adxdirect                              Default upload to ADX is using queued ingest. Use this option to do a direct ingest to ADX.