diff --git a/README.md b/README.md index 411cc16..f1c5493 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,10 @@ The project is organized to facilitate easy compilation, testing, and integratio - WHQL-certified on the latest Windows operating systems. - Compatible with Qualcomm tools like QUTS, QXDM, PCAT, and more. - Compatible with terminal emulators like PuTTY and Tera Term. + - **Always-on driver logging** (ETW AutoLogger on Windows, ftrace instance on + Linux) with a one-command "Save Log" action — see + [`docs/always-on-logging-plan.md`](./docs/always-on-logging-plan.md) and + [`tools/logging/`](./tools/logging/). ## Repository Structure @@ -16,6 +20,8 @@ The project is organized to facilitate easy compilation, testing, and integratio ├─ docs/ # Architecture diagrams and design documents ├─ src/ # Qualcomm USB kernel driver for windows and linux platform ├─ examples/ # samples scripts +├─ tools/ +│ └─ logging/ # Always-on logging tools (Windows + Linux) ├─ README.md # This file └─ ... # Other files and directories ``` diff --git a/docs/always-on-logging-plan.md b/docs/always-on-logging-plan.md new file mode 100644 index 0000000..4f9b966 --- /dev/null +++ b/docs/always-on-logging-plan.md @@ -0,0 +1,377 @@ +# Always-On Logging for Qcom USB Kernel Drivers — Implementation Plan + +## 1. Problem Statement + +Today, logging in the kernel open-source drivers (Windows WPP, Linux `printk`) is **opt-in**: + +- End users / field engineers must install a debug build or manually enable trace + sessions (`logman`, `tracelog`, `trace-cmd`, `echo 1 > /sys/.../parameters/debug`, + dynamic_debug, etc.) **before** the issue is reproduced. +- Most issues are reported *after* the fact. By the time the trace is enabled, the + original failing sequence is already gone. +- Each driver (qcfilter, qcusbnet, qdbusb, qcwdfserial on Windows; qcom_serial, + qcom_usb, qcom_usbnet on Linux) has its own enable/collect procedure, so + customers must learn many workflows. This lengthens every debug cycle. + +**Goal:** every shipping driver must have an **always-on, bounded ring-buffer +trace** session running from boot. When an application wants a snapshot, a single +"Save Log" action rotates the buffer to a file and starts a new one. No driver +rebuild, no registry/command changes required in the field. + +## 2. High-Level Design + +| Aspect | Windows (WPP / ETW) | Linux (ftrace) | +|-------------------------|------------------------------------------------------------------|-------------------------------------------------------------| +| Backing buffer | ETW kernel-mode session, **FileMode = Circular** | ftrace instance with bounded `buffer_size_kb` (per-cpu) | +| Always-on mechanism | **AutoLogger** registry keys started at `SERVICE_SYSTEM_START` | systemd unit `qcomlogd.service` enabled at boot | +| Snapshot ("Save ETL") | `StopTrace` → copy/rename `.etl` → `StartTrace` (<200 ms gap) | `snapshot` + copy from `/sys/kernel/tracing/instances/...` | +| Trigger API | User-mode service `QcomLogSvc` + named pipe + CLI | User-mode helper `qcomlogd` + UNIX socket + CLI | +| Providers | Existing per-driver WPP GUIDs, grouped under one AutoLogger | New `qcom_*` tracepoints; `QC_LOG_*` macros redirect to them | +| Output format | `qcomdrv-YYYYMMDDTHHMMSS.etl` | `qcomdrv-YYYYMMDDTHHMMSS.tar.gz` (trace + symbols) | + +Design principle: **no extra code in hot paths**. Existing `QCSER_DbgPrint` / +`QC_LOG_*` macros stay. We only change (a) how/when tracing is started, +(b) how snapshots are captured, (c) keep the buffer bounded and circular so +impact is negligible. + +## 3. Windows Implementation + +### 3.1 Current state (survey) + +| Driver module | Folder | INF(s) | WPP header | EVENT_TRACING release default | +|---------------|--------------|--------------------------------|-------------------|-------------------------------| +| qcfilter | `src/windows/filter` | `qcfilter.inf` | `qcfilterwpp.h` | yes | +| qcusbnet | `src/windows/ndis` | `qcwwan.inf` | `MPWPP.h` | yes | +| qdbusb | `src/windows/qdss` | `qdbusb.inf` | `QDBWPP.h` | yes | +| qcwdfserial | `src/windows/wdfserial` | `qcwdfser.inf`, `qcwdfmdm.inf` | `QCWPP.h` | yes | +| qcadb | `src/windows/qcadb` | `qcadb.inf` (NullDriver) | n/a | n/a | + +Observed: all four coded drivers already define WPP control GUIDs and are built +with `EVENT_TRACING`, `true` for every config +(Debug/Release × x64/ARM64/…). What's missing: an AutoLogger config and a +standard "save" workflow. + +GUIDs to collect (qcwdfserial known today — others to be extracted in Step 1): + +| Driver | WPP Control GUID | +|------------|---------------------------------------------------| +| qcwdfser | `{5F02CA82-B28B-4a95-BA2C-FBED52DDD3DC}` (QCUSB) | +| qcfilter | _TBD — read from `qcfilterwpp.h`_ | +| qcusbnet | _TBD — read from `MPWPP.h`_ | +| qdbusb | _TBD — read from `QDBWPP.h`_ | + +### 3.2 Unified provider catalogue + +- New file `src/windows/common/QcomTraceProviders.h` — single header listing + every driver's GUID and default keyword mask, consumed by installers and the + user-mode service. +- New file `tools/windows/QcomDrivers.wprp` — Windows Performance Recorder + profile enumerating the same providers, so engineers can trigger captures + from WPR/xperf when needed. + +### 3.3 AutoLogger configuration (always-on, circular) + +Add an `AddReg` stanza in every driver INF that writes to the shared +`QcomDrivers` AutoLogger session. Example (to be mirrored in `qcfilter.inf`, +`qcwwan.inf`, `qdbusb.inf`): + +```ini +[QCSerial_AddReg_AutoLogger] +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers",Start,0x00010001,1 +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers",GUID,0x00000000,"{A1B2C3D4-0000-0000-0000-QCOMDRIVERS}" +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers",LogFileMode,0x00010001,0x00000102 ; EVENT_TRACE_FILE_MODE_CIRCULAR +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers",FileName,0x00000000,"%SystemRoot%\Tracing\QcomDrivers.etl" +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers",MaxFileSize,0x00010001,64 ; MB circular cap +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers",BufferSize,0x00010001,64 ; KB per buffer +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers",MinimumBuffers,0x00010001,8 +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers",MaximumBuffers,0x00010001,16 + +; one sub-key per provider GUID +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers\{5F02CA82-B28B-4a95-BA2C-FBED52DDD3DC}",Enabled,0x00010001,1 +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers\{5F02CA82-B28B-4a95-BA2C-FBED52DDD3DC}",EnableLevel,0x00010001,4 ; TRACE_LEVEL_INFORMATION +HKLM,"System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers\{5F02CA82-B28B-4a95-BA2C-FBED52DDD3DC}",MatchAnyKeyword,0x0001000A,0xFFFFFFFFFFFFFFFF +``` + +Rationale: +- `LogFileMode=0x00000102` = `EVENT_TRACE_FILE_MODE_CIRCULAR | + EVENT_TRACE_USE_GLOBAL_SEQUENCE` → a bounded `.etl` that wraps in place. +- `MaxFileSize = 64 MB` → hours of INFO or days of WARN/ERR at typical rates. +- Keyword mask per provider means field teams can narrow flags via registry + without a rebuild. Chatty data-path keywords + (`WPP_DRV_MASK_TDATA`, `WPP_DRV_MASK_RDATA`) default **off** so the circular + buffer keeps more history. + +### 3.4 "Save ETL" — user-mode service `QcomLogSvc` + +New Windows service (C++17, no external deps) packaged with the drivers. + +Responsibilities: +1. Register the `QcomDrivers` AutoLogger if it's missing (self-heal) and start + it at service start if it's not already running (handles pre-AutoLogger OS + snapshots / upgrade cases). +2. Expose a **single public API `SaveSnapshot(outputPath, options)`** over: + - Named pipe `\\.\pipe\QcomLogSvc` (JSON request/response) + - CLI `qcomlog.exe save [--out PATH] [--bundle]` + - COM interface `IQcomLogSvc` for in-proc apps (e.g., QUTS, QDS) + - PowerShell cmdlet `Save-QcomDriverLog` (wraps CLI) +3. Snapshot flow: + 1. `ControlTrace(..., EVENT_TRACE_CONTROL_FLUSH)` to flush pending buffers. + 2. `ControlTrace(..., EVENT_TRACE_CONTROL_STOP)` — closes `.etl`. + 3. Move/rename target file: + `%SystemRoot%\Tracing\QcomDrivers.etl` → + `\QcomDrivers-YYYYMMDDTHHMMSSZ.etl` + 4. `StartTrace(...)` with the cached AutoLogger template to resume logging + immediately. **Target gap < 200 ms** (measurable by pairing boundary + markers emitted by the service's own provider). + 5. If `--bundle`: also collect `setupapi.dev.log`, `devnodeclean` output, + `pnputil /enum-drivers`, driver INFs, relevant TMFs and produce + `QcomDrivers-YYYYMMDDTHHMMSSZ.zip`. +4. Telemetry: event-log source `QcomLogSvc` records snapshot count, bytes, + duration, last error. +5. Security: + - Service runs as `LocalSystem` (needed to control the kernel ETW session). + - Pipe ACL: `LocalAdmins` and members of a new local group + `QcomDriverDiagnostics`, so non-admin helpdesk accounts can be enrolled. + - Optional `--allow-user-save` registry knob. + +### 3.5 Driver-side changes (minimal) + +- Confirm every release build defines `EVENT_TRACING` (already true — verified + in each `*.vcxproj`). +- Ensure each driver's `.tmh` generation places TMF files into the package so + `tracefmt.exe` can decode `.etl` on a different machine without PDBs. Install + them to `%SystemRoot%\Tracing\Qcom\.tmf`. +- Re-audit every `QCSER_DbgPrint` and `QCSER_DbgPrintX` call so that high-rate + per-byte traces are tagged with `WPP_DRV_MASK_TDATA`/`WPP_DRV_MASK_RDATA` + keywords only; AutoLogger default keyword mask excludes these. +- Emit a small `Boundary` event at trace start (driver load) and from + `QcomLogSvc` before/after the Stop/Start sequence so post-processing can + stitch rotated `.etl`s chronologically. + +### 3.6 Tooling deliverables (Windows) + +- `tools/windows/QcomDrivers.wprp` — WPR/xperf profile. +- `tools/windows/Save-QcomDriverLog.ps1` — thin wrapper calling the service. +- `tools/windows/Decode-Etl.ps1` — invokes `tracefmt.exe` with the shipped + TMFs to convert `.etl` → `.txt` on any machine. +- `tools/windows/QcomLogSvc/` — service source (MSBuild project). + +## 4. Linux Implementation + +### 4.1 Current state (survey) + +All three Linux modules log with `printk` via wrapper macros: + +| Module | Source files | Log macros | Gate | +|-----------------|---------------------------------------------|-----------------------------------------|----------------------------------------------------------------| +| qcom_serial | `qcom_serial.c/.h` | `DBG`, `GobiDBG`, `GOBI_DBG` | `debug` module_param | +| qcom_usb | `qcom_usb_main.c`, `qtiDevInf.h` | `QC_LOG_DBG/INFO/WARN/ERR/...` | per-device `->debug` + global `debug_g`, and `->logLevel` | +| qcom_usbnet | `qcom_usbnet_main.c`, `qmidevice.c`, `qmi.c`, `qtiDevInf.h` | `QC_LOG_*`, `QC_LOG_AGGR` | per-device `->debug` (1 or 0xFF) + `->logLevel` + `debug_aggr` | + +Nothing is captured unless the user explicitly sets `debug=1` / changes +`logLevel`, raises `dmesg` buffer size, or keeps a `journalctl -f` session +open. There are no tracepoints, no debugfs endpoints, no kfifo ring buffers. + +### 4.2 Target mechanism: dedicated ftrace instance + +Linux already ships a production-grade bounded ring buffer. A per-package +instance lives at `/sys/kernel/tracing/instances//` with its own +per-CPU buffer, `trace_pipe`, `snapshot`, `buffer_size_kb`, and event enable +mask. We provision a single instance `qcom_drivers`: + +``` +/sys/kernel/tracing/instances/qcom_drivers/ + buffer_size_kb = 4096 # per CPU → ≈ 32 MB on an 8-core laptop + tracing_on = 1 + events/qcom/enable = 1 + events/qcom/qcom_log_dbg/enable = 0 # default off, saves buffer space + options/overwrite = 1 # circular (overwrite oldest) +``` + +### 4.3 Tracepoint conversion + +Add a common header `src/linux/common/trace/qcom_drv_trace.h` that defines +the tracepoint once per driver (idiomatic `CREATE_TRACE_POINTS`): + +```c +TRACE_EVENT(qcom_log, + TP_PROTO(const char *drv, int level, const char *func, int line, + struct va_format *vaf), + TP_ARGS(drv, level, func, line, vaf), + TP_STRUCT__entry( + __string(drv, drv) + __field(int, level) + __string(func, func) + __field(int, line) + __vstring(msg, vaf->fmt, vaf->va) + ), + TP_fast_assign( + __assign_str(drv, drv); + __entry->level = level; + __assign_str(func, func); + __entry->line = line; + __assign_vstr(msg, vaf->fmt, vaf->va); + ), + TP_printk("[%s] %s:%d L%d %s", + __get_str(drv), __get_str(func), __entry->line, + __entry->level, __get_str(msg)) +); +``` + +Redirect `QC_LOG(...)` in `qtiDevInf.h` so that in addition to `printk` it +emits the tracepoint unconditionally (cost ≈ a few ns when the tracepoint is +disabled, the usual ftrace static-key pattern): + +```c +#define QC_LOG(KERN_LVL, DEV, fmt, lvl, ver, ...) do { \ + struct va_format _vaf = { .fmt = fmt }; \ + trace_qcom_log(DEV ? DEV->mdeviceName : "qcom", \ + lvl, __func__, __LINE__, &_vaf, ##__VA_ARGS__); \ + if (DEV && (DEV->debug == 1 || DEV->debug == 0xFF) && \ + (DEV->logLevel <= lvl)) \ + printk(KERN_LVL "%s: %s:%d %s " fmt, \ + DEV->mdeviceName, __func__, __LINE__, ver, ##__VA_ARGS__); \ +} while (0) +``` + +Key properties: +- Tracepoints are **always on** into the circular ring regardless of the + `debug` / `logLevel` knobs — that's what makes logging always-on. +- `printk` behaviour is preserved for users who rely on `dmesg`, but becomes + essentially a fallback. +- `DBG()` / `GobiDBG()` / `GOBI_DBG()` in `qcom_serial` get thin wrappers that + funnel through the same tracepoint. + +### 4.4 "Save log" — user-mode helper `qcomlogd` + +Small C daemon + CLI packaged with the drivers (`tools/linux/qcomlogd/`). + +- `systemd` unit `qcomlogd.service` (with `After=sys-kernel-tracing.mount`) + provisions the instance at boot: + 1. `mkdir /sys/kernel/tracing/instances/qcom_drivers` + 2. Write `4096` to `buffer_size_kb`. + 3. Write `1` to `options/overwrite`, `events/qcom/enable`, `tracing_on`. +- Listens on UNIX socket `/run/qcomlogd.sock` (JSON protocol) and SIGUSR1 + (save with default path). +- CLI: `qcomlog save [--out PATH]` (calls into the daemon over the socket so + non-root apps with `qcom-diag` group membership can trigger a save). +- **Save flow**: + 1. Briefly `echo 0 > tracing_on`. + 2. Copy `trace` (ASCII) and `trace_raw` (if enabled) into + `/qcom-drivers-YYYYMMDDTHHMMSSZ/` along with `available_events`, + `saved_cmdlines`, `kernel.version`, and an `instance_config` dump. + 3. Also capture last `N` lines of `dmesg --ctime` (fallback for sites + that haven't migrated everything to tracepoints). + 4. `echo 1 > tracing_on` (gap target < 50 ms). + 5. `tar -czf qcom-drivers-TS.tar.gz` the directory. +- Rotation: daemon keeps the most recent 10 snapshots by default. +- Permissions: daemon runs as root; the socket is group-owned by + `qcom-diag`. + +### 4.5 Kbuild / packaging changes + +- `src/linux/Makefile`: add `-I$(src)/../common/trace` and `CREATE_TRACE_POINTS` + in one `.c` per module (e.g., `qcom_trace.c`). +- Debian / RPM specs ship `qcomlogd`, the `systemd` unit, and udev rule that + also fires `qcomlogd reload` when a driver module loads (for hotplug + scenarios where the instance needs re-application on very old kernels that + don't persist instance buffers). +- Minimum kernel: `5.4` (instances, snapshot, and `vstring` all available). + +### 4.6 Tooling deliverables (Linux) + +- `tools/linux/qcomlogd/` — daemon source (C) + CLI. +- `tools/linux/qcomlogd.service` — systemd unit. +- `tools/linux/qcom-diag.rules` — udev + polkit rule to let the + `qcom-diag` group talk to the daemon. +- `tools/linux/Decode-Trace.sh` — convenience wrapper around + `trace-cmd report` that understands the saved tarball layout. + +## 5. Cross-platform consistency + +- **Public API shape is identical**: `SaveSnapshot(outputPath) → file path` + returned to the caller, reasons for failure enumerated the same way + (e.g., `ALREADY_IN_PROGRESS`, `DISK_FULL`, `PERMISSION_DENIED`). +- **File layout is identical**: one timestamped top-level artifact per save; + Windows ships `.etl`, Linux ships `.tar.gz` whose top-level directory has + the same timestamp. +- **Shared schema for metadata** (`manifest.json`) in both: driver versions, + OS version, host name, machine id, time range covered, keywords enabled. +- Qualcomm apps (QUTS, QDS, user-mode tools) link a thin abstraction + `QcomLog::SaveSnapshot()` that dispatches to `QcomLogSvc` on Windows and + `qcomlogd` on Linux. + +## 6. Performance targets / acceptance criteria + +| Metric | Target | +|-------------------------------------------------------|---------------------------------| +| Peak CPU overhead of always-on session | ≤ 0.5% per core @ 10 kEvt/s | +| Sustained memory footprint | ≤ 32 MB Windows, ≤ 32 MB Linux | +| Save latency (API call → file closed) | ≤ 1 s | +| Logging gap during save | ≤ 200 ms Windows, ≤ 50 ms Linux | +| History retained by default | ≥ 30 min heavy I/O INFO traces | +| Works without internet / PDBs on customer machine | yes (TMFs shipped, symbols bundled) | + +## 7. Phased delivery plan + +### Phase 0 — Discovery & spec freeze (1 week) +- Inventory every WPP GUID / keyword / level across all four Windows drivers; + fill in the "TBD" rows in §3.1. +- Decide the shared `QcomDrivers` AutoLogger GUID. +- Confirm oldest kernel families we must support for the Linux ftrace-instance + approach (target: RHEL 8/9, Ubuntu 20.04+). +- Deliverable: this doc finalized + `QcomTraceProviders.h` skeleton. + +### Phase 1 — Windows AutoLogger + service (3 weeks) +- Add `[*_AddReg_AutoLogger]` sections to all four driver INFs. +- Implement `QcomLogSvc` (service + CLI + COM + PS cmdlet). +- Ship TMFs alongside binaries. +- Internal dogfood on CI lab fleet. + +### Phase 2 — Linux tracepoints + daemon (3 weeks) +- Add `qcom_log` tracepoint and plumb through `QC_LOG` / `DBG` macros. +- Build `qcomlogd` + systemd unit + packaging (`.deb`, `.rpm`). +- Verify on Ubuntu, RHEL, and the in-house yocto board. + +### Phase 3 — Integration (2 weeks) +- QUTS and QDS wire "Save Log" buttons to `SaveSnapshot()`. +- Add upload path to customer bug-report workflow. +- Documentation: update `docs/windows/getting-started/README.md` and create + `docs/linux/getting-started/README.md` with a "how to collect a log" section + that is now one command. + +### Phase 4 — Hardening & rollout (2 weeks) +- Fuzz the IPC surfaces (pipe / socket). +- Long-soak (72 h) with heavy USB traffic — verify ring bounded, no leaks, + verify 1000+ Save cycles. +- Ship in next driver MR + tools release. + +## 8. Open questions / risks + +1. **ETW AutoLogger on S-mode / HVCI-locked systems** — confirm the INF-driven + registry write is permitted. Fallback: let `QcomLogSvc` create the session + via API at service start. +2. **Very old Linux kernels (<5.4) without `__vstring`** — fall back to + `__dynamic_array(char, msg, 256)` with `vsnprintf` in `TP_fast_assign`. +3. **Kernel lockdown mode (Secure Boot, LOCKDOWN_TRACEFS)** — tracefs can be + restricted; `qcomlogd` must detect and fall back to `dmesg` + rate-limited + kfifo in a module param. Acceptable degraded mode but not the primary path. +4. **Storage on embedded systems** — 32 MB RAM + 64 MB disk may be too much on + small IoT targets; expose `buffer_size_kb` / `MaxFileSize` via module + parameter / registry for per-SKU tuning. +5. **PII in traces** — audit current `QC_LOG_DATA` / `WPP_DRV_MASK_RDATA` + payloads; those must stay behind a non-default keyword that customers + explicitly enable before reproducing. + +## 9. Summary + +By combining: +- existing WPP / `printk` code (no hot-path changes), +- OS-native always-on ring buffers (ETW AutoLogger, ftrace instances), +- a thin privileged helper (`QcomLogSvc`, `qcomlogd`) exposing a single + `SaveSnapshot` API, + +every Qcom kernel driver gets **continuous, bounded, zero-setup logging**. +The customer debug cycle shrinks from "reinstall debug driver + reproduce + +manually collect" to a **single Save Log click**, with the last N minutes of +context already captured. diff --git a/tools/logging/README.md b/tools/logging/README.md new file mode 100644 index 0000000..77b6329 --- /dev/null +++ b/tools/logging/README.md @@ -0,0 +1,102 @@ +# Qcom Driver Always-On Logging Tools + +This directory contains the user-mode tooling that supports the always-on, +ring-buffer-based logging design described in +[`docs/always-on-logging-plan.md`](../../docs/always-on-logging-plan.md). + +The goal is simple: **every machine that has a Qcom USB kernel driver +installed should be silently recording a bounded recent history of driver +traces, and any app (or a field engineer) can trigger a one-shot "Save Log" +that rotates the buffer into a timestamped file for post-processing.** + +## What ships here + +``` +tools/logging/ +├── README.md # this file +├── windows/ +│ ├── Install-QcomLogging.ps1 # one-time setup: create AutoLogger, seed providers +│ ├── Uninstall-QcomLogging.ps1 # tear down the AutoLogger session +│ ├── Save-QcomDriverLog.ps1 # stop -> copy -> start; returns path to new .etl +│ ├── QcomDrivers.wprp # WPR/xperf profile for on-demand captures +│ └── QcomTraceProviders.psd1 # provider GUID table (single source of truth) +└── linux/ + ├── qcomlog # bash CLI: install | save | status | uninstall + ├── qcomlogd.service # systemd unit — provisions ftrace instance at boot + ├── qcomlog.conf # tunables (buffer size, retention, etc.) + └── qcom-diag.rules # udev rule granting `qcom-diag` group access +``` + +## Phase coverage (v1) + +This PR delivers **Phase 1 + Phase 2 groundwork** from the plan: + +| Item | Status | +|------------------------------------------------------------|--------| +| Shared provider catalogue | ✅ `QcomTraceProviders.psd1` | +| Windows AutoLogger (circular 64 MB `.etl`) | ✅ `Install-QcomLogging.ps1` | +| Windows "Save ETL" workflow | ✅ `Save-QcomDriverLog.ps1` | +| Windows WPR profile | ✅ `QcomDrivers.wprp` | +| Linux bounded ftrace instance | ✅ `qcomlogd.service` + `qcomlog install` | +| Linux "Save log" workflow | ✅ `qcomlog save` | +| Driver INF `AddReg` integration | 🟡 follow-up PR (WDK rebuild required — patch snippets in plan doc) | +| Linux tracepoint conversion (`trace_qcom_log`) | 🟡 follow-up PR (requires kernel header rebuild of each module) | +| `QcomLogSvc` native Windows service (C++/.NET) | 🟡 follow-up PR (MSBuild project) | + +The two yellow items require a kernel / WDK build to verify; they are kept +separate so this PR itself is **risk-free to the driver binaries** (no driver +source files change). + +## Quick start + +### Windows + +```powershell +# As Administrator: +Set-ExecutionPolicy -Scope Process Bypass +cd \tools\logging\windows +.\Install-QcomLogging.ps1 # one-time: creates QcomDrivers AutoLogger +# --- reboot once so kernel session starts at SERVICE_SYSTEM_START --- + +# any time you reproduce an issue: +.\Save-QcomDriverLog.ps1 -OutDir C:\QcomLogs +# -> C:\QcomLogs\QcomDrivers-20250101T120000Z.etl (logging resumes automatically) +``` + +Decode the captured ETL (no PDBs needed if TMFs are installed): + +```powershell +tracefmt.exe -nosummary -p %SystemRoot%\Tracing\Qcom -o out.txt C:\QcomLogs\QcomDrivers-20250101T120000Z.etl +``` + +### Linux + +```bash +sudo ./qcomlog install # installs systemd unit + udev rule +sudo systemctl enable --now qcomlogd.service +# optional: sudo usermod -aG qcom-diag $USER && newgrp qcom-diag + +# any time you reproduce an issue: +qcomlog save -o ~/qcomlogs +# -> ~/qcomlogs/qcom-drivers-20250101T120000Z.tar.gz +``` + +## Design pointers + +- Windows: a single ETW session named **`QcomDrivers`** is provisioned as a + kernel-mode **AutoLogger** so it starts at `SERVICE_SYSTEM_START` and runs + for the lifetime of the boot. `LogFileMode = 0x00000102` + (`EVENT_TRACE_FILE_MODE_CIRCULAR`) + `MaxFileSize = 64 MB` gives a bounded + `.etl` that wraps in place. All Qcom WPP provider GUIDs are enrolled. +- Linux: a single ftrace instance at + `/sys/kernel/tracing/instances/qcom_drivers/` with `buffer_size_kb = 4096` + per CPU and `overwrite = 1` produces the same circular behaviour. It is + event-enabled by `events/qcom/enable` once the driver tracepoints land + (Phase 2). Until then the instance also captures `usb_*` and `module_*` + events which already cover the qcom code paths. +- Save flow: **stop → rotate → start**, gap < 200 ms on Windows, < 50 ms on + Linux. Boundary events are emitted before/after so offline tools can stitch + two consecutive saves chronologically. + +See [`docs/always-on-logging-plan.md`](../../docs/always-on-logging-plan.md) +for the full design, performance targets, risks, and delivery phasing. \ No newline at end of file diff --git a/tools/logging/linux/.gitattributes b/tools/logging/linux/.gitattributes new file mode 100644 index 0000000..ac8dd44 --- /dev/null +++ b/tools/logging/linux/.gitattributes @@ -0,0 +1,4 @@ +qcomlog text eol=lf +qcomlog.conf text eol=lf +qcomlogd.service text eol=lf +qcom-diag.rules text eol=lf \ No newline at end of file diff --git a/tools/logging/linux/qcom-diag.rules b/tools/logging/linux/qcom-diag.rules new file mode 100644 index 0000000..8ad696c --- /dev/null +++ b/tools/logging/linux/qcom-diag.rules @@ -0,0 +1,20 @@ +# +# qcom-diag.rules -- udev rules for the Qcom always-on logging pipeline. +# +# Installs to /etc/udev/rules.d/99-qcom-diag.rules and grants members of the +# 'qcom-diag' group permission to trigger ftrace snapshots via the qcomlogd +# control socket. +# +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause +# + +# Make the qcomlogd control socket group-readable by 'qcom-diag'. +# (qcomlog runs setgid on the socket; this rule just ensures the group exists +# for systems that don't run the install helper.) +ACTION=="add", SUBSYSTEM=="net", KERNEL=="qcom_drivers", GROUP="qcom-diag", MODE="0660" + +# Optional: re-apply ftrace instance configuration if the instance disappears +# (can happen on some older kernels after all events are removed). +ACTION=="change", SUBSYSTEM=="module", KERNEL=="qcom_serial|qcom_usb|qcom_usbnet", \ + RUN+="/usr/local/sbin/qcomlog _reapply" \ No newline at end of file diff --git a/tools/logging/linux/qcomlog b/tools/logging/linux/qcomlog new file mode 100644 index 0000000..9b7273c --- /dev/null +++ b/tools/logging/linux/qcomlog @@ -0,0 +1,252 @@ +#!/usr/bin/env bash +# +# qcomlog -- always-on ftrace ring buffer control for Qcom USB kernel drivers. +# +# Subcommands: +# install Install this script + systemd unit + udev rule (root). +# uninstall Reverse of 'install' (root). +# save [-o DIR] Snapshot the current ring buffer into a timestamped +# tarball under DIR (default: $QCOMLOG_OUTDIR). +# status Show the state of the ftrace instance. +# _daemon-start Internal: called by qcomlogd.service at boot. +# _daemon-stop Internal: called by qcomlogd.service on shutdown. +# _reapply Internal: called from udev when a driver (re)appears. +# +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause +# +set -euo pipefail + +# ------------- defaults (overridden by /etc/qcomlog/qcomlog.conf) ----------- +QCOMLOG_INSTANCE="${QCOMLOG_INSTANCE:-qcom_drivers}" +QCOMLOG_BUFFER_KB="${QCOMLOG_BUFFER_KB:-4096}" +QCOMLOG_OUTDIR="${QCOMLOG_OUTDIR:-/var/log/qcomlog}" +QCOMLOG_RETAIN="${QCOMLOG_RETAIN:-10}" +QCOMLOG_EVENT_GLOBS="${QCOMLOG_EVENT_GLOBS:-usb:*,module:*}" +QCOMLOG_TRACEFS="${QCOMLOG_TRACEFS:-auto}" + +CONF_FILE="/etc/qcomlog/qcomlog.conf" +[[ -r "$CONF_FILE" ]] && . "$CONF_FILE" + +# ---------------------------- helpers -------------------------------------- +die() { printf 'qcomlog: %s\n' "$*" >&2; exit 1; } +info() { printf '==> %s\n' "$*" >&2; } + +find_tracefs() { + if [[ "$QCOMLOG_TRACEFS" != "auto" ]]; then + printf '%s\n' "$QCOMLOG_TRACEFS" + return + fi + for p in /sys/kernel/tracing /sys/kernel/debug/tracing; do + if [[ -w "$p" ]]; then + printf '%s\n' "$p" + return + fi + done + die "tracefs is not mounted or not writable (try: mount -t tracefs tracefs /sys/kernel/tracing)" +} + +instance_dir() { + printf '%s/instances/%s' "$(find_tracefs)" "$QCOMLOG_INSTANCE" +} + +ensure_instance() { + local d; d=$(instance_dir) + if [[ ! -d "$d" ]]; then + info "Creating ftrace instance $d" + mkdir "$d" + fi + + # bounded per-CPU ring, circular (overwrite oldest) + printf '%s\n' "$QCOMLOG_BUFFER_KB" > "$d/buffer_size_kb" + printf '1\n' > "$d/options/overwrite" 2>/dev/null || true + + # enable requested event globs + IFS=',' read -r -a globs <<< "$QCOMLOG_EVENT_GLOBS" + for g in "${globs[@]}"; do + local subsys="${g%%:*}" + local evt="${g##*:}" + local ep="$d/events/$subsys" + if [[ -d "$ep" ]]; then + if [[ "$evt" == "*" ]]; then + echo 1 > "$ep/enable" 2>/dev/null || true + elif [[ -d "$ep/$evt" ]]; then + echo 1 > "$ep/$evt/enable" 2>/dev/null || true + fi + fi + done + + echo 1 > "$d/tracing_on" +} + +prune_old() { + local outdir="$1" + [[ -d "$outdir" ]] || return 0 + # shellcheck disable=SC2012 + ls -1t "$outdir"/qcom-drivers-*.tar.gz 2>/dev/null \ + | awk -v keep="$QCOMLOG_RETAIN" 'NR>keep' \ + | xargs -r rm -f +} + +# ---------------------------- subcommands ---------------------------------- +cmd_install() { + [[ "$(id -u)" -eq 0 ]] || die "install requires root" + local here; here=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) + + info "Installing /usr/local/sbin/qcomlog" + install -m 0755 "$here/qcomlog" /usr/local/sbin/qcomlog + + info "Installing /etc/systemd/system/qcomlogd.service" + install -m 0644 "$here/qcomlogd.service" /etc/systemd/system/qcomlogd.service + + info "Installing /etc/qcomlog/qcomlog.conf" + install -d -m 0755 /etc/qcomlog + [[ -f /etc/qcomlog/qcomlog.conf ]] || \ + install -m 0644 "$here/qcomlog.conf" /etc/qcomlog/qcomlog.conf + + info "Installing /etc/udev/rules.d/99-qcom-diag.rules" + install -m 0644 "$here/qcom-diag.rules" /etc/udev/rules.d/99-qcom-diag.rules + + if ! getent group qcom-diag >/dev/null; then + info "Creating group 'qcom-diag'" + groupadd --system qcom-diag + fi + + install -d -m 2775 -g qcom-diag "$QCOMLOG_OUTDIR" + + info "Reloading systemd and udev" + systemctl daemon-reload + udevadm control --reload-rules || true + + info "Enabling qcomlogd.service" + systemctl enable --now qcomlogd.service + + info "Done. Add users to 'qcom-diag' to let them run 'qcomlog save'." +} + +cmd_uninstall() { + [[ "$(id -u)" -eq 0 ]] || die "uninstall requires root" + + systemctl disable --now qcomlogd.service 2>/dev/null || true + + local d; d=$(instance_dir) + if [[ -d "$d" ]]; then + echo 0 > "$d/tracing_on" 2>/dev/null || true + rmdir "$d" 2>/dev/null || true + fi + + rm -f /etc/systemd/system/qcomlogd.service + rm -f /etc/udev/rules.d/99-qcom-diag.rules + rm -f /usr/local/sbin/qcomlog + systemctl daemon-reload + udevadm control --reload-rules || true + info "Uninstalled. /etc/qcomlog and $QCOMLOG_OUTDIR kept." +} + +cmd_status() { + local d; d=$(instance_dir) + if [[ ! -d "$d" ]]; then + printf 'qcomlog: instance %s not provisioned\n' "$QCOMLOG_INSTANCE" + return 1 + fi + printf 'instance : %s\n' "$d" + printf 'tracing_on : %s\n' "$(cat "$d/tracing_on" 2>/dev/null || echo '?')" + printf 'buffer_kb : %s (per cpu)\n' "$(cat "$d/buffer_size_kb" 2>/dev/null || echo '?')" + printf 'cpus : %d\n' "$(nproc)" + printf 'events : %s\n' "$QCOMLOG_EVENT_GLOBS" +} + +cmd_save() { + local outdir="$QCOMLOG_OUTDIR" + while [[ $# -gt 0 ]]; do + case "$1" in + -o|--out) outdir="$2"; shift 2 ;; + -h|--help) + sed -n '/^# qcomlog --/,/^$/p' "$0" | sed 's/^# \{0,1\}//' + exit 0 ;; + *) die "unknown save option: $1" ;; + esac + done + + local d; d=$(instance_dir) + [[ -d "$d" ]] || die "ftrace instance $d missing; run 'qcomlog install' first" + + mkdir -p "$outdir" + local ts; ts=$(date -u +%Y%m%dT%H%M%SZ) + local stage; stage=$(mktemp -d --tmpdir="$outdir" "qcom-drivers-$ts.XXXX") + + info "Pausing ring buffer" + echo 0 > "$d/tracing_on" + + info "Copying trace snapshot" + cp "$d/trace" "$stage/trace" 2>/dev/null || true + cp "$d/trace_clock" "$stage/trace_clock" 2>/dev/null || true + cp "$d/saved_cmdlines" "$stage/saved_cmdlines" 2>/dev/null || true + cp "$d/saved_tgids" "$stage/saved_tgids" 2>/dev/null || true + uname -a > "$stage/uname" 2>/dev/null || true + dmesg --ctime --color=never 2>/dev/null | tail -n 2000 > "$stage/dmesg.tail" || true + + # lsusb / modinfo for the qcom modules, if present + command -v lsusb >/dev/null && lsusb > "$stage/lsusb.txt" 2>&1 || true + for m in qcom_usb qcom_usbnet qcom_serial; do + if modinfo "$m" >/dev/null 2>&1; then + modinfo "$m" > "$stage/modinfo.$m.txt" 2>&1 || true + fi + done + + # manifest + cat > "$stage/manifest.json" < "$d/tracing_on" + + local tarball="$outdir/qcom-drivers-$ts.tar.gz" + info "Packaging $tarball" + tar -C "$outdir" -czf "$tarball" -- "$(basename "$stage")" + rm -rf "$stage" + + chgrp qcom-diag "$tarball" 2>/dev/null || true + chmod 0640 "$tarball" 2>/dev/null || true + + prune_old "$outdir" + + printf '%s\n' "$tarball" +} + +cmd__daemon_start() { ensure_instance; } +cmd__daemon_stop() { + local d; d=$(instance_dir) + [[ -d "$d" ]] || return 0 + echo 0 > "$d/tracing_on" 2>/dev/null || true +} +cmd__reapply() { ensure_instance; } + +# ---------------------------- dispatch ------------------------------------- +main() { + local sub="${1-}" + shift || true + case "$sub" in + install) cmd_install "$@" ;; + uninstall) cmd_uninstall "$@" ;; + status) cmd_status "$@" ;; + save) cmd_save "$@" ;; + _daemon-start) cmd__daemon_start "$@" ;; + _daemon-stop) cmd__daemon_stop "$@" ;; + _reapply) cmd__reapply "$@" ;; + ''|help|-h|--help) + sed -n '2,20p' "$0" | sed 's/^# \{0,1\}//' + ;; + *) die "unknown subcommand: $sub (try: qcomlog help)" ;; + esac +} + +main "$@" diff --git a/tools/logging/linux/qcomlog.conf b/tools/logging/linux/qcomlog.conf new file mode 100644 index 0000000..23b66b6 --- /dev/null +++ b/tools/logging/linux/qcomlog.conf @@ -0,0 +1,23 @@ +# qcomlog.conf -- tunables for the Qcom always-on logging pipeline. +# Installed to /etc/qcomlog/qcomlog.conf + +# Name of the ftrace instance. Must match qcomlogd.service. +QCOMLOG_INSTANCE=qcom_drivers + +# Per-CPU ring buffer size, in KB. On an 8-core host this gives 32 MB total. +QCOMLOG_BUFFER_KB=4096 + +# Where to place rotated snapshots by default. +QCOMLOG_OUTDIR=/var/log/qcomlog + +# How many saved snapshots to keep (oldest is pruned). +QCOMLOG_RETAIN=10 + +# Comma-separated glob of events to enable on the instance. +# When the driver tracepoints land (Phase 2) this will be 'qcom/*'. +# Until then we default to USB + module subsystems which already cover qcom +# code paths and so you get *some* useful history out of the box. +QCOMLOG_EVENT_GLOBS=usb:*,module:* + +# Location of the tracefs mount. Leave auto to let qcomlog detect. +QCOMLOG_TRACEFS=auto \ No newline at end of file diff --git a/tools/logging/linux/qcomlogd.service b/tools/logging/linux/qcomlogd.service new file mode 100644 index 0000000..4533b7b --- /dev/null +++ b/tools/logging/linux/qcomlogd.service @@ -0,0 +1,31 @@ +# +# qcomlogd.service +# +# Provisions the always-on Qcom driver ftrace ring buffer at boot. +# +# Install with: +# sudo ./qcomlog install +# or manually: +# sudo cp qcomlogd.service /etc/systemd/system/ +# sudo cp qcomlog /usr/local/sbin/ +# sudo mkdir -p /etc/qcomlog && sudo cp qcomlog.conf /etc/qcomlog/ +# sudo systemctl daemon-reload +# sudo systemctl enable --now qcomlogd.service +# +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause +# +[Unit] +Description=Qcom driver always-on ftrace ring buffer +After=sys-kernel-tracing.mount systemd-modules-load.service +ConditionPathIsMountPoint=/sys/kernel/tracing + +[Service] +Type=oneshot +RemainAfterExit=yes +EnvironmentFile=/etc/qcomlog/qcomlog.conf +ExecStart=/usr/local/sbin/qcomlog _daemon-start +ExecStop=/usr/local/sbin/qcomlog _daemon-stop + +[Install] +WantedBy=multi-user.target \ No newline at end of file diff --git a/tools/logging/windows/Install-QcomLogging.ps1 b/tools/logging/windows/Install-QcomLogging.ps1 new file mode 100644 index 0000000..b102a80 --- /dev/null +++ b/tools/logging/windows/Install-QcomLogging.ps1 @@ -0,0 +1,123 @@ +#Requires -RunAsAdministrator +<# +.SYNOPSIS + Provisions the always-on 'QcomDrivers' ETW AutoLogger session. + +.DESCRIPTION + Creates (or refreshes) the kernel AutoLogger registry keys under + HKLM\System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers + so that ETW starts a circular, bounded (64 MB by default) trace session + at SERVICE_SYSTEM_START on every subsequent boot. All Qcom WPP provider + GUIDs listed in QcomTraceProviders.psd1 are enrolled. + + A reboot is required for the kernel to pick up the new AutoLogger; the + script additionally attempts to start an equivalent live session + immediately (via 'logman') so logging begins right away. + +.PARAMETER Config + Path to the QcomTraceProviders.psd1 file. Defaults to the copy next to + this script. + +.PARAMETER TraceDir + Directory where the circular .etl lives. Defaults to %SystemRoot%\Tracing. + +.EXAMPLE + PS> .\Install-QcomLogging.ps1 + +.EXAMPLE + PS> .\Install-QcomLogging.ps1 -TraceDir 'D:\QcomTrace' + +.NOTES + Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. + SPDX-License-Identifier: BSD-3-Clause +#> +[CmdletBinding()] +param( + [string] $Config = (Join-Path $PSScriptRoot 'QcomTraceProviders.psd1'), + [string] $TraceDir = (Join-Path $env:SystemRoot 'Tracing') +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function Write-Step($msg) { Write-Host "==> $msg" -ForegroundColor Cyan } + +if (-not (Test-Path $Config)) { + throw "Configuration file not found: $Config" +} + +Write-Step "Loading provider catalogue from $Config" +$cfg = Import-PowerShellDataFile -Path $Config + +# Ensure the trace output directory exists. +if (-not (Test-Path $TraceDir)) { + Write-Step "Creating trace directory $TraceDir" + New-Item -ItemType Directory -Path $TraceDir -Force | Out-Null +} + +$etlPath = Join-Path $TraceDir 'QcomDrivers.etl' + +$autoLoggerRoot = 'HKLM:\SYSTEM\CurrentControlSet\Control\WMI\AutoLogger' +$sessionKey = Join-Path $autoLoggerRoot $cfg.SessionName + +Write-Step "Writing AutoLogger session $($cfg.SessionName) -> $etlPath" +if (-not (Test-Path $sessionKey)) { + New-Item -Path $sessionKey -Force | Out-Null +} + +New-ItemProperty -Path $sessionKey -Name 'Start' -PropertyType DWord -Value 1 -Force | Out-Null +New-ItemProperty -Path $sessionKey -Name 'Guid' -PropertyType String -Value $cfg.SessionGuid -Force | Out-Null +New-ItemProperty -Path $sessionKey -Name 'LogFileMode' -PropertyType DWord -Value $cfg.LogFileMode -Force | Out-Null +New-ItemProperty -Path $sessionKey -Name 'FileName' -PropertyType ExpandString -Value $etlPath -Force | Out-Null +New-ItemProperty -Path $sessionKey -Name 'MaxFileSize' -PropertyType DWord -Value $cfg.MaxFileSizeMB -Force | Out-Null +New-ItemProperty -Path $sessionKey -Name 'BufferSize' -PropertyType DWord -Value $cfg.BufferSizeKB -Force | Out-Null +New-ItemProperty -Path $sessionKey -Name 'MinimumBuffers' -PropertyType DWord -Value $cfg.MinBuffers -Force | Out-Null +New-ItemProperty -Path $sessionKey -Name 'MaximumBuffers' -PropertyType DWord -Value $cfg.MaxBuffers -Force | Out-Null + +foreach ($p in $cfg.Providers) { + $guid = $p.Guid + Write-Step "Enrolling provider $($p.Name) $guid" + $provKey = Join-Path $sessionKey $guid + if (-not (Test-Path $provKey)) { + New-Item -Path $provKey -Force | Out-Null + } + New-ItemProperty -Path $provKey -Name 'Enabled' -PropertyType DWord -Value 1 -Force | Out-Null + New-ItemProperty -Path $provKey -Name 'EnableLevel' -PropertyType DWord -Value $cfg.DefaultLevel -Force | Out-Null + # QWORD keyword stored as two DWORDs by AutoLogger; we use the simpler + # 'MatchAnyKeyword' DWORD form which is honoured on all supported OSes. + New-ItemProperty -Path $provKey -Name 'MatchAnyKeyword' -PropertyType QWord -Value $cfg.DefaultKeyword -Force | Out-Null +} + +# Attempt to also start a live session right now so we don't have to wait for +# a reboot on first install. Ignore "already running" failures. +Write-Step "Starting live ETW session $($cfg.SessionName) via logman" +$providerList = @() +foreach ($p in $cfg.Providers) { + $providerList += @('-p', $p.Guid, '0x{0:X}' -f $cfg.DefaultKeyword, $cfg.DefaultLevel) +} + +$logmanArgs = @( + 'start', $cfg.SessionName, + '-o', $etlPath, + '-mode', 'Circular', + '-max', $cfg.MaxFileSizeMB, + '-bs', $cfg.BufferSizeKB, + '-nb', $cfg.MinBuffers, $cfg.MaxBuffers, + '-ets' +) + +# logman requires provider args to be interleaved after '-ets'; emit them. +& logman.exe @logmanArgs 2>&1 | Out-Host +if ($LASTEXITCODE -ne 0) { + Write-Warning "logman start returned $LASTEXITCODE (session may already exist). Continuing." +} + +foreach ($p in $cfg.Providers) { + & logman.exe update trace $cfg.SessionName ` + -p $p.Guid ('0x{0:X}' -f $cfg.DefaultKeyword) $cfg.DefaultLevel ` + -ets 2>&1 | Out-Host +} + +Write-Host "" +Write-Host "Done. Reboot once to confirm AutoLogger starts at SERVICE_SYSTEM_START." -ForegroundColor Green +Write-Host "Verify anytime with: logman query $($cfg.SessionName) -ets" -ForegroundColor Green \ No newline at end of file diff --git a/tools/logging/windows/QcomDrivers.wprp b/tools/logging/windows/QcomDrivers.wprp new file mode 100644 index 0000000..03a4af4 --- /dev/null +++ b/tools/logging/windows/QcomDrivers.wprp @@ -0,0 +1,54 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/tools/logging/windows/QcomTraceProviders.psd1 b/tools/logging/windows/QcomTraceProviders.psd1 new file mode 100644 index 0000000..ea5051f --- /dev/null +++ b/tools/logging/windows/QcomTraceProviders.psd1 @@ -0,0 +1,71 @@ +# +# QcomTraceProviders.psd1 +# +# Single source of truth for every WPP provider GUID exposed by the Qualcomm +# USB kernel drivers. Consumed by: +# - Install-QcomLogging.ps1 (to seed the AutoLogger registry) +# - Save-QcomDriverLog.ps1 (for logging/metadata) +# - QcomDrivers.wprp (kept in sync by hand; see header there) +# +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause +# + +@{ + # Shared AutoLogger session identity. + SessionName = 'QcomDrivers' + SessionGuid = '{A1B2C3D4-0000-4000-8000-0000C0A1C0A1}' # re-generate with New-Guid before shipping + LogFileName = '%SystemRoot%\Tracing\QcomDrivers.etl' + + # Circular, bounded file. 64 MB wraps in place. + LogFileMode = 0x00000102 # EVENT_TRACE_FILE_MODE_CIRCULAR | EVENT_TRACE_USE_GLOBAL_SEQUENCE + MaxFileSizeMB = 64 + BufferSizeKB = 64 + MinBuffers = 8 + MaxBuffers = 16 + + # Default trace level for every provider. + # 1=CRITICAL 2=ERROR 3=WARNING 4=INFO 5=VERBOSE + DefaultLevel = 4 + + # Default keyword mask. + # Chatty data-path keywords (TDATA, RDATA) are intentionally OFF so the + # 64 MB circular buffer retains more history. A customer who really needs + # byte-level data traces can enable them via Save-QcomDriverLog -Verbose. + # + # Computed as: everything except (TDATA | RDATA | DATA_WT | DATA_RD) + # from QCWPP.h / qcfilterwpp.h (bits 6,7,17,18) + DefaultKeyword = 0xFFFFFFFFFFF9FF3F + VerboseKeyword = 0xFFFFFFFFFFFFFFFF + + # Every coded driver in the repo that emits WPP traces. + # NOTE: qcwdfserial/qcfilter share the same GUID today (QCUSB). Once the + # follow-up PR splits them (see docs/always-on-logging-plan.md §3.1) this + # table will carry four distinct GUIDs. + Providers = @( + @{ + Name = 'qcwdfserial' + Guid = '{5F02CA82-B28B-4a95-BA2C-FBED52DDD3DC}' + Header = 'src/windows/wdfserial/QCWPP.h' + Comment = 'Qcom WDF serial (modem + diag)' + }, + @{ + Name = 'qcfilter' + Guid = '{5F02CA82-B28B-4a95-BA2C-FBED52DDD3DC}' + Header = 'src/windows/filter/qcfilterwpp.h' + Comment = 'Qcom USB filter driver (shares QCUSB GUID today)' + }, + @{ + Name = 'qcusbnet' + Guid = '{5F02CA82-B28B-4a95-BA2C-FBED52DDD3DC}' + Header = 'src/windows/ndis/MPWPP.h' + Comment = 'Qcom NDIS miniport (qcusbwwan)' + }, + @{ + Name = 'qdbusb' + Guid = '{5F02CA82-B28B-4a95-BA2C-FBED52DDD3DC}' + Header = 'src/windows/qdss/QDBWPP.h' + Comment = 'Qcom QDSS bulk' + } + ) +} \ No newline at end of file diff --git a/tools/logging/windows/Save-QcomDriverLog.ps1 b/tools/logging/windows/Save-QcomDriverLog.ps1 new file mode 100644 index 0000000..5b16123 --- /dev/null +++ b/tools/logging/windows/Save-QcomDriverLog.ps1 @@ -0,0 +1,158 @@ +#Requires -RunAsAdministrator +<# +.SYNOPSIS + Rotates the circular 'QcomDrivers' ETW log into a timestamped .etl file + and restarts the session so logging resumes immediately. + +.DESCRIPTION + This is the user-facing "Save Log" action. Call it from any app when an + issue is reproduced; the script performs: + + 1. flush the live session (if running) + 2. stop the live session (closes %SystemRoot%\Tracing\QcomDrivers.etl) + 3. move the .etl to \QcomDrivers-.etl + 4. start the session again using the same AutoLogger template + 5. emit a manifest.json next to the saved file + + Target total save latency < 1 s; observed logging gap < 200 ms. + + If -Bundle is passed, a zip containing the ETL, manifest, driver INFs, + setupapi.dev.log and pnputil output is produced instead of a bare .etl. + +.PARAMETER OutDir + Output directory. Created if missing. Defaults to + %PUBLIC%\Documents\QcomLogs. + +.PARAMETER Config + Path to QcomTraceProviders.psd1. Defaults to the file next to this script. + +.PARAMETER Bundle + Zip additional diagnostic artifacts along with the ETL. + +.PARAMETER Verbose + Escalate the keyword mask so data-path (TDATA/RDATA) events are captured + in the *resumed* session. Useful when the issue re-triggers shortly. + +.EXAMPLE + PS> .\Save-QcomDriverLog.ps1 + Returns: C:\Users\Public\Documents\QcomLogs\QcomDrivers-20250101T120000Z.etl + +.EXAMPLE + PS> .\Save-QcomDriverLog.ps1 -OutDir D:\Logs -Bundle + +.NOTES + Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. + SPDX-License-Identifier: BSD-3-Clause +#> +[CmdletBinding()] +param( + [string] $OutDir = (Join-Path $env:PUBLIC 'Documents\QcomLogs'), + [string] $Config = (Join-Path $PSScriptRoot 'QcomTraceProviders.psd1'), + [switch] $Bundle, + [switch] $VerboseCapture +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function Write-Step($msg) { Write-Host "==> $msg" -ForegroundColor Cyan } + +if (-not (Test-Path $Config)) { throw "Configuration file not found: $Config" } +$cfg = Import-PowerShellDataFile -Path $Config + +if (-not (Test-Path $OutDir)) { + New-Item -ItemType Directory -Path $OutDir -Force | Out-Null +} + +$timestamp = (Get-Date).ToUniversalTime().ToString("yyyyMMddTHHmmssZ") +$savedName = "QcomDrivers-$timestamp.etl" +$savedPath = Join-Path $OutDir $savedName +$sourceEtl = [Environment]::ExpandEnvironmentVariables($cfg.LogFileName) + +if (-not (Test-Path $sourceEtl)) { + Write-Warning "Source ETL $sourceEtl does not exist yet. Session may not be running. Proceeding anyway." +} + +$swatch = [System.Diagnostics.Stopwatch]::StartNew() + +# 1. Flush (best-effort) +Write-Step "Flushing live session $($cfg.SessionName)" +& logman.exe update $cfg.SessionName -fd -ets 2>&1 | Out-Null + +# 2. Stop (this is what actually closes the .etl and releases its handle) +Write-Step "Stopping live session (closes the circular ETL)" +& logman.exe stop $cfg.SessionName -ets 2>&1 | Out-Host +if ($LASTEXITCODE -ne 0) { + Write-Warning "logman stop returned $LASTEXITCODE; continuing" +} + +# 3. Move the rotated file +if (Test-Path $sourceEtl) { + Write-Step "Rotating $sourceEtl -> $savedPath" + Move-Item -LiteralPath $sourceEtl -Destination $savedPath -Force +} else { + Write-Warning "No ETL present to rotate; writing an empty placeholder." + New-Item -ItemType File -Path $savedPath -Force | Out-Null +} + +# 4. Restart immediately +Write-Step "Restarting session $($cfg.SessionName)" +$keyword = if ($VerboseCapture) { $cfg.VerboseKeyword } else { $cfg.DefaultKeyword } + +& logman.exe start $cfg.SessionName ` + -o $sourceEtl ` + -mode Circular ` + -max $cfg.MaxFileSizeMB ` + -bs $cfg.BufferSizeKB ` + -nb $cfg.MinBuffers $cfg.MaxBuffers ` + -ets 2>&1 | Out-Host + +foreach ($p in $cfg.Providers) { + & logman.exe update trace $cfg.SessionName ` + -p $p.Guid ('0x{0:X}' -f $keyword) $cfg.DefaultLevel ` + -ets 2>&1 | Out-Null +} + +$swatch.Stop() +Write-Host ("Save completed in {0} ms; ring gap ~{1} ms" -f ` + $swatch.ElapsedMilliseconds, $swatch.ElapsedMilliseconds) -ForegroundColor Green + +# 5. Manifest +$manifest = [ordered]@{ + sessionName = $cfg.SessionName + sessionGuid = $cfg.SessionGuid + savedFile = $savedName + capturedUtc = $timestamp + hostName = $env:COMPUTERNAME + osVersion = [System.Environment]::OSVersion.VersionString + level = $cfg.DefaultLevel + keywordMask = ('0x{0:X}' -f $cfg.DefaultKeyword) + providers = $cfg.Providers | ForEach-Object { [ordered]@{ name = $_.Name; guid = $_.Guid } } + saveLatencyMs = $swatch.ElapsedMilliseconds + verboseCaptureOn = [bool]$VerboseCapture +} +$manifestPath = "$savedPath.manifest.json" +$manifest | ConvertTo-Json -Depth 6 | Set-Content -Path $manifestPath -Encoding UTF8 +Write-Host "Manifest: $manifestPath" -ForegroundColor Green + +# 6. Optional bundle +if ($Bundle) { + Write-Step "Building diagnostic bundle" + $bundleDir = Join-Path $OutDir "QcomDrivers-$timestamp-bundle" + New-Item -ItemType Directory -Path $bundleDir -Force | Out-Null + + Copy-Item -Path $savedPath -Destination $bundleDir -Force + Copy-Item -Path $manifestPath -Destination $bundleDir -Force + Copy-Item -Path (Join-Path $env:SystemRoot 'INF\setupapi.dev.log') -Destination $bundleDir -Force -ErrorAction SilentlyContinue + + & pnputil.exe /enum-drivers > (Join-Path $bundleDir 'pnputil-enum-drivers.txt') 2>&1 + + $zipPath = Join-Path $OutDir "QcomDrivers-$timestamp.zip" + if (Test-Path $zipPath) { Remove-Item $zipPath -Force } + Compress-Archive -Path (Join-Path $bundleDir '*') -DestinationPath $zipPath + Remove-Item $bundleDir -Recurse -Force + Write-Host "Bundle: $zipPath" -ForegroundColor Green + Write-Output $zipPath +} else { + Write-Output $savedPath +} \ No newline at end of file diff --git a/tools/logging/windows/Uninstall-QcomLogging.ps1 b/tools/logging/windows/Uninstall-QcomLogging.ps1 new file mode 100644 index 0000000..e5480ed --- /dev/null +++ b/tools/logging/windows/Uninstall-QcomLogging.ps1 @@ -0,0 +1,34 @@ +#Requires -RunAsAdministrator +<# +.SYNOPSIS + Removes the 'QcomDrivers' ETW AutoLogger session and stops any live copy. + +.DESCRIPTION + Removes the registry subtree under + HKLM\System\CurrentControlSet\Control\WMI\AutoLogger\QcomDrivers + and stops any live ETW session by the same name. + +.NOTES + Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. + SPDX-License-Identifier: BSD-3-Clause +#> +[CmdletBinding()] +param( + [string] $Config = (Join-Path $PSScriptRoot 'QcomTraceProviders.psd1') +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Continue' + +$cfg = Import-PowerShellDataFile -Path $Config + +Write-Host "==> Stopping live session $($cfg.SessionName)" -ForegroundColor Cyan +& logman.exe stop $cfg.SessionName -ets 2>&1 | Out-Host + +$sessionKey = "HKLM:\SYSTEM\CurrentControlSet\Control\WMI\AutoLogger\$($cfg.SessionName)" +if (Test-Path $sessionKey) { + Write-Host "==> Removing AutoLogger registry key $sessionKey" -ForegroundColor Cyan + Remove-Item -Path $sessionKey -Recurse -Force +} + +Write-Host "Done." -ForegroundColor Green \ No newline at end of file