Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions docs/5-integrations/outputs/stream-structures.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,13 +344,72 @@
| `sensor_uninstalled` | Sensor removal | `action`, `reason`, `uninstall_time` |
| `sensor_upgraded` | Sensor version update | `old_version`, `new_version`, `upgrade_method` |
| `sensor_checkin` | Periodic sensor heartbeat | `last_seen`, `connectivity_status` |
| `output_drop` | One or more records were dropped by an output that could not keep up or whose retries were exhausted | `output`, `module`, `stream`, `count`, `bytes` |

### `output_drop` Events

When an output cannot keep up with incoming data (saturated in-flight queue) or its
per-record send fails after the platform's bounded retry budget, the affected
records are dropped to protect the rest of the pipeline. Rather than emit one
deployment event per dropped record — which would flood the stream under
sustained pressure — drops are aggregated and a single summary event is
published per output approximately once per minute.

```json
{
"routing": {
"oid": "8cbe27f4-aaaa-aaaa-aaaa-138cd51389cd",
"sid": "00000000-0000-0000-0000-000000000000",
"event_type": "output_drop",
"event_time": 1709308800123,
"event_id": "f3e7a1c2-aaaa-aaaa-aaaa-9d2b8f4c5e10",
"tags": []
},
"event": {
"output": "splunk-events",
"module": "webhook_bulk",
"stream": "event",
"count": 1742,
"bytes": 9483120
}
}
```

**Event Body Fields**

Check failure on line 378 in docs/5-integrations/outputs/stream-structures.md

View workflow job for this annotation

GitHub Actions / check-markdown

Emphasis used instead of a heading

docs/5-integrations/outputs/stream-structures.md:378 MD036/no-emphasis-as-heading Emphasis used instead of a heading [Context: "Event Body Fields"] https://github.com/DavidAnson/markdownlint/blob/v0.40.0/doc/md036.md

| Field | Type | Description |
|-------|------|-------------|
| `output` | string | Name of the output that dropped data |
| `module` | string | Output module type (`webhook`, `s3`, `kafka`, etc.) |
| `stream` | string | Stream the dropped records belonged to (`event`, `detect`, `audit`, `artifact`, `billing`) |
| `count` | integer | Number of records dropped in the aggregation window |
| `bytes` | integer | Total serialized bytes dropped in the window. May be 0 for saturation drops where the record is dropped before serialization. |

**Behavior Notes**

Check failure on line 388 in docs/5-integrations/outputs/stream-structures.md

View workflow job for this annotation

GitHub Actions / check-markdown

Emphasis used instead of a heading

docs/5-integrations/outputs/stream-structures.md:388 MD036/no-emphasis-as-heading Emphasis used instead of a heading [Context: "Behavior Notes"] https://github.com/DavidAnson/markdownlint/blob/v0.40.0/doc/md036.md

- **Aggregated, not per-record.** Drops are coalesced across an approximate
one-minute window per output, so a high-throughput output that's failing
on every record still produces at most ~1 event per minute on the
deployment stream — with `count` reflecting the full window's losses.
- **Drops on the deployment stream itself are not reported.** This avoids a
feedback loop where a broken deployment-stream output would emit drop
events that immediately get dropped again.
- **Live tail outputs are not reported.** Transient outputs whose names start
with `tmp_live_` are excluded.

`output_drop` events arrive on the deployment stream alongside the existing
sensor lifecycle events and can be filtered, alerted on, or routed via
[D&R rules](../../3-detection-response/index.md) like any other deployment
event — for example, paging an on-call when a specific high-value output
starts losing data.

### Use Cases

- **Asset Tracking**: Monitor endpoint agent deployment status
- **Compliance**: Ensure all required endpoints have sensors
- **Lifecycle Management**: Track sensor versions and upgrades
- **Alerting**: Detect unexpected sensor removals
- **Output Health**: Catch outputs that are dropping data (`output_drop`) before users notice missing telemetry downstream

---

Expand Down
Loading