diff --git a/skills/SKILL.md b/skills/SKILL.md new file mode 100644 index 0000000000..96de19a2ba --- /dev/null +++ b/skills/SKILL.md @@ -0,0 +1,114 @@ +--- +name: supervision +description: use when working with roboflow/supervision +--- + +# supervision + +`supervision` is a model-agnostic computer vision library (roboflow/supervision) for working with detection/segmentation results: building `sv.Detections`, drawing with annotators, tracking objects across frames, and processing video/streams. + +This skill covers the patterns an agent gets wrong most often. For anything not covered here, read the source under `src/supervision/` rather than guessing at an API — many method/parameter names look plausible but don't exist (see "common mistakes" in each reference file). + +## Most common pattern: detect + annotate + +The standard loop is: run a model, wrap its output in `sv.Detections`, draw boxes and labels, output/save the frame. + +```python +import cv2 +import supervision as sv +from ultralytics import YOLO + +model = YOLO("yolov8n.pt") +box_annotator = sv.BoxAnnotator() +label_annotator = sv.LabelAnnotator() + +image = cv2.imread("image.jpg") +result = model(image)[0] +detections = sv.Detections.from_ultralytics(result) + +labels = [ + f"{class_name} {confidence:.2f}" + for class_name, confidence in zip(detections["class_name"], detections.confidence) +] + +annotated = box_annotator.annotate(scene=image.copy(), detections=detections) +annotated = label_annotator.annotate( + scene=annotated, detections=detections, labels=labels +) + +cv2.imwrite("annotated.jpg", annotated) +``` + +See `references/detection.md` for building `Detections` from other sources (inference, SAM3) and filtering, and `references/annotators.md` for the full annotator list and the compose pattern. + +The repo also publishes `docs/llms.txt` for general model-level facts and API surface; this skill focuses specifically on the mistakes agents repeatedly make in practice (wrong method names, deprecated APIs, silently-ignored kwargs) with runnable patterns, rather than restating the API reference. + +## Critical decision: InferencePipeline vs sv.process_video + +These solve the same problem — "run a model over every frame of a video/stream" — but they are not interchangeable. Picking the wrong one is the single most common architectural mistake in supervision code. + +**Use `sv.process_video` when:** + +- The input is a finite video *file* on disk and you want an output video file. +- You bring your own model call inside a `callback(frame, frame_index) -> np.ndarray`. +- You want the simplest possible script — no threading, no queues. + +```python +import supervision as sv + + +def callback(frame: sv.numpy.ndarray, frame_index: int) -> sv.numpy.ndarray: + result = model(frame)[0] + detections = sv.Detections.from_ultralytics(result) + return box_annotator.annotate(scene=frame.copy(), detections=detections) + + +sv.process_video( + source_path="input.mp4", + target_path="output.mp4", + callback=callback, + show_progress=True, # NOT `progress=True` — see references/video.md +) +``` + +**Use `roboflow/inference`'s `InferencePipeline` when:** + +- The source is a *live* stream — webcam, RTSP, or an infinite feed — not a video file you're transcoding. +- You need the model inference itself decoupled/threaded from frame reading for real-time throughput (InferencePipeline runs inference in a background thread and calls your `on_prediction` callback as results become available). +- You don't need a saved output video, or you'll build one yourself (e.g. with `sv.VideoSink`) inside the callback. +- You want Roboflow-hosted or local Roboflow models run for you, rather than calling `model(frame)` yourself each iteration. + +```python +from inference import InferencePipeline +import supervision as sv + +box_annotator = sv.BoxAnnotator() + + +def on_prediction(result, video_frame): + detections = sv.Detections.from_inference(result) + annotated = box_annotator.annotate( + scene=video_frame.image.copy(), detections=detections + ) + cv2.imshow("frame", annotated) + cv2.waitKey(1) + + +pipeline = InferencePipeline.init( + model_id="your-model/1", + video_reference=0, # webcam, or an RTSP URL + on_prediction=on_prediction, +) +pipeline.start() +pipeline.join() +``` + +**Rule of thumb:** file in, file out → `sv.process_video`. Live/streaming source, or you need async/threaded inference → `InferencePipeline`. Don't reach for `InferencePipeline` just to process a static mp4 — it adds threading complexity `process_video` doesn't need, and don't use `process_video` on an infinite/live source — it assumes a finite frame count from `VideoInfo`. + +## Reference files + +- `references/detection.md` — building `sv.Detections`, key attributes, filtering, common mistakes. +- `references/annotators.md` — annotator classes, correct parameter names, the compose pattern. +- `references/tracking.md` — tracking with the `trackers` package (`ByteTrackTracker`), why `sv.ByteTrack` is deprecated, correct parameter/method names, filtering by `tracker_id`. +- `references/video.md` — `sv.process_video`, `VideoInfo`, `VideoSink`. +- `references/utils.md` — `PolygonZone`, `LineZone`, `sv.Color` / `sv.ColorPalette`. diff --git a/skills/references/annotators.md b/skills/references/annotators.md new file mode 100644 index 0000000000..74591b2d3c --- /dev/null +++ b/skills/references/annotators.md @@ -0,0 +1,60 @@ +# Annotators + +All annotators implement `.annotate(scene, detections, ...) -> np.ndarray`. They never mutate `scene` in place as documented usage — pass `scene.copy()` (or reuse the returned array as the next annotator's input) rather than the original frame if you need the original preserved. + +## Common annotator classes + +| class | draws | notable params | +| --------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------- | +| `sv.BoxAnnotator` | rectangle bounding boxes | `color`, `thickness` | +| `sv.RoundBoxAnnotator` | rounded-corner boxes | `color`, `thickness`, `roundness` | +| `sv.BoxCornerAnnotator` | corner-only brackets | `color`, `thickness`, `corner_length` | +| `sv.LabelAnnotator` | text labels (needs `labels=[...]`) | `color`, `text_color`, `text_scale`, `text_padding`, `text_position` | +| `sv.RichLabelAnnotator` | text labels with custom font/unicode support | `font_path`, `text_color`, `text_scale` | +| `sv.MaskAnnotator` | filled segmentation masks | `color`, `opacity` | +| `sv.PolygonAnnotator` | mask/box outline as polygon | `color`, `thickness` | +| `sv.EllipseAnnotator` | ellipse under each box (good for people-tracking) | `color`, `thickness`, `start_angle`, `end_angle` | +| `sv.CircleAnnotator` | circle around each box center | `color`, `thickness` | +| `sv.DotAnnotator` | filled dot at box center/anchor | `color`, `radius`, `position` | +| `sv.TriangleAnnotator` | triangle marker above box | `color`, `base`, `height` | +| `sv.HaloAnnotator` | glow/halo around mask | `color`, `opacity`, `kernel_size` | +| `sv.HeatMapAnnotator` | cumulative heatmap across frames | `position`, `opacity`, `radius` | +| `sv.BlurAnnotator` | blur out detected regions | `kernel_size` | +| `sv.PixelateAnnotator` | pixelate detected regions | `pixel_size` | +| `sv.TraceAnnotator` | draws tracked path history, needs `tracker_id` | `color`, `position`, `trace_length` | +| `sv.CropAnnotator` | pastes a zoomed crop of each detection back on the scene | `position`, `scale` | +| `sv.IconAnnotator` | places a custom image/icon at each detection | `icon_path` or `icon_resolver`, `icon_scale` | +| `sv.PercentageBarAnnotator` | draws a confidence bar under each box | `color`, `height`, `width` | + +Most annotators accept `color=sv.Color(...)` / `sv.ColorPalette(...)` and a `color_lookup` argument (`sv.ColorLookup.CLASS`, `.INDEX`, `.TRACK`) controlling whether color is assigned per class, per detection index, or per tracker id. + +## Compose pattern (chain annotators) + +Annotators are meant to be chained — call each one with the previous output as the new `scene`: + +```python +box_annotator = sv.BoxAnnotator() +label_annotator = sv.LabelAnnotator() +trace_annotator = sv.TraceAnnotator() + +annotated = scene.copy() +annotated = trace_annotator.annotate(scene=annotated, detections=detections) +annotated = box_annotator.annotate(scene=annotated, detections=detections) +annotated = label_annotator.annotate( + scene=annotated, detections=detections, labels=labels +) +``` + +Order matters visually — draw fills/masks/traces first, then outlines, then labels on top so text isn't obscured. + +## Common mistake + +```python +# WRONG — this class does not exist in supervision +annotator = sv.BoundingBoxAnnotator() + +# RIGHT +annotator = sv.BoxAnnotator() +``` + +Other name mix-ups worth double-checking against `src/supervision/annotators/core.py` before using: `sv.LabelAnnotator` (not `TextAnnotator`), `sv.MaskAnnotator` (not `SegmentationAnnotator`), and `sv.TraceAnnotator` (not `PathAnnotator` / `TrajectoryAnnotator`). diff --git a/skills/references/detection.md b/skills/references/detection.md new file mode 100644 index 0000000000..12d5e89aa7 --- /dev/null +++ b/skills/references/detection.md @@ -0,0 +1,94 @@ +# Detections + +`sv.Detections` is a single dataclass-like container used across the whole library. Every model integration converts its native output into one of these — always prefer the `from_*` constructor over building `Detections(...)` by hand. + +## Creating Detections from common sources + +```python +import supervision as sv + +# Ultralytics (YOLOv8/v9/v10/11, SAM, etc.) +result = model(image)[0] +detections = sv.Detections.from_ultralytics(result) + +# Roboflow inference (hosted or `inference` package) +result = model.infer(image)[0] +detections = sv.Detections.from_inference(result) + +# Segment Anything (SAM / SAM2 / SAM3) +sam_result = mask_generator.generate(image) +detections = sv.Detections.from_sam(sam_result) + +# Transformers (e.g. DETR, Grounding DINO via HF pipeline) +detections = sv.Detections.from_transformers(transformers_results, id2label=id2label) +``` + +Each `from_*` method normalizes the model's native output into the same set of attributes below — this is the whole point of using them instead of parsing raw model output yourself. + +## Key attributes + +| attribute | shape / type | notes | +| ------------ | -------------------------------- | ----------------------------------------------------------------------------------------------------- | +| `xyxy` | `np.ndarray (N, 4)` | float, `[x1, y1, x2, y2]` per box, always present | +| `confidence` | `np.ndarray (N,)` or `None` | float scores | +| `class_id` | `np.ndarray (N,)` or `None` | integer class ids | +| `tracker_id` | `np.ndarray (N,)` or `None` | set after running a tracker, not by detection alone | +| `mask` | `np.ndarray (N, H, W)` or `None` | boolean segmentation masks | +| `data` | `dict` | extra per-detection arrays, e.g. `data["class_name"]`; also accessible via `detections["class_name"]` | + +`len(detections)` gives the number of boxes (`N`). `Detections` is empty-safe: with zero detections, arrays have shape `(0, 4)` / `(0,)` rather than being `None`. + +## Filtering patterns + +`Detections` supports NumPy-style boolean-mask indexing directly — this is the correct and idiomatic way to filter. It does **not** have a `.filter()` method. + +```python +# keep only class_id == 0 (e.g. "person") +detections = detections[detections.class_id == 0] + +# confidence threshold +detections = detections[detections.confidence > 0.5] + +# combine conditions +detections = detections[(detections.class_id == 0) & (detections.confidence > 0.5)] + +# keep detections inside a set of classes +detections = detections[np.isin(detections.class_id, [0, 2, 3])] + +# by area +detections = detections[detections.area > 1000] + +# slicing / indexing a single detection +first = detections[0] +``` + +`Detections` also supports `+` to merge two instances and `sv.Detections.merge([d1, d2])` for combining more than two. + +## Common mistakes + +```python +# WRONG — Detections has no .filter() method +detections = detections.filter(lambda d: d.class_id == 0) + +# RIGHT — boolean-mask indexing +detections = detections[detections.class_id == 0] +``` + +```python +# WRONG — comparing class_id to a class name string +detections = detections[detections.class_id == "person"] + +# RIGHT — class_id is an integer id; compare to the class name via class_name data, +# or map the name to its integer id first +detections = detections[detections["class_name"] == "person"] +``` + +```python +# WRONG — assuming confidence/class_id are always populated +avg_conf = detections.confidence.mean() # crashes if confidence is None + +# RIGHT — guard when the source may not populate a field (e.g. some SAM masks +# have no class_id/confidence) +if detections.confidence is not None: + avg_conf = detections.confidence.mean() +``` diff --git a/skills/references/tracking.md b/skills/references/tracking.md new file mode 100644 index 0000000000..6a0ad9bfad --- /dev/null +++ b/skills/references/tracking.md @@ -0,0 +1,95 @@ +# Tracking + +## sv.ByteTrack is deprecated — use the `trackers` package + +`supervision`'s built-in `sv.ByteTrack` is deprecated. The current, maintained way to track objects is the standalone `trackers` package (`pip install trackers`), which provides `ByteTrackTracker` (plus `SORTTracker`, `OCSORTTracker`, `BoTSORTTracker`). It still consumes/returns `sv.Detections`, so everything else in this skill (filtering, annotating) works unchanged — only the tracker object and its update method differ. + +```python +import cv2 +import supervision as sv +from trackers import ByteTrackTracker + +tracker = ByteTrackTracker( + track_activation_threshold=0.25, + lost_track_buffer=30, + minimum_consecutive_frames=3, + minimum_iou_threshold=0.3, +) + +trace_annotator = sv.TraceAnnotator() +box_annotator = sv.BoxAnnotator() +label_annotator = sv.LabelAnnotator() + +for frame in frames: + result = model(frame)[0] + detections = sv.Detections.from_ultralytics(result) + + detections = tracker.update(detections) # NOT `.update_with_detections()` + + labels = [f"#{tracker_id}" for tracker_id in detections.tracker_id] + + annotated = trace_annotator.annotate(scene=frame.copy(), detections=detections) + annotated = box_annotator.annotate(scene=annotated, detections=detections) + annotated = label_annotator.annotate( + scene=annotated, detections=detections, labels=labels + ) + +tracker.reset() # call between videos/streams to clear track state +``` + +`ByteTrackTracker` is stateful, like the old `sv.ByteTrack` — create **one** instance per video/stream and reuse it every frame; creating a new instance per frame resets tracking. + +### Correct constructor parameters (`ByteTrackTracker`) + +- `track_activation_threshold` (default `0.25`) — minimum detection confidence to start a new track. Not `confidence_threshold`. +- `lost_track_buffer` (default `30`) — frames to keep a track alive with no matching detection before dropping it. +- `minimum_consecutive_frames` (default `3`) — consecutive detections required before a track is confirmed; suppresses spurious one-frame detections. +- `minimum_iou_threshold` (default `0.3`) — IOU threshold for matching detections to existing tracks. This replaced the old `minimum_matching_threshold` name. + +### Correct method name + +- `tracker.update(detections) -> Detections` — NOT `update_with_detections()` (that was the `sv.ByteTrack` method name) and NOT plain `update()` semantics from other libraries — pass the `Detections` object, get a new `Detections` back with `tracker_id` populated. +- `tracker.update(detections, timestamp=...)` — pass a monotonic `timestamp` in seconds if your pipeline has irregular/dropped frames, so Kalman prediction and lost-track pruning match the real time gap instead of assuming a fixed frame rate. +- `tracker.reset()` — clears all track state; call this between videos, not just once at startup. + +```python +# WRONG — this is the deprecated sv.ByteTrack method name +detections = tracker.update_with_detections(detections) + +# RIGHT — ByteTrackTracker from the `trackers` package +detections = tracker.update(detections) +``` + +## Legacy: sv.ByteTrack (deprecated, still present in supervision) + +You may still encounter `sv.ByteTrack` in older code. It behaves the same way conceptually but with different names — recognize it, don't write new code against it: + +```python +tracker = sv.ByteTrack( + track_activation_threshold=0.25, + lost_track_buffer=30, + minimum_matching_threshold=0.8, # note: different default/name than the new package + frame_rate=30, +) +detections = tracker.update_with_detections(detections) # old method name +``` + +If you see this pattern in an existing codebase, prefer migrating it to `ByteTrackTracker` from `trackers` rather than extending it further. + +## Filtering by tracker_id + +After tracking, `tracker_id` is just another attribute you can boolean-index on, same as `class_id`: + +```python +# only detections that have been assigned a tracker id (drop unmatched, if any) +detections = detections[detections.tracker_id != None] # noqa: E711 (elementwise, not `is not None`) + +# keep only a specific tracked object +detections = detections[detections.tracker_id == 7] + +# exclude ids you've already counted/processed +seen_ids = {1, 2, 3} +detections = detections[~np.isin(detections.tracker_id, list(seen_ids))] +``` + +Note `tracker_id` is `None` on a `Detections` object until it has been passed through `tracker.update(...)` at least once — accessing it before that raises/returns `None`, it is not auto-populated by `from_ultralytics` / `from_inference` alone. Also note `detections.tracker_id != None` (elementwise numpy comparison) is intentional here, not a mistake — `is not None` would do a Python identity check on the whole array instead of an elementwise mask. diff --git a/skills/references/utils.md b/skills/references/utils.md new file mode 100644 index 0000000000..fd61c47fe5 --- /dev/null +++ b/skills/references/utils.md @@ -0,0 +1,77 @@ +# Zones, colors, and other utils + +## PolygonZone + +`sv.PolygonZone` tests which detections fall inside an arbitrary polygon region (e.g. a parking spot, a doorway, a lane). It does not draw anything itself — pair it with `sv.PolygonZoneAnnotator` for visualization. + +```python +import numpy as np +import supervision as sv + +polygon = np.array([[100, 200], [400, 200], [400, 500], [100, 500]]) + +zone = sv.PolygonZone(polygon=polygon) +zone_annotator = sv.PolygonZoneAnnotator(zone=zone, color=sv.Color.RED) + +for frame in frames: + result = model(frame)[0] + detections = sv.Detections.from_ultralytics(result) + + mask = zone.trigger(detections=detections) # bool array, one per detection + detections_in_zone = detections[mask] + + annotated = zone_annotator.annotate(scene=frame.copy()) + annotated = box_annotator.annotate(scene=annotated, detections=detections_in_zone) +``` + +`zone.trigger(detections)` returns a boolean NumPy array the same length as `detections` — `True` where that detection's anchor point is inside the polygon. Use it directly as a boolean mask; it does not filter `detections` for you. `zone.current_count` holds the count of detections inside the zone after the most recent `trigger()` call. + +## LineZone for counting + +`sv.LineZone` counts detections crossing a line, split into two directions. + +```python +start, end = sv.Point(0, 300), sv.Point(1280, 300) +line_zone = sv.LineZone(start=start, end=end) +line_zone_annotator = sv.LineZoneAnnotator() + +for frame in frames: + result = model(frame)[0] + detections = sv.Detections.from_ultralytics(result) + detections = tracker.update_with_detections(detections) # LineZone needs tracker_id + + line_zone.trigger(detections=detections) + + annotated = line_zone_annotator.annotate(frame=frame.copy(), line_counter=line_zone) + +print(line_zone.in_count, line_zone.out_count) +``` + +- `line_zone.in_count` / `line_zone.out_count` — cumulative counters, updated in-place by each `trigger()` call (there's no return value to capture). +- `LineZone.trigger` requires `detections.tracker_id` to be populated — run a tracker (`sv.ByteTrack`) before calling it, otherwise a crossing can't be attributed to a consistent object between frames. + +## sv.Color constants and from_hex + +```python +sv.Color.RED +sv.Color.GREEN +sv.Color.BLUE +sv.Color.BLACK +sv.Color.WHITE + +sv.Color.from_hex("#FF5733") +sv.Color.from_hex("FF5733") # leading # optional +sv.Color.from_rgb_tuple((255, 87, 51)) +``` + +## sv.ColorPalette.DEFAULT + +Most annotators default to `sv.ColorPalette.DEFAULT` when no `color=` is given, which cycles a distinct color per class/track index automatically — you usually don't need to build a custom palette unless you want specific brand colors. + +```python +box_annotator = sv.BoxAnnotator(color=sv.ColorPalette.DEFAULT) + +# custom palette +palette = sv.ColorPalette.from_hex(["#e6194b", "#3cb44b", "#ffe119", "#4363d8"]) +box_annotator = sv.BoxAnnotator(color=palette, color_lookup=sv.ColorLookup.CLASS) +``` diff --git a/skills/references/video.md b/skills/references/video.md new file mode 100644 index 0000000000..020ee41411 --- /dev/null +++ b/skills/references/video.md @@ -0,0 +1,79 @@ +# Video + +## sv.process_video + +`sv.process_video` reads a finite video file frame-by-frame, applies a callback, and writes the result to a new video file. It handles the read/write loop and progress bar for you. + +```python +import supervision as sv + + +def callback(frame, frame_index: int): + result = model(frame)[0] + detections = sv.Detections.from_ultralytics(result) + return box_annotator.annotate(scene=frame.copy(), detections=detections) + + +sv.process_video( + source_path="input.mp4", + target_path="output.mp4", + callback=callback, + show_progress=True, +) +``` + +### Critical: the parameter is `show_progress`, not `progress` + +```python +# WRONG — `progress` is not a recognized kwarg; this silently does nothing +# (no error is raised, you just get no progress bar — easy to miss in code review) +sv.process_video( + source_path="input.mp4", target_path="output.mp4", callback=callback, progress=True +) + +# RIGHT +sv.process_video( + source_path="input.mp4", + target_path="output.mp4", + callback=callback, + show_progress=True, +) +``` + +Because passing an unexpected keyword like `progress` doesn't always raise immediately depending on the function signature/version, always verify the actual parameter name against the installed version's signature (`help(sv.process_video)`) rather than assuming. + +## VideoInfo + +`sv.VideoInfo.from_video_path(path)` returns metadata about a video file: + +```python +video_info = sv.VideoInfo.from_video_path("input.mp4") + +video_info.width # int, pixels +video_info.height # int, pixels +video_info.fps # float — NOT guaranteed to be a whole number (e.g. 29.97) +video_info.total_frames # int, or None if it can't be determined +``` + +`fps` is a **float**. Don't do `int(video_info.fps)` when the real value matters (e.g. computing timestamps from frame index) — truncating 29.97 to 29 introduces drift over a long video. Use it directly in float math: + +```python +timestamp_seconds = frame_index / video_info.fps +``` + +## VideoSink for manual frame writing + +Use `sv.VideoSink` directly (instead of `process_video`) when you need more control than a single callback gives you — e.g. skipping frames, writing frames from a live source, or writing inside an `InferencePipeline` callback. + +```python +video_info = sv.VideoInfo.from_video_path("input.mp4") + +with sv.VideoSink(target_path="output.mp4", video_info=video_info) as sink: + for frame in sv.get_video_frames_generator(source_path="input.mp4"): + result = model(frame)[0] + detections = sv.Detections.from_ultralytics(result) + annotated = box_annotator.annotate(scene=frame.copy(), detections=detections) + sink.write_frame(annotated) +``` + +`sv.get_video_frames_generator(source_path=...)` is the underlying frame-reading generator `process_video` uses internally — reach for it when you need a `for` loop over frames instead of a callback style.