Open-source SAE Level 2 ADAS for CAN-closed vehicles.
Most retrofit ADAS projects assume you can write to the vehicle's CAN bus. On the vast majority of production cars you cannot — the bus broadcasts, but accepts no control commands. Vision Pilot works anyway: it reads speed from the factory bus and actuates every control at the analog signal level, upstream of the stock ECU. The factory wiring stays intact, and cutting one switch returns the car to stock.
Road-validated on a Honda Brio (2014) and a Toyota Avanza.
| Feature | What it does | Status |
|---|---|---|
| ACC — Adaptive Cruise Control | Holds a set speed, backs off behind a lead vehicle using a Mamdani fuzzy controller over measured gap | Road-validated |
| AEB — Automatic Emergency Braking | Always armed; full brake below an 8 m in-path obstacle | Road-validated |
| LKAS — Lane Keeping Assist | PID on lateral offset from lane centre, actuated through the EPS torque lines | Road-validated |
| DMS — Driver Monitoring | Drowsiness (eye/mouth aspect ratio) and gaze-off-road from 68 facial landmarks | Road-validated |
| Teleoperation | Remote/keyboard driving with pedal and steering feedback | Bench-validated |
The stack runs YOLOPv2 for joint object detection, drivable-area segmentation and lane-line segmentation, with YOLOv8n as a secondary detector and SORT for tracking. Depth comes from a ZED stereo camera, so obstacle distance is measured rather than inferred from box size.
 |
 |
| Detection (yellow), drivable area (green), lane lines (red) | Dense urban traffic |
 |
 |
| Lane geometry through a curve | Low-light performance |
The four frames above are reference output of the YOLOPv2 network the stack uses. Below is a raw capture from the vehicle's own ZED camera, and the head-unit photo at the top of this page shows the complete pipeline running live in the car.
Two processing units. The Jetson Orin Nano runs perception, the feature controllers and the dashboard. The STM32 Nucleo-F446RE owns the actuators and closes the fast inner loops. A USB serial link joins them; the vehicle CAN bus is tapped read-only.
flowchart LR
subgraph SENSE["Sensors"]
ZED["ZED stereo camera"]
DRVCAM["Driver camera"]
CANBUS["OBD-II CAN tap"]
end
subgraph HIGH["High-level — Jetson Orin Nano"]
PERC["Perception<br/>YOLOPv2 + YOLOv8 + SORT"]
CTRL["Controller<br/>ACC / AEB / LKAS"]
DMS["Driver monitoring"]
GUI["Dashboard"]
end
subgraph LOW["Low-level — STM32 F446RE"]
PID["Cascaded PID"]
end
subgraph ACT["Actuators"]
ETC["Throttle<br/>signal injection"]
SERVO["Brake<br/>servo on pedal"]
EPS["Steer<br/>EPS torque lines"]
end
ZED --> PERC
DRVCAM --> DMS
CANBUS -- "speed 0x1D0" --> CTRL
PERC -- "distance, lane delta" --> CTRL
PERC --> GUI
DMS --> GUI
GUI -- "mode" --> PERC
CTRL -- "serial 9600" --> PID
PID --> ETC
PID --> SERVO
PID --> EPS
How each actuator is driven, without CAN write access:
- Throttle — the ECU's two accelerator-position signals are synthesised by the STM32 instead of coming from the pedal. Signal 2 is held at half of signal 1; the ECU cross-checks the pair.
- Brake — a 2.5 N·m servo pulls the pedal through a steel cable. Purely additive, so the driver can always override by pressing harder.
- Steering — the EPS torque-sensor pair is synthesised anti-phase, so the power steering assists as though the driver were applying torque.
Full detail in docs/architecture.md.
git clone https://github.com/adeirman46/Vision_Pilot_CAN_Closed.git
cd Vision_Pilot_CAN_Closed
scripts/setup-env.sh --dms # venv + deps + .env
source .venv/bin/activate
scripts/fetch-models.sh # weights (~250 MB, not committed)
scripts/can-up.sh # bring up slcan0 from the CANable
scripts/run.sh calibrate # click the road ROI, once per camera mount
scripts/run.sh adas # launch the full stackThe ZED SDK must be installed separately — pyzed is not on PyPI. See
docs/setup.md.
Weights are not committed. scripts/fetch-models.sh pulls them automatically;
if Drive rate-limits the download, get them from the same folder by hand:
➡ Google Drive — model weights
| File | Size | Used by |
|---|---|---|
yolopv2.pt |
~150 MB | detection + drivable area + lane segmentation |
yolov8n.pt |
~6 MB | secondary object detector |
shape_predictor_68_face_landmarks.dat |
~100 MB | driver monitoring (optional) |
Place all three in models/.
Everything runnable goes through one script. Multi-process targets start in dependency order, prefix each process's output, and tear the whole group down if any member dies.
scripts/run.sh adas # full stack, mode selectable from the dashboard
scripts/run.sh acc # one feature at a time
scripts/run.sh aeb --no-gui # headless
scripts/run.sh lkas
scripts/run.sh perception # single processes, for debugging
scripts/run.sh dashboard
scripts/run.sh dms
scripts/run.sh bench:brake # open-loop actuator characterisation
scripts/run.sh bench:steer
scripts/run.sh bench:throttle
scripts/run.sh adas --dry-run # print the launch plan and exit
scripts/run.sh perception -- --conf-thres 0.5 --device 0
scripts/run.sh --helpFlash the matching firmware to the STM32 first — see firmware/README.md.
Measured on-vehicle. Full tables and plots in docs/test-results.md.
| Test | Requirement | Measured | |
|---|---|---|---|
| Brake, command → full travel | < 1.17 s | ~0.60 s | pass |
| Throttle, rise for +1 km/h | < 0.51 s | 337 ms ± 46 ms | pass |
| Steering response | < 0.30 s | 0.12 – 0.20 s | pass |
| AEB stop from ~9 km/h | — | ~2.1 s from 8 m | pass |
| ACC throttle response | ≤ 0.5 s | 0.14 – 0.59 s | pass |
| LKAS centreline hold | — | converges, tracks curves | pass |
| Driver override | reverts to manual | immediate | pass |
| System power | < 180 W | 57.5 W | pass |
 |
 |
| AEB — brake commanded at 8 m, standstill in 2033 ms | LKAS — converges from an off-centre start and holds |
src/vision_pilot/ Python package — shared by every feature
config.py paths, UDP port map, CAN IDs, tuning (single source of truth)
bus/ CAN speed reader
perception/ YOLOPv2 + YOLOv8 + SORT; vendored YOLOPv2 helpers
control/ adas, acc, aeb, lkas
driver_monitoring/ drowsiness and gaze detection
ui/ PyQt6 dashboard and plan-view widget
firmware/ STM32 sketches — one per run target
tools/bench/ actuator step-response logging
scripts/ run.sh launcher + CAN, model and env setup
config/roi.txt calibrated perspective ROI
models/ weights (fetched, not committed)
docs/ architecture, hardware, CAN, setup, results
tests/ configuration smoke tests
Perception, dashboard, CAN and plan-view code used to be copy-pasted into every
feature folder — read_can.py alone existed in seven identical copies. They now
live once in src/vision_pilot/ and every feature imports them.
No machine-specific paths or magic numbers in the source. Everything lives in
src/vision_pilot/config.py and is overridable
per vehicle via .env (copy from .env.example):
VP_CAN_ID_SPEED=0x1D0 # speed frame arbitration ID
VP_SPEED_POLY_A=-0.00000016 # raw counts -> km/h, refit per vehicle
VP_SPEED_POLY_B=0.00650007
VP_SPEED_POLY_C=-1.15230758
VP_MCU_PORT=/dev/ttyACM0
VP_AEB_TRIGGER_DISTANCE_M=8.0Porting to another vehicle mostly means refitting the speed polynomial — procedure in docs/can-bus.md.
| Setup | Install, ZED SDK, CAN bring-up, calibration, troubleshooting |
| Architecture | System and process topology, per-feature control flow |
| Hardware | Bill of materials, power budget, enclosures, actuator interfaces |
| CAN bus | Wiring, decoded frames, speed conversion, porting |
| Test results | Full validation campaign |
| Firmware | Pin map, host protocol, which sketch to flash |
A printable walkthrough covering all of the above is generated by
scripts/build-docs-pdf.sh → docs/VisionPilot-Tutorial.pdf.
This drives a real vehicle. Read this before running it on a road.
- It is Level 2. The driver is responsible at all times and must stay ready to take over.
- Test on closed ground first. Bench each actuator
(
scripts/run.sh bench:*) before any closed-loop drive. - Verify the override every time. Two panel switches cut power to the processing units; with them off the vehicle is fully manual.
- The CAN tap is read-only. Vision Pilot never transmits on the vehicle bus. Actuation goes over a separate serial link to a dedicated microcontroller, so a fault in the perception stack cannot inject frames onto the vehicle network.
- The speed polynomial is vehicle-specific. Running with another car's coefficients gives wrong speed, and every controller downstream depends on it.
- YOLOPv2 — panoptic driving perception;
helper utilities are vendored under
src/vision_pilot/perception/yolopv2/. - SORT — multi-object tracking.
- Ultralytics YOLOv8 — secondary detector.
- Stereolabs ZED SDK — stereo depth.