vnc: fix blue channel shift using red_max instead of blue_max in convert_pixels

[Copilot]

VGA output via VNC showed severe color degradation (posterization, channel bleed) when VNC clients requested sub-32bpp pixel formats such as 8bpp RGB 3:3:2.

Root cause

Copy-paste bug in convert_pixels: shift_b was computed using red_max instead of blue_max:

// Before (wrong):
let shift_b = 8 - fmt.red_max.get().count_ones();

// After (fixed):
let shift_b = 8 - fmt.blue_max.get().count_ones();

Source pixels are 0x00RRGGBB. The shift for each channel is derived from how many significant bits the destination format allocates to that channel:

Channel	Source bits	Correct formula
R (bits 23–16)	8	24 - count_ones(red_max)
G (bits 15–8)	8	16 - count_ones(green_max)
B (bits 7–0)	8	8 - count_ones(blue_max) ← was red_max

This is invisible for symmetric formats (32bpp, RGB565) where red_max == blue_max. It breaks 8bpp RGB 3:3:2 (red_max=7, blue_max=3): shift_b is 5 instead of 6, producing out-of-range blue values (0–7 instead of 0–3) that overflow into the adjacent green channel.

Changes

• workers/vnc_worker/vnc/src/lib.rs: One-character fix to shift_b calculation.
• Tests added: convert_pixels unit tests covering the 32bpp no-convert fast path, 8bpp 3:3:2 asymmetric channel depths (regression case), and 16bpp RGB565.

[bitranox]

Nice catch on the blue-channel shift, we hit the same bug while doing a broader VNC overhaul on our fork (vnc-multiclient-dirtydirtydirty branch), and ended up fixing it as part of a small refactor of convert_pixels. Sharing in case it's useful for follow-up:

The same fix, factored out: Instead of patching the shift in place, we extracted a PixelConversion struct that's built once when the client's pixel format is set (lib.rs:103-162). convert_pixels itself then just does shift/mask in the hot loop, no per-call recomputation of red_bits/green_bits/blue_bits. For VGA at 60 Hz this is a small but free win, and it makes the channel-width logic exist in exactly one place, which is what made the original red_max/blue_max typo possible in the first place.

Passthrough fast-path: Most real clients negotiate native 32bpp BGRX, where every shift/mask is a no-op. We detect that case in the constructor (passthrough: bool) and skip the whole loop. Worth considering as a follow up here too, it's maybe 15 lines and pays for itself on every frame for the common case.

Bit width via leading_zeros rather than count_ones: Equivalent for any 2^n - 1 mask (so identical for every spec-compliant client), but robust to non contiguous masks like red_max = 5 (0b101) where count_ones would underreport the spanned width. Pure defensive; costs nothing.

Tests: We landed the same three cases you have (32bpp identity, RGB565, 8bpp 3:3:2 asymmetric) plus an empty_input edge case and a dedicated regression named after the bug, so a future bisect lands on a test whose name says exactly what broke.

only fixing the blue_max gave us wrong grey colour on gray windows decorators with real-vnc viewer (check visually !!)

[bitranox]

@benhills : check #3233