feat(wasm-hv): in-memory CXO telemetry — browser visor reports transport bandwidth+latency to TPD#3362
Open
0pcom wants to merge 13 commits into
Open
feat(wasm-hv): in-memory CXO telemetry — browser visor reports transport bandwidth+latency to TPD#33620pcom wants to merge 13 commits into
0pcom wants to merge 13 commits into
Conversation
…ilds under js/wasm The CXO/skyobject stack transitively imported bbolt (via cxds+idxdb on-disk drives), which does not build for GOOS=js (arch-specific consts + mmap) — so the CXO publisher could not run in a wasm visor even with InMemoryDB. Split the bbolt-backed drive.go behind //go:build !js and add drive_js.go stubs providing DriveOptions + the NewDrive* constructors (which error, since a wasm visor uses the in-memory CXDS/IdxDB via skyobject's InMemoryDB path). Moved the shared panicf helper out of drive.go into the untagged memory.go. Result: pkg/cxo/treestore (the full publish path) now compiles under js/wasm; native on-disk path unchanged (cxds/idxdb tests pass). Foundation for wasm-visor in-memory CXO telemetry -> TPD.
…atency to TPD The browser visor now runs a CXO TreeStore publisher (InMemoryDB — no filesystem) that reports each transport's bandwidth + latency to TPD via its cxo-aggregator, exactly like a native visor (init_stats.go) but without the bbolt stats.Store. telemetry_js.go: builds the in-memory publisher, wires tpM.SetTPDLeafPublisher (transports/<uuid>/entry + tombstone), samples transports/<uuid>/current (liveSnapshot: sent/recv + latency min/max/avg) each minute, and announces to TPD every 30s so it subscribes. Wired into bootEdge after the transport manager serves. Also fix skyobject.NewContainer to skip the DataDir mkdir when InMemoryDB — the default DataDir resolves to /tmp under js/wasm where 'mkdir not implemented on js' was fatal; in-memory mode never touches those files anyway. Runtime-verified: publisher comes up (feed=<self> → TPD), builds on native + js/wasm.
… stale deadcode nolint Compliance pass against the Go coding standard's mechanical review engine (gofmt / go vet / golangci-lint / native + js-wasm build): - gofmt the wasm-visor JS-hooks map (alignment drift from telemetry wiring) - pkg/dmsg/dmsg/util.go: //nolint:gosec G709 — gob frame arrives over a Noise-authenticated dmsg session, fixed internal decode target (false positive) - usermanager cookies: //nolint:gosec G124 — Secure/HttpOnly/SameSite ARE set via s.c config accessors gosec's flow analysis can't see through (false positive) - mux-probe-assert: drop stale ,deadcode from nolint (folded into unused upstream) All pre-existing findings surfaced only by a newer-than-CI golangci-lint; CI was already green. No behavioral change.
Routine dependency refresh — all minor/patch bumps, no major-version changes: grpc 1.81.1→1.82.0, pion/webrtc v4.2.15→v4.2.16 (+sctp/rtp/rtcp/turn), go-proxyproto 0.12.0→0.14.0, maxminddb v2.4.0→v2.4.1, starlark, genproto, klauspost/cpuid, gopherjs, lufia/plan9stats, shoenig/go-m1cpu. Verified: native build, GOOS=js GOARCH=wasm build (wasm-visor + CXO), and go vet all clean.
…code table - refresh docs/skywire-goda-graph.svg (goda graph | dot -Tsvg) - append a Lines of Code section (gocloc, vendor/node_modules/.git excluded): 1740 Go files / 267,754 LOC; ~403k LOC total across all languages
browse.js: add createChatWindow — a 1:1 skychat client as a WinBox desktop window (the missing peer to the skynet browser), driving the existing wasm-visor JS hooks skychatSend(peerPk,text)/skychatMessages(). Distinct sender PKs from the message buffer surface as clickable chips so an incoming message from an unknown peer is discoverable. Added to the app menu as 'chat', gated on skywireVisor.skychatSend (wasm-visor only; native keeps its Angular skychat tab), exactly like the host window. Rebuild ./skywire re-embeds it (browse.js is go:embed'd in browseui, not in the wasm.gz blob). docs/skychat-refactor-rfc.md: design for extracting a shared pkg/skychat core (one message model + wire codec, federated-only groups, one Transport interface, history behind a build-tag-split store) so native + wasm stop diverging — the visorcore convergence pattern. Includes public-group service discovery via a new servicedisc ServiceTypeSkychat='skychat' (same directory mechanism as type=proxy), and causal-ish reorder-buffer display ordering.
… step 2) The skychat length-prefixed frame protocol (4-byte big-endian length + payload, 64 KiB cap, chat-msg/chat-ack envelope) was implemented THREE times — the native app (framedConn), the browser-tab wasm visor (chatConn), and the group relay (session.go readFrame/writeFrame) — each with its own constant and write mutex, free to drift. This collapses them into one codec: pkg/skychat/message. New package pkg/skychat/message (pure Go, compiles under js/wasm): - WriteFrame/ReadFrame (io.Writer/io.Reader), MaxFrameSize - Conn: net.Conn + write mutex, WriteFrame/WriteFrameDeadline/ReadFrame - Envelope (chat-msg/chat-ack) + TypeMsg/TypeAck + ParseEnvelope/Marshal - message_test.go: wire round-trip, reject cases, Conn framing-compat, envelope parse Consumers migrated (wire bytes unchanged): - cmd/apps/skychat: framedConn is now a type alias for message.Conn; chatEnvelope/ chatTypeMsg/chatTypeAck alias the message types; tryHandleChatEnvelope uses message.ParseEnvelope. Deleted the local frame methods + skychatMaxFrameSize. - cmd/apps/skychat/group/session.go: relay framing uses message.ReadFrame/WriteFrame; deleted local readFrame/writeFrame + relayMaxFrameSize. - cmd/wasm-visor/skychat_js.go: chatConn/readFrame/writeFrame/skychatMaxFrame/ chatAckEnvelope replaced with message.Conn + message.Envelope. Blob regenerated. Validated: message unit tests + full skychat suite (commands/group/pairing) pass; native + js/wasm build/vet/lint clean; live harness — wasm visor boots, skychat listens on dmsg:1, desktop chat window opens.
… (step 4) Measured: a member joining a federated CXO group took ~30-40s to receive its first message. Root cause: when two members start close together each fails the other's INITIAL peer-sub dial (peer not listening yet), and the retry cadence was slow — the in-visor Manager reconnect loop ticks every 30s, and the standalone CXO-group driver re-attempted every 15s (20s timeout). A group.Session does not self-retry a peer-sub whose first Connect failed, so first-message latency was gated by whichever slow driver applied. Fix (both drivers, same class): - Manager.runReconnectLoop: adaptive cadence — tick at reconnectWarmupInterval (3s) while any peer is still 'warming' (never connected AND under its reconnectBackoffFailures1 budget), easing back to reconnectInterval (30s) once every peer is attached. New hasWarmingPeer() + peerReconnectFailures() gate it. Steady-state groups keep the low-churn 30s cadence; a dead peer stops qualifying after its budget and falls to the 5min/30min backoff, so it can't pin the loop fast. - Standalone startCXOGroup driver: 15s->3s ticker, 20s->10s per-attempt timeout. It already skips attached peers, so a fast tick idles cheaply post-convergence. Verified with two standalone CXO-TCP group processes (simultaneous start, the hard case): member->owner first arrival 2s (was ~30-40s), owner->member 3s. Full skychat suite + group tests pass; gofmt/vet/lint clean.
Prep for wasm-visor group chat. The group record store was bbolt-backed, and bbolt can't compile under GOOS=js GOARCH=wasm (arch-const MaxAllocSize + mmap) — so the group package (and thus group chat) couldn't build for the browser visor. Split store.go the same way the CXO cxds/idxdb datastore was split: - store_bbolt.go (//go:build !js): the existing bbolt impl, unchanged. - store_memory.go (//go:build js): a wire-identical in-memory impl — same *Store type + method set, records held as JSON bytes keyed by group ID so value semantics (independent copies on read, legacy-Admins normalization, lexicographic List order) match bbolt exactly. Ephemeral (resets on tab reload; the federated network retains published messages). Designed so an IndexedDB-backed durable variant can hydrate/flush this map behind the identical API later. The group package now compiles under js/wasm; native build + group tests unchanged. No behavior change on native. Next: plumb InMemoryDB through group.Config and wire group.Manager into the wasm visor.
…r wasm Adds Config.InMemoryDB: when set, newPublisher runs the session's CXO tree fully in memory over the dmsg transport (the native-TCP standalone path already forced this internally). Relaxes the 'DataDir required' check when InMemoryDB is set. This lets a browser (js/wasm) visor — which has no filesystem — be a full federated group member: publish its own feed in memory, subscribe to peers over dmsg. Native behavior unchanged (InMemoryDB defaults false). Builds native + js/wasm; group tests pass.
…vity log pane Two additions to the desktop skychat window (createChatWindow), matching the browser window's controls: - Transport selector (dmsg | skynet): skychatSend now takes an optional network arg; sendChat dials the chosen appnet.Type and caches conns per (network, peer) so the two transports don't clobber each other. Default dmsg (unchanged). - Activity log pane (🐞): subscribes to the shared window.skywireLog ring (same source the 'logs' window reads), filtered to skychat lines, so the operator sees the dial/connect/send/receive steps — like the skysocks-lite route-setup log in the browser window. sendChat now vlog's those steps. Verified live via hvinspect on the :8443 harness: selector has dmsg+skynet, log pane shows 'skychat: dialing dmsg <pk>:1…' after a send. wasm blob regenerated.
docs/wasm-visor-ui-tour.md — a screenshot-driven tour of the browser-tab visor: dashboard/visor-list, the WinBox mini-desktop app menu, and each window (skynet browser, skychat with the dmsg/skynet transport selector + activity log pane, host-content, console REPL, live logs). Captured headless via cmd/hvinspect against a live 'skywire cli hv serve' harness; includes the regeneration recipe. Images under docs/img/wasm-visor/.
The browser-tab visor is now a full federated group-chat member. group_js.go
starts a group.Manager over the wasm dmsg client with an in-memory store
(store_memory.go) + in-memory CXO tree (ManagerConfig.InMemoryDB) — no filesystem
needed. It publishes its own member feed and subscribes to peers' feeds over dmsg,
exactly like a native visor; local state is ephemeral (resets on tab reload; the
network retains what was published).
manager.go: plumb InMemoryDB through ManagerConfig → Manager → each opened
group.Config (relaxes the DataDir-required check when set). Native default false.
JS surface on skywireVisor: skychatGroupCreate(name[,mode]) → {id,name,invite},
skychatGroupJoin(inviteLink), skychatGroupSend(id,text), skychatGroupAddMember(
id,pk), skychatGroupList() (JSON), skychatGroupMessages([id]) (JSON). Inbound
group messages surface into an in-memory ring + the visor log.
Builds native + js/wasm; group tests + native full build + js vet + lint clean
(group_js.go lint-clean). Blob regenerated. Next: UI + wasm↔native demo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A browser (wasm) visor had no telemetry integration, so its transports never reached TPD
/metrics— invisible to rewards. This gives it the same CXO reporting as a native visor, but with an in-memory datastore (no filesystem in a tab; telemetry resets on reload, which is fine — TPD retains published rollups in its window).The blocker + fix
The CXO/skyobject stack transitively imported bbolt (via the in-repo
cxds/idxdbon-disk drives), which won't build forGOOS=js(arch consts + mmap) — so the CXO publisher couldn't run in wasm even withInMemoryDB.pkg/cxo/data/cxds+idxdb: bbolt on-diskdrive.go→//go:build !js;drive_js.gostubs provideDriveOptions+NewDrive*(which error — wasm uses the in-memory CXDS/IdxDB via skyobject'sInMemoryDB). Sharedpanicfmoved to the untaggedmemory.go. Native on-disk path unchanged (tests pass).skyobject.NewContainer: skip theDataDirmkdir whenInMemoryDB— the default DataDir resolves to/tmpunder js/wasm wheremkdiris "not implemented on js" (was fatal); in-memory mode never touches those files.The wasm telemetry publisher (
telemetry_js.go)treestore.Publisher(InMemoryDB: true), feed PK = visor PK.tpM.SetTPDLeafPublisher(pub)→transports/<uuid>/entry+/tombstone.transports/<uuid>/current(liveSnapshot: sent/recv + latency min/max/avg) each minute.bootEdgeafter the transport manager serves.Verified live
Reloaded the wasm visor → publisher comes up (
feed=<self> → TPD), and after ~2.5 min TPD/metricsshows the wasm visor's 5 transports (swtr,webrtc), each with latency (e.g. webrtc avg 11000, swtr avg 150000) and bandwidth (daily sent/recv) populated. Builds native + js/wasm; cxds/idxdb tests pass.🤖 Generated with Claude Code