feat(sdk): DataSource per-topic pause — advertise + pj.topic_subscription.v1#140
Merged
Conversation
…tion.v1
Add a backward-compatible (additive) ABI so a streaming source can expose
all its topics cheaply while only transmitting data for topics the host is
actually displaying.
- New capability PJ_DATA_SOURCE_CAPABILITY_PER_TOPIC_PAUSE (1<<6) +
kCapabilityPerTopicPause mirror.
- Plugin->host advertise: PJ_available_topic_t + notify_available_topics
runtime-host tail slot, wrapped by
DataSourceRuntimeHostView::notifyAvailableTopics (PJ_HAS_TAIL_SLOT-gated;
errors on an old host so a new plugin can fall back).
- Host->plugin control: get_plugin_extension("pj.topic_subscription.v1")
-> PJ_topic_subscription_v1_t::set_active_topics (declarative full set),
wrapped by DataSourceHandle::setActiveTopics (passes context() as ctx;
safe no-op when the extension is absent).
- Fix the stale "same thread as start()" runtime-host doc.
ABI is additions-only: MIN_VTABLE_SIZE (128), PROTOCOL_VERSION (4) and
PJ_ABI_VERSION (5) are unchanged; the runtime-host vtable deliberately
grows 96->104. Sentinels updated. No version-source bump: recorded under
CHANGELOG [Unreleased] (release number is a bundled, later decision).
Tests: notify_available_topics_test (advertise round-trip, old-host
degrade, extension round-trip, handle no-op) + updated helpers; 49/49
SDK tests pass. Docs + capability table updated; pj_base/CLAUDE.md
versioning gotcha reconciled with Release Versioning.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
REQUIREMENTS.md's DataSource capability table was missing kCapabilityPerTopicPause (data-source-guide.md had it, this table didn't). ARCHITECTURE.md had no mention of the new notify_available_topics tail slot or the pj.topic_subscription.v1 extension at all. Also fixes an unrelated pre-existing stale offset for push_message (documented as 96, actually 88 per the ABI sentinel test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DataSourcePluginBase::bind(), FileSourceBase/StreamSourceBase lifecycle methods, the DataSourceRuntimeHostView/ToolboxRuntimeHostView non-trivial methods, ToolboxPluginBase::bind(), and both families' ABI trampolines were inline in headers with no technical requirement to be (none are templates; pj_base already builds as a real STATIC lib that every plugin .so links privately, so these compile once into the archive instead of once per consumer TU). Declarations stay in the headers; bodies move to matching pj_base/src/*.cpp files. No behavior change — build clean, 49/49 tests pass, ABI layout sentinels unaffected. Templates (vtableWithCreate<CreateFn>, pushMessage<FetchMessageData>) and trivial one-line getters/forwards are left inline as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-topic-pause work on this branch is strictly additive (tail slot + plugin extension, both struct_size/PJ_HAS_TAIL_SLOT-gated), and the last publicly tagged release (v0.14.0) already carries PR #135's pj.data_processors.v1 break -- git history confirms ca052ce is an ancestor of v0.14.0. So the delta since the last tag is additive-only: a MINOR bump is correct per CLAUDE.md's versioning rules, not the 1.0.0 the stale CHANGELOG/conanfile.py comments implied. Bump all three canonical version sources (conanfile.py, CMakeLists.txt PJ_PACKAGE_VERSION, recipe.yaml). Also fixes CHANGELOG.md: the pj.data_processors.v1 entry was still sitting under [Unreleased] with a "must ship as 1.0.0" note, even though it already shipped in v0.14.0 -- moved it to a backfilled [0.14.0] section with the versioning note corrected to past tense. packaging/conan-center-index/** intentionally untouched: it pins the real published v0.14.0 tag + its sha256 for eventual Conan Center submission, not one of the three synced version sources. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Streaming sources today make topic selection a one-shot decision before
start(): subscribe to everything (bandwidth/CPU paid for topics nobody looks at) or a narrow set (stop/restart to explore anything else). There is no ABI for a source to advertise its topic universe cheaply and let the host drive which topics actually transmit.What this adds (strictly additive ABI)
New capability —
PJ_DATA_SOURCE_CAPABILITY_PER_TOPIC_PAUSE(1<<6) +kCapabilityPerTopicPausemirror. Everything below is gated on it; non-supporting sources are untouched.Plugin → host advertise —
PJ_available_topic_t(topic_name,parser_encoding,type_name,schema) + anotify_available_topicsruntime-host tail slot (vtable grows 96→104,PJ_HAS_TAIL_SLOT-gated), wrapped byDataSourceRuntimeHostView::notifyAvailableTopics. Contract: every call carries the FULL current set (declarative, no deltas), entered on the plugin's poll thread. On an old host the wrapper errors, so a new plugin can detect and fall back to its legacy subscribe-selected-at-start path.Host → plugin control — new extension
"pj.topic_subscription.v1"(PJ_TOPIC_SUBSCRIPTION_EXTENSION_V1):set_active_topics(ctx, names, count, err)— again a declarative full set, entered from the host GUI thread withctx = handle.context(). Wrapped byDataSourceHandle::setActiveTopics(safe no-op when the extension is absent). The intended plugin implementation is a mutex-protected latest-wins slot drained byonPoll— not a command queue.Compat discipline — additions only:
MIN_VTABLE_SIZE(128),PROTOCOL_VERSION(4),PJ_ABI_VERSION(5) unchanged; ABI layout sentinels updated. No version-source bump yet — recorded under CHANGELOG[Unreleased]; the release number is a bundled, later decision (queued behind #135).Verification
notify_available_topics_test: advertise round-trip, old-host graceful degrade, extension round-trip throughhandle.context(), handle no-op without the extension. 49/49 SDK tests pass..so+ mock host vtable): advertise → default-paused (0 msgs) →set_active_topics→ data flows → empty set → flow stops. Verified for both foxglove-bridge (mcap ws player) and ROS 2 (live talker via the production proxy).Consumers (review together)
CatalogModelpaused placeholders,TopicDemandTracker, demand wiring) — pins this branch (e566d96) as its submodule gitlink.🤖 Generated with Claude Code