feat(sdk): lazy per-topic subscription for DataSources (0.15.0)#141
Draft
Alvvalencia wants to merge 3 commits into
Draft
feat(sdk): lazy per-topic subscription for DataSources (0.15.0)#141Alvvalencia wants to merge 3 commits into
Alvvalencia wants to merge 3 commits into
Conversation
Additive only — every existing plugin keeps working with no recompile:
- PJ_DATA_SOURCE_CAPABILITY_LAZY_SUBSCRIPTION (1<<6) + kCapabilityLazySubscription.
- Runtime-host tail slot set_advertised_topics: declarative full-set topic
advertise with parser metadata (reuses PJ_parser_binding_request_t) so
advertised topics can appear in the host catalog with zero data.
DataSourceRuntimeHostView::setAdvertisedTopics returns a distinct
'host does not support lazy subscription' error for old-host fallback.
- New extension pj.topic_subscription.v1 (data_source_topic_subscription.h):
declarative set_active_topics, called on the poll thread strictly
serialized with poll/start/stop. DataSourcePluginBase wires it to the new
onActiveTopicsChanged() virtual iff the capability is declared;
DataSourceHandle::topicSubscriptionExtension()/setActiveTopics() host-side.
- MessageParser tail slot describe_schema_columns: flattened scalar-column
manifest (JSON [{path,type}], parse() emission order) so hosts can
pre-create catalog columns before the first sample. C++ ColumnSpec helper
+ base-class serialization; MessageParserHandle::describeSchemaColumns.
- ABI layout sentinels extended (runtime host 96->104, parser vtable 88->96,
extension struct pinned); tests: topic_subscription_extension_test,
describe_schema_columns_test. 50/50 green in debug+ASAN and release.
- Version 0.14.0 -> 0.15.0 (conanfile, CMakeLists, recipe.yaml), CHANGELOG,
docs (ARCHITECTURE, REQUIREMENTS, data-source-guide, message-parser-guide).
Note: local abi_check fails on clean main too (toolchain-vs-baseline noise);
CI's abi_check is the authoritative additions-only gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WUAom7BH7x39mRZFhBw79z
…break marker, DataSourceHandle subscription test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUAom7BH7x39mRZFhBw79z
setAdvertisedTopics duplicated ensureParserBinding's field-for-field struct conversion; both now go through toAbiParserBindingRequest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUAom7BH7x39mRZFhBw79z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Additive ABI so streaming sources can connect WITHOUT subscribing and let the host drive per-topic subscriptions from consumer demand.
Additions (MINOR, abidiff additions-only)
PJ_DATA_SOURCE_CAPABILITY_LAZY_SUBSCRIPTION(1<<6).set_advertised_topics: declarative full-set topic advertise with parser metadata (reusesPJ_parser_binding_request_t) so advertised topics appear in the host catalog with zero data. Old hosts yield a distinct error → plugins fall back to eager.pj.topic_subscription.v1(set_active_topics, poll-thread contract) wired automatically byDataSourcePluginBaseto the newonActiveTopicsChanged()virtual.describe_schema_columns(JSON column manifest) so hosts can pre-create catalog columns before the first sample.Versioning note
In-process C++ parser-path break marker (see CHANGELOG): the
describeSchemaColumnsvirtual changesMessageParserPluginBaselayout — parser plugins built ≤0.14 must be rebuilt (host calls parseScalars/parseObject on the C++ pointer). No public tag ever shipped 0.14/0.15; first public release of this line must be 1.0.0.Tests
51/51 (debug+ASAN), incl. new topic_subscription_extension_test, describe_schema_columns_test, data_source_handle_subscription_test; ABI layout sentinels extended. Local abidiff is toolchain-noisy (fails on clean main too) — CI is the gate.
Layered PRs: PJ4 host + pj-official-plugins follow (links in comments).
🤖 Generated with Claude Code
https://claude.ai/code/session_01WUAom7BH7x39mRZFhBw79z