Rewrite shadow system with KV-based persistence and derive macros#82
Open
MathiasKoch wants to merge 27 commits intofeature/asyncfrom
Open
Rewrite shadow system with KV-based persistence and derive macros#82MathiasKoch wants to merge 27 commits intofeature/asyncfrom
MathiasKoch wants to merge 27 commits intofeature/asyncfrom
Conversation
Refactor the derive macros to use more idiomatic Rust patterns while maintaining identical functionality. Key changes: - Replace builder/visitor pattern with declarative TypeTransformConfig - Add proper error handling with syn::Result and span information - Cleaner Option<T> detection using type path analysis - Separate concerns into dedicated modules: - attr/: Attribute parsing (#[shadow], #[shadow_patch], #[shadow_attr]) - types/: Type utilities (primitive detection, Option extraction) - codegen/: Code generation (type defs, ShadowPatch/ShadowState impls) - transform.rs: Declarative type transformations - Document magic constants (DEFAULT_TOPIC_PREFIX, DEFAULT_MAX_PAYLOAD_SIZE) - Add comprehensive unit tests for new modules The public API remains unchanged. All existing tests pass. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement the foundational KV storage layer for shadow persistence: - KVStore trait with async fetch/store/remove/remove_if methods - SequentialKVStore for embedded NOR flash (sequential-storage v7) - FileKVStore for std environments and testing - Migration types (MigrationSource, MigrationError, LoadResult) - CommitStats for tracking save operations - FNV-1a hash functions for schema versioning - Updated error types (KvError, EnumFieldError, ScanError) Note: SequentialKVStore stores MapStorage in mutex (deviation from plan due to sequential-storage v7 API). See planning/deviations.md. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Implement lightweight no_std JSON scanner for tag/content fields - Find byte ranges in single pass, order independent - Handle nested objects, arrays, and string escapes correctly - Support partial updates (tag or content may be absent) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ReportedUnionFields trait for flat union serialization - Add serialize_null_fields() helper for inactive variant fields - Add ShadowNode trait with: - Associated types: Delta, Reported - Constants: MAX_DEPTH, MAX_KEY_LEN, MAX_VALUE_LEN, SCHEMA_HASH - Core methods: apply_and_persist(), into_reported() - Key enumeration: keys(KeySet), migration_sources() - Custom defaults: apply_field_default() - Enum handling: enum_fields(), set/get_enum_variant(), is_field_active() - Add ShadowRoot trait extending ShadowNode with NAME constant - Clean up phase references from module comments These traits define the interface for KV-based shadow persistence, to be implemented by the derive macro in subsequent work. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…e 4) - Add NoPersist KVStore: zero-cost noop for non-persisted shadows - All operations return success - fetch() returns None, triggering first-boot behavior - Add KvShadow<'a, S, K> struct with borrowed KVStore reference (&'a K) - Enables multiple shadows to share the same KVStore - Named KvShadow to avoid conflict with existing MQTT Shadow - Add two constructors: - new_in_memory() -> KvShadow<'static, S, NoPersist> - new_persistent(&kv) -> KvShadow<'a, S, K> - Implement load() with three-way behavior: - First boot: initialize defaults, persist all fields, write hash - Normal boot: load fields from KV (hash matches) - Migration: load fields, set schema_changed flag (hash mismatch) - Add KeyTooLong variant to KvError - Add FileKVStore::temp() helper for test fixtures - Enable miniconf postcard feature for path-based serialization - Add ignored tests for Phase 8 derive macro integration Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…se 5) - Implement load_fields_with_migration() with full migration support: - Tries primary key first, falls back to migration sources - Applies type conversion functions when specified - Performs "soft migration": writes to new key, preserves old for rollback - Add try_migrations() helper for iterating migration sources - Add commit() method for finalizing schema changes: - Writes new schema hash after boot confirmed successful - Removes orphaned keys using remove_if() with FnvIndexSet lookup - Returns CommitStats with cleanup information - Add comprehensive Phase 5 test stubs (ignored until Phase 8 derive macros) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- load_fields() now reads _variant keys first to set up enum variants before loading field values - load_fields_with_migration() uses same two-phase approach - persist_all_fields() writes _variant keys as plain UTF-8, then fields - _variant keys stored as UTF-8 (not postcard) for debuggability - Inactive variant fields preserved in KV (not treated as orphans) - If no _variant key exists, enum uses its #[default] variant - Added Phase 6 tests (ignored until Phase 8 derive macros) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add apply_and_save() method on KvShadow as thin wrapper - Takes delta by reference (&S::Delta) so caller retains ownership - Delegates to ShadowNode::apply_and_persist() generated by derive macro - Only Some fields are written to KV - reduces flash wear - No Clone bound needed on Delta (reference passing) - Enables wait_delta() pattern to return delta to caller - Add Phase 7 tests (ignored until Phase 8 provides derive macro support) Note: The actual delta iteration and persistence logic is generated by the derive macro in Phase 8. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Generate ShadowNode trait implementations via #[shadow] and #[shadow_patch] macros: - Parse field attributes: migrate(from=), default, opaque, report_only, leaf - Generate apply_and_persist() with per-field codegen for structs and enums - Generate enum_fields(), set_enum_variant(), get_enum_variant() for enums - Generate MAX_VALUE_LEN, SCHEMA_HASH constants - Generate Delta and Reported types with proper serde attributes - Support adjacently-tagged enums with struct-shaped Delta - Generate ReportedUnionFields for flat union serialization Known limitations: - enum_fields() only generates for enum types, not for structs containing enums - This means nested struct -> enum loading fails (see ignored test) - Next: Implement load_from_kv()/persist_to_kv() with per-field recursive codegen to make each type self-contained and remove miniconf dependency Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace miniconf-based schema traversal with direct per-field code generation: - Add load_from_kv() and persist_to_kv() methods to ShadowNode trait - Add collect_valid_keys() for GC in commit() - Generate per-field loading/persistence code for structs and enums - Enums read _variant key and construct appropriate variant - Remove miniconf::Tree derives from all types - Remove KeyPath, path_to_key(), try_path_to_key() helpers - Simplify KvShadow methods to use new trait methods Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement ShadowNode and ShadowPatch for all primitive types using a new impl_opaque! macro, eliminating the need for is_primitive() checks in codegen entirely. Key changes: - Add src/shadows/opaque.rs with impl_opaque! macro implementing ShadowNode and ShadowPatch for primitives (bool, char, integers, floats) and generic containers (heapless::String, heapless::Vec) - Delete rustot_derive/src/types/primitive.rs and is_primitive() - Update codegen to use attrs.opaque instead of is_primitive() checks - Fields with migration attributes are still treated as leaf types to ensure migration logic runs at struct level - Adjacently-tagged enum serialization uses FIELD_NAMES.is_empty() runtime check (optimized away by compiler) for leaf detection The key insight: primitives now implement ShadowNode with Delta = Self, so <u32 as ShadowNode>::Delta = u32, allowing codegen to uniformly use <T as ShadowNode>::Delta for all fields. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Integrate MQTT-based AWS IoT Shadow communication directly into KvShadow, combining KV-based field-level persistence with cloud synchronization. Changes: - Add mqtt and subscription fields to KvShadow struct - Make state field private with state() accessor - Update constructors to require MQTT client reference: - new_in_memory(mqtt) - new_persistent(kv, mqtt) - Port MQTT helper methods from ShadowHandler - Add public cloud methods: - wait_delta() - subscribe, apply, persist, acknowledge - update(f) - report state changes to cloud - update_desired(f) - request state changes from cloud - sync_shadow() - fetch and sync from cloud - delete_shadow() - delete from cloud AND KV - Make apply_and_save() private (internal helper) - Add PREFIX and MAX_PAYLOAD_SIZE constants to ShadowRoot trait - Add KvShadowKvOnly test struct for KV-only testing without MQTT Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove the legacy shadow system that was superseded by the KV-based ShadowNode/ShadowRoot system. This removes: - ShadowPatch, ShadowState traits - Shadow, PersistedShadow, ShadowHandler structs - ShadowDAO trait - #[shadow] and #[shadow_patch] proc macros - Related codegen (shadow_patch_impl, shadow_state_impl, type_def) - ShadowPatch implementations from impl_opaque! macro - alloc_impl.rs (std ShadowPatch impls) - Legacy test files - Unused derive crate modules (transform.rs, types/) The modern KV-based system is preserved: - ShadowNode, ShadowRoot traits - KvShadow - #[shadow_node] and #[shadow_root] macros - impl_opaque! macro (now generates ShadowNode + ReportedUnionFields only) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Part 1: Split rustot_derive/src/codegen/shadow_node_impl.rs (2,302 lines) - codegen/mod.rs: Entry point, rustot_crate_path(), ShadowNodeConfig - codegen/struct_codegen.rs: generate_struct_code() - codegen/enum_codegen.rs: generate_enum_code(), generate_simple_enum_code() - codegen/adjacently_tagged.rs: generate_adjacently_tagged_enum_code() Part 2: Split src/shadows/kv_shadow.rs (2,563 lines) + rename - shadow/mod.rs: Shadow struct and persistence methods - shadow/cloud.rs: MQTT cloud communication methods - shadow/tests.rs: All tests (~1,400 lines) Part 3: Rename KvShadow to Shadow across codebase - Public API: KvShadow -> Shadow - Test helper: KvShadowKvOnly -> ShadowKvOnly Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add two new optional parameters to the shadow_root attribute macro: - topic_prefix: sets ShadowRoot::PREFIX constant - max_payload_len: sets ShadowRoot::MAX_PAYLOAD_SIZE constant When not specified, the constants are not generated, allowing trait defaults to apply. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The scan module was planned for handling adjacently-tagged enums where the content field might appear before the tag field in JSON. However, the actual implementation uses struct-shaped Delta types instead, which is simpler and more idiomatic. The scan module was never integrated. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Refactor the shadow persistence system to separate core functionality from KV-specific persistence: - ShadowNode trait now contains only core methods: apply_delta, into_reported, and SCHEMA_HASH constant - New KVPersist trait (feature-gated with shadows_kv_persist) handles field-level KV persistence operations - New StateStore trait provides state-level operations for Shadow struct - Rename kv_store module to store - InMemory now implements StateStore directly without serialization - Update derive macros to generate split trait implementations - Feature-gate KVPersist impl with #[cfg(feature = "shadows_kv_persist")] This enables memory-efficient operation where Shadow doesn't hold state internally, and supports multiple storage strategies (in-memory, blob, field-level KV). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Enable tests that verify enum variant handling in KV persistence: - test_commit_preserves_inactive_enum_variant_fields - test_enum_apply_and_save_writes_variant_key_as_utf8 - test_enum_variant_switch_preserves_inactive_fields - test_enum_variant_switch_and_back_restores_values_on_reload - test_apply_and_save_enum_variant_change Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Apply semantic improvements from upstream feature/async commits: - Use from_slice_escaped for all JSON deserialization to handle escaped characters correctly - Wrap handle_delta() in a retry loop for clean session recovery instead of returning an error - Increase topic format buffer from <64> to <65> for accepted/rejected topic subscriptions - Add Debug, Clone derives to DeltaState for broader usability Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use heapless-09 feature flag for sequential-storage to match the crate's heapless 0.9 dependency, and import FnvIndexSet from the correct module path. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds tests/shadows.rs exercising the Shadow API against live AWS IoT: wait_delta, update, sync_shadow, delete_shadow with FileKVStore persistence and a report_only field. Also fixes get_shadow_from_cloud to return the created shadow state on 404 instead of discarding it, and adds FileKVStore::base_path() accessor for test assertions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Each KVPersist impl now allocates its own serialization buffer internally: no-std uses stack arrays sized by MAX_VALUE_LEN, std uses postcard::to_allocvec() and kv.fetch_to_vec(). This fixes the MaxSize derive bug where String/Vec fields failed to compile, removes the unused MAX_DEPTH constant, and eliminates the hardcoded [0u8; 512] buffers in store implementations. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…try KV persistence Restructure src/shadows/opaque.rs into src/shadows/impls/ submodule with separate files for opaque primitives, heapless containers, and std containers. Add ShadowNode + KVPersist for heapless::LinearMap<K, V, N> and std::collections::HashMap<K, V> with per-entry Patch::Set/Patch::Unset deltas and manifest-based KV persistence (__keys__ stores active keys via postcard). Introduce MapKey trait for map key types, collect_valid_prefixes() on KVPersist for GC support of dynamic collections, and update commit() in both store implementations to preserve prefix-matching keys. Propagate collect_valid_prefixes in all three derive macro codegen files. Add opaque ShadowNode impl for core::time::Duration. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the monolithic
PersistentShadow/ShadowPatch/ShadowStatearchitecture with a trait-based KV-persistence system. Shadows are now defined with#[shadow_root]and#[shadow_node]derive macros that generate field-level serialization, delta types, and persistence code — eliminating theminiconfdependency.The new architecture separates concerns into composable layers:
ShadowNode(delta/reported generation),KVPersist(field-level storage),StateStore(state management abstraction), andShadow(MQTT cloud connectivity). This enables features that were impossible with the old system: OTA-safe schema migrations, adjacently-tagged enum support, map-type collections, and report-only fields.Architecture
Core Traits (
src/shadows/mod.rs)ShadowNode— DefinesDeltaandReportedassociated types,parse_delta(),apply_delta(), compile-timeSCHEMA_HASHKVPersist— Field-level load/persist/collect with self-managed buffer types (ValueBuf,MaxKeyLEN)ShadowRoot— Top-level shadow metadata: name, MQTT topic prefix, max payload sizeMapKey— Key trait for map-based collections with compile-time display lengthVariantResolver— Resolves current variant for adjacently-tagged enum deltas missing the tag fieldStateStore Abstraction (
src/shadows/store/)InMemory<S>— Volatile, Mutex-wrapped state (testing)SequentialKVStore— Flash-based field-level persistence viasequential-storage(feature-gated)FileKVStore— File-based persistence forstdenvironmentsDerive Macro System (
rustot_derive/src/codegen/)Delta{Name}(all fieldsOption<T>) andReported{Name}(withskip_serializing_if) structs_variantkey loading), and adjacently-tagged enums (flat union reporting)#[shadow_attr(opaque)],#[shadow_attr(report_only)],#[shadow_attr(migrate(from = "old_key"))],#[shadow_attr(default = value)]impl_opaque!macro implementsShadowNode+KVPersistfor primitive typesOTA-Safe Migrations
SCHEMA_HASHenables compile-time detection of schema changesload_from_kv_with_migration()handles renamed fields with optional value conversioncommit()finalizes schema hash and garbage-collects orphaned keys after successful bootChangelog
PersistentShadow,ShadowPatch,ShadowStatewithShadowNode/KVPersisttrait systemminiconfdependency; addsequential-storage,postcard,darling#[shadow_root]and#[shadow_node]derive macros with per-field KV codegenStateStoreabstraction withInMemory,SequentialKVStore, andFileKVStoreimplementationsimpl_opaque!macro for primitiveShadowNodeimplementationsShadowNodeimplementations withPatch<T>semantics and per-entry KV persistenceSCHEMA_HASH,LoadResult, andcommit()GCShadowstruct for combined KV persistence + MQTT cloud connectivityrustot_derivemacro architecture with Darling-based attribute parsingFixes #47