feat(streams): Records API#2535
Conversation
Tests & local verificationUnit tests: Live smoke script (requires CDF credentials per CONTRIBUTING.md): poetry install -E all && poetry run python scripts/ila_streams_records_smoke.py --help
poetry run python scripts/ila_streams_records_smoke.py
# optional ingest (set ILA_STREAM_EXTERNAL_ID + ILA_RECORD_ITEM_JSON):
ILA_STREAM_EXTERNAL_ID=my-stream ILA_RECORD_ITEM_JSON='{"space":"...","externalId":"...","sources":[...]}' \
poetry run python scripts/ila_streams_records_smoke.py --ingest-oneCI note: This PR targets |
Add StreamsAPI (create, list, retrieve, delete) under client.streams, typed stream models in cognite.client.data_classes.streams, and unit tests. Register streams in non-retryable POST paths for create/delete. Records API will follow in a separate change. Made-with: Cursor
c0ac1ca to
b83f313
Compare
Made-with: Cursor
Add StreamsRecordsAPI at client.streams.records (ingest, upsert, delete, filter, aggregate, sync), typed record/sync/response models, POST retry rules for record write paths, CogniteClientMock wiring, and tests. Made-with: Cursor
- Add ingest_items, upsert_items, delete_items on StreamsRecordsAPI (wraps items array). - Add tests/tests_unit/test_api/test_streams_records.py with grouped tests for all record endpoints. - Add scripts/ila_streams_records_smoke.py for live verification (list streams; optional ingest). Made-with: Cursor
Made-with: Cursor
- Poll list_runs for scheduled trigger history (1-min cron) with skip if empty. - Retrieve workflow by field equality; lastUpdatedTime may advance between calls. - Use list_runs instead of deprecated get_trigger_run_history. Made-with: Cursor
- Require semaphore on BasicAsyncAPIClient._post for ingest/upsert/delete (write) and filter/aggregate/sync (read). - Remove stray codegen imports from SyncStreamsAPI; refresh sync records hash. - Simplify ingest_items unit test (avoid gzip-compressed request body decode). Made-with: Cursor
b589772 to
0de6c78
Compare
create_autospec cannot set nested ``records`` when the spec is built from StreamsAPI/ SyncStreamsAPI (``records`` is not visible to autospec the same way as SimulatorsAPI constructor kwargs). Bypass MagicMock.__setattr__ so CogniteClientMock keeps spec_set and nested records API mocks. Made-with: Cursor
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## pysdk-release-v8 #2535 +/- ##
====================================================
+ Coverage 93.41% 93.45% +0.03%
====================================================
Files 478 488 +10
Lines 48217 48695 +478
====================================================
+ Hits 45044 45510 +466
- Misses 3173 3185 +12
🚀 New features to boost your workflow:
|
… PR 2534 structure This refactors the Streams and Records API to be part of the data_modeling module rather than a separate top-level streams module, matching the structure of PR #2534. Changes: - Move StreamsAPI from _api/streams to _api/data_modeling/streams - Move StreamsRecordsAPI from _api/streams/records to _api/data_modeling/records - Move stream data classes to data_modeling/streams.py (includes record classes) - Integrate records API into StreamsAPI via self.records attribute - Remove direct self.streams attribute from AsyncCogniteClient - Streams now accessed via client.data_modeling.streams - Generate sync API wrappers for new locations - Update all imports and exports Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Remove parenthetical type references and descriptive asides from docstrings: - Removed "(human-readable)", "(\`\`Type\`\`)" style comments - Simplified endpoint descriptions to just describe the response Removed StreamTemplate.version field as it's not in the API specification. The field was optional and only stored, never used. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…PI move - Update testing.py to import StreamsAPI from new data_modeling location - Fix unused imports in auto-generated sync API init file - Update sync_api_template to remove unused imports - Fix testing.py to import from new data_modeling.records location Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Update test_streams.py and test_streams_records.py to import from cognite.client.data_classes.data_modeling.streams instead of old location - Update client API calls to use client.data_modeling.streams - Fix mypy type: ignore comment for nest_asyncio import Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Summary
Adds
StreamsRecordsAPIatclient.streams.records(ingest, upsert, delete, filter, aggregate, sync), record/sync/response data classes, POST retry rules for record write paths,CogniteClientMockwiring, and tests.Stack
andersfylling/cognite-sdk/streams-api) — merge that PR first, then rebase this branch ontomasterif needed or merge as-is after feat(streams): Streams API #2534 lands.Example: create and delete a stream (live project)
Streams are created and removed with
client.streams(see #2534). The API accepts one stream per create/delete request.End-to-end smoke (list → create → retrieve → delete, and optional records when
ILA_E2E_RECORD_JSONis set):Credentials and project access follow the same pattern as other integration tests in
CONTRIBUTING.md.Made with Cursor