Skip to content

feat(streams): Records API#2535

Draft
andersfylling wants to merge 12 commits intopysdk-release-v8from
andersfylling/cognite-sdk/streams-records-api
Draft

feat(streams): Records API#2535
andersfylling wants to merge 12 commits intopysdk-release-v8from
andersfylling/cognite-sdk/streams-records-api

Conversation

@andersfylling
Copy link
Copy Markdown
Contributor

@andersfylling andersfylling commented Mar 25, 2026

Summary

Adds StreamsRecordsAPI at client.streams.records (ingest, upsert, delete, filter, aggregate, sync), record/sync/response data classes, POST retry rules for record write paths, CogniteClientMock wiring, and tests.

Stack

Example: create and delete a stream (live project)

Streams are created and removed with client.streams (see #2534). The API accepts one stream per create/delete request.

import uuid

from cognite.client import CogniteClient
from cognite.client.data_classes.streams import StreamDeleteItem, StreamWrite

# Build a client (e.g. client credentials against your CDF cluster)
client = CogniteClient.default_oauth_client_credentials(
    project="your-project",
    cdf_cluster="api",  # or greenfield, etc.
    tenant_id="your-azure-tenant-id",
    client_id="your-app-id",
    client_secret="your-secret",
    client_name="my-app-streams-example",
)

# Create (template name must exist in your tenant; rate limits apply)
external_id = f"example_stream_{uuid.uuid4().hex[:12]}"
client.streams.create(
    [StreamWrite(external_id, {"template": {"name": "ImmutableTestStream"}})]
)

# Use the stream (e.g. client.streams.records.ingest_items(...)), then soft-delete when done
client.streams.delete([StreamDeleteItem(external_id)])

End-to-end smoke (list → create → retrieve → delete, and optional records when ILA_E2E_RECORD_JSON is set):

uv run python scripts/ila_streams_e2e_flow.py

Credentials and project access follow the same pattern as other integration tests in CONTRIBUTING.md.

Made with Cursor

@andersfylling
Copy link
Copy Markdown
Contributor Author

Tests & local verification

Unit tests: tests/tests_unit/test_api/test_streams_records.py covers all record endpoints (ingest, upsert, delete, filter, aggregate, sync) plus ingest_items / upsert_items / delete_items.

Live smoke script (requires CDF credentials per CONTRIBUTING.md):

poetry install -E all && poetry run python scripts/ila_streams_records_smoke.py --help
poetry run python scripts/ila_streams_records_smoke.py
# optional ingest (set ILA_STREAM_EXTERNAL_ID + ILA_RECORD_ITEM_JSON):
ILA_STREAM_EXTERNAL_ID=my-stream ILA_RECORD_ITEM_JSON='{"space":"...","externalId":"...","sources":[...]}' \
  poetry run python scripts/ila_streams_records_smoke.py --ingest-one

CI note: This PR targets andersfylling/cognite-sdk/streams-api, not master, so the repo\u2019s build workflow (pull_request \u2192 master) does not run on this PR. After #2534 merges, rebase this branch onto master and retarget the PR (or open a new PR) to get test_core / test_full on GitHub Actions.

Add StreamsAPI (create, list, retrieve, delete) under client.streams,
typed stream models in cognite.client.data_classes.streams, and unit tests.
Register streams in non-retryable POST paths for create/delete.

Records API will follow in a separate change.

Made-with: Cursor
@andersfylling andersfylling force-pushed the andersfylling/cognite-sdk/streams-api branch from c0ac1ca to b83f313 Compare March 26, 2026 15:15
Add StreamsRecordsAPI at client.streams.records (ingest, upsert, delete,
filter, aggregate, sync), typed record/sync/response models, POST retry rules
for record write paths, CogniteClientMock wiring, and tests.

Made-with: Cursor
- Add ingest_items, upsert_items, delete_items on StreamsRecordsAPI (wraps items array).
- Add tests/tests_unit/test_api/test_streams_records.py with grouped tests for all record endpoints.
- Add scripts/ila_streams_records_smoke.py for live verification (list streams; optional ingest).

Made-with: Cursor
- Poll list_runs for scheduled trigger history (1-min cron) with skip if empty.
- Retrieve workflow by field equality; lastUpdatedTime may advance between calls.
- Use list_runs instead of deprecated get_trigger_run_history.

Made-with: Cursor
- Require semaphore on BasicAsyncAPIClient._post for ingest/upsert/delete (write) and filter/aggregate/sync (read).
- Remove stray codegen imports from SyncStreamsAPI; refresh sync records hash.
- Simplify ingest_items unit test (avoid gzip-compressed request body decode).

Made-with: Cursor
@andersfylling andersfylling force-pushed the andersfylling/cognite-sdk/streams-records-api branch from b589772 to 0de6c78 Compare March 26, 2026 20:49
@andersfylling andersfylling changed the base branch from andersfylling/cognite-sdk/streams-api to pysdk-release-v8 March 26, 2026 20:49
create_autospec cannot set nested ``records`` when the spec is built from
StreamsAPI/ SyncStreamsAPI (``records`` is not visible to autospec the same
way as SimulatorsAPI constructor kwargs). Bypass MagicMock.__setattr__ so
CogniteClientMock keeps spec_set and nested records API mocks.

Made-with: Cursor
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 26, 2026

Codecov Report

❌ Patch coverage is 97.13701% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.45%. Comparing base (ffcc432) to head (9405be5).
⚠️ Report is 12 commits behind head on pysdk-release-v8.

Files with missing lines Patch % Lines
tests/tests_integration/test_api/test_workflows.py 61.11% 7 Missing ⚠️
cognite/client/_api/streams/__init__.py 87.17% 5 Missing ⚠️
cognite/client/_api/streams/records.py 97.61% 1 Missing ⚠️
...gnite/client/data_classes/streams/stream_record.py 98.96% 1 Missing ⚠️
Additional details and impacted files
@@                 Coverage Diff                  @@
##           pysdk-release-v8    #2535      +/-   ##
====================================================
+ Coverage             93.41%   93.45%   +0.03%     
====================================================
  Files                   478      488      +10     
  Lines                 48217    48695     +478     
====================================================
+ Hits                  45044    45510     +466     
- Misses                 3173     3185      +12     
Files with missing lines Coverage Δ
cognite/client/_cognite_client.py 92.72% <100.00%> (+0.13%) ⬆️
cognite/client/_sync_api/streams/__init__.py 100.00% <100.00%> (ø)
cognite/client/_sync_api/streams/records.py 100.00% <100.00%> (ø)
cognite/client/_sync_cognite_client.py 91.42% <100.00%> (+0.16%) ⬆️
cognite/client/data_classes/__init__.py 100.00% <100.00%> (ø)
cognite/client/data_classes/streams/__init__.py 100.00% <100.00%> (ø)
cognite/client/data_classes/streams/stream.py 100.00% <100.00%> (ø)
cognite/client/testing.py 100.00% <100.00%> (ø)
cognite/client/utils/_url.py 100.00% <ø> (ø)
tests/tests_unit/test_api/test_streams.py 100.00% <100.00%> (ø)
... and 7 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andersfylling andersfylling changed the title feat(streams): ILA Records API feat(streams): Records API Apr 14, 2026
andersfylling and others added 4 commits April 20, 2026 11:08
… PR 2534 structure

This refactors the Streams and Records API to be part of the data_modeling module
rather than a separate top-level streams module, matching the structure of PR #2534.

Changes:
- Move StreamsAPI from _api/streams to _api/data_modeling/streams
- Move StreamsRecordsAPI from _api/streams/records to _api/data_modeling/records
- Move stream data classes to data_modeling/streams.py (includes record classes)
- Integrate records API into StreamsAPI via self.records attribute
- Remove direct self.streams attribute from AsyncCogniteClient
- Streams now accessed via client.data_modeling.streams
- Generate sync API wrappers for new locations
- Update all imports and exports

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Remove parenthetical type references and descriptive asides from docstrings:
- Removed "(human-readable)", "(\`\`Type\`\`)" style comments
- Simplified endpoint descriptions to just describe the response

Removed StreamTemplate.version field as it's not in the API specification.
The field was optional and only stored, never used.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…PI move

- Update testing.py to import StreamsAPI from new data_modeling location
- Fix unused imports in auto-generated sync API init file
- Update sync_api_template to remove unused imports
- Fix testing.py to import from new data_modeling.records location

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Update test_streams.py and test_streams_records.py to import from
  cognite.client.data_classes.data_modeling.streams instead of old location
- Update client API calls to use client.data_modeling.streams
- Fix mypy type: ignore comment for nest_asyncio import

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant