Skip to content

feat(streams): Streams API#2534

Open
andersfylling wants to merge 15 commits intomasterfrom
andersfylling/cognite-sdk/streams-api
Open

feat(streams): Streams API#2534
andersfylling wants to merge 15 commits intomasterfrom
andersfylling/cognite-sdk/streams-api

Conversation

@andersfylling
Copy link
Copy Markdown
Contributor

@andersfylling andersfylling commented Mar 25, 2026

Summary

Add streams api to sdk.

Example

from cognite.client import CogniteClient
from cognite.client.data_classes.data_modeling.streams import StreamDeleteItem, StreamWrite

client = CogniteClient(...)  # your usual ClientConfig / credentials

external_id = "my_stream"
client.data_modeling.streams.create([StreamWrite(external_id, {"template": {"name": "ImmutableTestStream"}})])
client.data_modeling.streams.list()
client.data_modeling.streams.retrieve(external_id)
client.data_modeling.streams.delete([StreamDeleteItem(external_id)])

This PR does not add the Records API (#2535).

closes #2519
closes #2246

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 26, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.80%. Comparing base (c9cc644) to head (ec2959d).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2534      +/-   ##
==========================================
- Coverage   92.82%   92.80%   -0.03%     
==========================================
  Files         480      485       +5     
  Lines       48502    48763     +261     
==========================================
+ Hits        45021    45253     +232     
- Misses       3481     3510      +29     
Files with missing lines Coverage Δ
cognite/client/_api/data_modeling/__init__.py 100.00% <100.00%> (ø)
cognite/client/_api/data_modeling/streams.py 100.00% <100.00%> (ø)
cognite/client/_sync_api/data_modeling/__init__.py 100.00% <100.00%> (ø)
cognite/client/_sync_api/data_modeling/streams.py 100.00% <100.00%> (ø)
...nite/client/data_classes/data_modeling/__init__.py 100.00% <100.00%> (ø)
...gnite/client/data_classes/data_modeling/streams.py 100.00% <100.00%> (ø)
cognite/client/testing.py 100.00% <100.00%> (ø)
cognite/client/utils/_url.py 100.00% <ø> (ø)
...s_unit/test_api/test_data_modeling/test_streams.py 100.00% <100.00%> (ø)
tests/tests_unit/test_api_client.py 99.71% <ø> (ø)
... and 1 more

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andersfylling andersfylling marked this pull request as ready for review March 26, 2026 14:56
@andersfylling andersfylling requested review from a team as code owners March 26, 2026 14:57
@gemini-code-assist

This comment was marked as outdated.

gemini-code-assist[bot]

This comment was marked as outdated.

@andersfylling andersfylling changed the base branch from master to pysdk-release-v8 March 26, 2026 15:15
@andersfylling andersfylling force-pushed the andersfylling/cognite-sdk/streams-api branch from c0ac1ca to b83f313 Compare March 26, 2026 15:15
@andersfylling

This comment was marked as resolved.

gemini-code-assist[bot]

This comment was marked as resolved.

Copy link
Copy Markdown
Contributor

@haakonvt haakonvt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part 1 of review: cognite/client/_api/streams/__init__.py

Comment thread cognite/client/_api/streams/__init__.py Outdated
Comment thread cognite/client/_api/streams/__init__.py Outdated
Comment thread cognite/client/_api/streams/__init__.py Outdated
Comment thread cognite/client/_api/streams/__init__.py Outdated
Comment thread cognite/client/_api/streams/__init__.py Outdated
Comment thread cognite/client/_api/streams/__init__.py Outdated
Comment thread cognite/client/_api/streams/__init__.py Outdated
Comment thread cognite/client/_api/streams/__init__.py Outdated
Comment thread cognite/client/_api/streams/__init__.py Outdated
Comment thread cognite/client/_api/streams/__init__.py Outdated
@andersfylling andersfylling changed the base branch from pysdk-release-v8 to master March 30, 2026 12:52
@andersfylling andersfylling force-pushed the andersfylling/cognite-sdk/streams-api branch from 1902069 to 02d53d4 Compare March 30, 2026 13:34
@evertoncolling

This comment was marked as resolved.

@andersfylling andersfylling changed the title feat(streams): ILA Streams API feat(streams): Streams API Apr 14, 2026
@andersfylling andersfylling force-pushed the andersfylling/cognite-sdk/streams-api branch from 29a0184 to 6829245 Compare April 14, 2026 11:28
@andersfylling andersfylling force-pushed the andersfylling/cognite-sdk/streams-api branch from c2d5bad to 3225b62 Compare April 14, 2026 13:32
andersfylling and others added 9 commits April 14, 2026 15:44
The StreamsAPI needs to chunk items one at a time for create and delete
operations, similar to other APIs like AgentsAPI. This ensures each stream
is processed in a separate API request rather than being bundled together.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Remove unused imports that were causing lint failures (F401 errors):
- asyncio, Coroutine, Iterator (not used)
- Any, Literal, overload, TYPE_CHECKING (not used)
- APIClient (not used)
- SyncIterator, run_sync (not used)
- _get_event_loop_executor (not used)
- pandas, ClientConfig (not used)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
andersfylling added a commit that referenced this pull request Apr 20, 2026
… PR 2534 structure

This refactors the Streams and Records API to be part of the data_modeling module
rather than a separate top-level streams module, matching the structure of PR #2534.

Changes:
- Move StreamsAPI from _api/streams to _api/data_modeling/streams
- Move StreamsRecordsAPI from _api/streams/records to _api/data_modeling/records
- Move stream data classes to data_modeling/streams.py (includes record classes)
- Integrate records API into StreamsAPI via self.records attribute
- Remove direct self.streams attribute from AsyncCogniteClient
- Streams now accessed via client.data_modeling.streams
- Generate sync API wrappers for new locations
- Update all imports and exports

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
andersfylling and others added 2 commits April 23, 2026 10:08
Remove parenthetical type references and descriptive asides from docstrings:
- Removed "(human-readable)", "(\`\`Type\`\`)" style comments
- Simplified endpoint descriptions to just describe the response

Removed StreamTemplate.version field as it's not in the API specification.
The field was optional and only stored, never used.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
StreamTemplate.version was removed as it's not in the API spec.
Updated test to remove assertions checking for this field.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@polomani polomani self-assigned this Apr 23, 2026
Comment thread cognite/client/_api/data_modeling/streams.py
def __init__(self, config: ClientConfig, api_version: str | None, cognite_client: AsyncCogniteClient) -> None:
super().__init__(config, api_version, cognite_client)
self._CREATE_LIMIT = 1
self._DELETE_LIMIT = 1
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps it's better to add some warning first in case the SDK implementation changes?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self._warning = FeaturePreviewWarning(
            api_maturity="General Availability",
            sdk_maturity="alpha",
            feature_name="Streams",
        )

Comment thread cognite/client/_api/data_modeling/streams.py
Comment thread cognite/client/_api/data_modeling/streams.py
Comment thread cognite/client/data_classes/data_modeling/streams.py
Comment thread cognite/client/data_classes/data_modeling/streams.py
Comment thread cognite/client/data_classes/data_modeling/streams.py
Addressed all review comments from PR #2534:
- Using Sequence[StreamWrite] for type safety (no untyped dicts)
- Chunking handled automatically via _create_multiple and _delete_multiple
- Chunking limits set (_CREATE_LIMIT=1, _DELETE_LIMIT=1)
- Clean docstrings without unnecessary internal details
- Using params dict for query parameters
- Clear documentation on soft delete behavior
- Proper Note about paging limits in list()
- Statistics cost documentation in retrieve()

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Comment thread cognite/client/_sync_api/data_modeling/streams.py
Comment thread cognite/client/data_classes/data_modeling/streams.py
@polomani
Copy link
Copy Markdown
Contributor

@andersfylling is this PR dead?

@haakonvt
Copy link
Copy Markdown
Contributor

@andersfylling is this PR dead?

He is probably out of Claude tokens 😆

Comment on lines +63 to +64
res = await self._get(url_path=self._RESOURCE_PATH, semaphore=self._get_semaphore("read"))
return StreamList._load(res.json()["items"])
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use self._list here to avoid having to manually supply the semaphore + load data class

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't support limit, and we don't have cursor implementation there.

Comment thread cognite/client/_api/data_modeling/streams.py
Comment thread cognite/client/data_classes/data_modeling/streams.py
Comment thread cognite/client/data_classes/data_modeling/streams.py
"""Returns a write version."""
return StreamWrite(
external_id=self.external_id,
settings=StreamTemplateWriteSettings(template=StreamTemplate(name=self.created_from_template)),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using createdFromTemplate seems to not be too reliable according to the docs:

Name of the template used for creating this stream. Note: This value is for information only. The template might have been modified or even entirely deleted after the stream has been created.

Comment thread cognite/client/data_classes/data_modeling/streams.py
Comment thread cognite/client/data_classes/data_modeling/streams.py
class StreamTemplateWriteSettings(CogniteResource):
"""Write-side settings for creating a stream from a template."""

def __init__(self, template: StreamTemplate) -> None:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a FeaturePreviewWarning here since the docs says:

Note: Stream Templates are in Beta

Optionally - or maybe better - lets add a "SDK in alpha" warning to the StreamsAPI itself to give ourselves some freedom in changing/modifying the SDK API design for some time.

Comment thread cognite/client/data_classes/data_modeling/streams.py
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please split mock responses into fixtures in order to keep more "condense test logic" in the actual tests

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think most of these tests are already covered by our automatic testing, at least load/dump roundtrips of varying sorts, meaning most if not all of these tests can be removed

Comment thread cognite/client/_api/data_modeling/streams.py
Comment thread cognite/client/_api/data_modeling/streams.py
Comment thread cognite/client/_api/data_modeling/streams.py
Comment thread cognite/client/_api/data_modeling/streams.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants