Real-Time Data Sharing outline

content/en/docs/marketplace/platform-supported-content/services/data-sharing/_index.md

@@ -0,0 +1,75 @@
+---
+title: "Real-Time Data Sharing"
+url: /appstore/services/data-sharing/
+description: "Describes the Real-Time Data Sharing capabilities in Mendix, which enable efficient data integration through change data capture."
+aliases:
+---
+
+## Introduction
+
+Traditional data integration methods often result in delays, inefficiencies, and increased complexity, requiring developers to build custom solutions that handle transaction consistency, error recovery, and schema evolution. Real-Time Data Sharing provides a platform-level solution for data integration through change data capture. Instead of implementing custom replication logic, you specify which entities and attributes to publish, and the runtime handles change detection, streaming, and delivery.
+
+{{% alert color="info" %}}
+Real-Time Data Sharing is currently in beta and supported in Studio Pro 11.12 and above. 
+{{% /alert %}}
+
+## Key Capabilities
+
+* **Data Stream Definition** – Applications define which entities and attributes are published as data streams. Sensitive attributes or irrelevant records can be excluded.
+* **Automatic Topic Creation** – On deployment, Kafka topics are created in the Mendix Event Broker. Entity changes (inserts, updates, deletes) are automatically streamed to these topics.
+* **Contract Generation** – AsyncAPI 3.0 contracts are generated for each data stream and registered in the Mendix Catalog for discovery.
+* **Stream Consumption** – Applications consume data streams by importing AsyncAPI contracts. Received data is stored in the application database and can be transformed using view entities.
+* **External System Integration** – Event Broker bridges publish data streams to external systems such as S3 (Parquet files), Azure Blob Storage, or SQS queues.
+
+## Prerequisites
+
+To use Real-Time Data Sharing, you will need the following:
+
+* Studio Pro 11.12 and above
+* An event broker; this can be a licensed [Mendix Event Broker](/appstore/services/event-broker/) for apps running in Mendix Cloud
+
+## How It Works
+
+Real-Time Data Sharing uses Change Data Capture (CDC) to detect and publish entity changes from Mendix applications.
+
+Change Data Capture is a pattern that detects database changes and makes them available for downstream processing. Changes are tracked at the data storage layer and published to a message broker or other sink. Downstream systems can be databases, search indexes, caches, analytics platforms, or other consumers.
+
+The Mendix implementation captures domain model entity changes (inserts, updates, deletes) and their associations, then publishes them as change event streams to Kafka topics.
+
+## Architecture Overview
+
+Real-Time Data Sharing is built on Apache Kafka and uses the following components:
+
+* **Change Data Capture (CDC)** – The runtime detects entity changes at the data storage layer
+* **Change Event Streams** – Changes are organized into streams, where each stream contains a sequence of change events
+* **Event Broker Integration** – Streams are published to Kafka-compatible topics in the Mendix Event Broker
+* **Standardized Contracts** – AsyncAPI 3.0 contracts define stream schemas and metadata
+
+### Change Event Streams
+
+A change event stream is a sequence of change events for a specific entity type. Each change event contains:
+
+* The operation type (insert, update, or delete)
+* The entity state before the change 
+* The entity state after the change 
+
+Change events are published to Kafka topics and consumed by downstream systems.
+
+## Known Limitations
+
+Real-Time Data Sharing is currently in beta and has the following known limitations:
+
+* **Multi-Node Deployments** 
+* **Snapshot Behavior** – Delete events are not included in initial snapshots. Snapshots are only generated for major version changes to the data stream source, not minor version changes.
+* **Entity Renaming** – Removing an entity from a data stream and adding it back with a different exposed name in a new version will not produce a snapshot, as the same topic name is reused.
+* **Re-Enabling CDC** – If you disable a CDC document and later re-enable it, changes made while it was disabled are not captured or published. Use the force send snapshot feature to resynchronize data after re-enabling.
+* **Version Migration with Pending Events** – Migrating to a new data stream version while events remain in the outbox may result in schema mismatches, as the events will be published to the new topic with the old schema.
+* **Uncommitted Objects** – Creating an object and canceling the transaction (without committing) may produce a delete event. 
+* **Inherited Properties** – Inherited properties are not supported.
+
+## Read More
+
+Read more to learn how to configure and deploy Real-Time Data Sharing in the following documents:
+
+* [Real-Time Data Sharing Configuration](/appstore/services/data-sharing-configuration/)
+* [Deploy Real-Time Data Sharing](/appstore/services/data-sharing-deployment/)

content/en/docs/marketplace/platform-supported-content/services/data-sharing/data-sharing-configuration.md

@@ -0,0 +1,76 @@
+---
+title: "Real-Time Data Sharing Configuration"
+url: /appstore/services/data-sharing-configuration/
+description: "Describes how to configure Real-Time Data Sharing to publish data streams from your Mendix application."
+---
+
+## Introduction
+
+This document describes how to configure Real-Time Data Sharing in your Mendix application to publish entity changes as data streams.
+
+## Modeling 
+
+### Modeling the Change Event Stream
+
+Studio Pro provides interfaces to model which entities and associations are published as change event streams.
+
+Configuration options include:
+
+* Field selection – specify which attributes are published
+* Reference handling – publish denormalized data (embedded) or normalized data (references only)
+* Partition keys – configure keys to divide streams into partitions for parallel processing
+
+Ordering is guaranteed within a partition. Events with different partition keys may be processed in parallel.
+
+### Contract Definition
+
+Each data stream has an associated contract that defines:
+
+* Stream metadata (name, version, description)
+* Schema definition (field names, types, constraints)
+
+#### Exporting the Contract
+
+Contracts are exported as AsyncAPI 3.0 documents with the following mappings:
+
+* Streams → AsyncAPI channels
+* Change events → AsyncAPI messages
+
+## Event Broker Configuration Settings
+
+Event Broker and Kafka-compatible systems provide multiple configuration options beyond basic constants. Event Broker configuration is available in app settings, similar to database or workflow configuration.
+
+### Connection
+
+Configure the following connection settings:
+
+* **Server URL** – 
+* **Authentication** – Username and password
+* **Authentication Type** – Mendix Cloud, SASL, AWS, Confluent, or other compatible authentication mechanisms
+
+### Publishing
+
+Configure publishing behavior:
+
+* **Timeouts** – Timeout values for publishing operations
+* **Queue Size** – Maximum queue size for pending events in the outbox
+
+### Topic Naming
+
+Configure how streams map to Kafka topics:
+
+* Topic prefix – optional prefix for all topics
+* Topic pattern – custom naming pattern for topics
+
+### Consumer Groups
+
+Configure consumer group IDs for each topic. 
+
+## Metamodel
+
+The current metamodel has limited support for expressing streams as a concept. Future versions may introduce native stream representations rather than adapting existing metamodel components.
+
+## Read More
+
+* [Deploy Real-Time Data Sharing](/appstore/services/data-sharing-deployment/)
+* [Mendix Event Broker](/appstore/services/event-broker/)

content/en/docs/marketplace/platform-supported-content/services/data-sharing/data-sharing-deployment.md

@@ -0,0 +1,90 @@
+---
+title: "Deploy Real-Time Data Sharing"
+url: /appstore/services/data-sharing-deployment/
+description: "Describes how to deploy Real-Time Data Sharing and publish data to data lakes and other systems."
+---
+
+## Introduction
+
+This document describes deployment options for Real-Time Data Sharing, including CDC implementation approaches, runtime architecture, and integration with external systems.
+
+## CDC Implementation
+
+Two approaches are available for implementing change data capture in Mendix applications.
+
+### Using Runtime Data Storage Triggers
+
+The runtime data storage layer detects entity changes and publishes them directly. This approach:
+
+* Works consistently across all supported databases
+* Avoids deployment complexity by integrating CDC into the existing runtime
+* Operates at the domain model level rather than the database level
+* Supports all domain model features including view entities
+
+### Using an Independent CDC Component
+
+An external CDC component (such as Debezium) reads database changes from the Write Ahead Log (WAL) and publishes them to Kafka. This approach has several limitations in the Mendix context:
+
+* The platform does not natively support multi-component runtime deployments
+* Debezium does not support all databases that Mendix supports
+* The Mendix domain model does not map directly to the database schema, requiring additional transformation logic
+* A stream processor would be needed to apply the same data storage logic as the runtime, creating tight coupling
+* Domain model features like view entities are not represented as database tables
+
+## Data Streaming Primitives
+
+### Kafka Producer
+
+The Kafka producer publishes change events to the Event Broker. The producer is configured using the settings described in [Real-Time Data Sharing Configuration](/appstore/services/data-sharing-configuration/).
+
+### Kafka Consumer
+
+The Kafka consumer reads change events from topics and maps them to Mendix entities in the consuming application's database.
+
+## Publishing Change Event Streams to Event Broker
+
+The publishing flow consists of the following steps:
+
+1. Data storage detects entity changes (inserts, updates, deletes) for published entities.
+2. Change events are queued in the outbox.
+3. The producer dequeues events, encodes them as Kafka messages, and publishes to the Event Broker.
+
+The outbox pattern ensures reliable delivery. Events are persisted locally before publishing, and remain in the outbox until successfully delivered.
+
+## Historical Data and Snapshots
+
+Change events capture incremental updates. To provide the complete current state of entities, a snapshot is triggered during the initial deployment. The snapshot publishes all existing entity records as insert events. Snapshots are not triggered on subsequent application restarts.
+
+## Compacted Topics
+
+Kafka log compaction retains only the latest event for each entity key. This reduces storage requirements and allows new consumers to catch up faster by reading only the current state rather than the full change history.
+
+## Message Format
+
+CDC message formats lack standardization. The Debezium format is commonly used for database change events. Cloud Events (used by Business Events) is less common for CDC use cases. The message format will be determined during implementation.
+
+## Publishing Change Event Streams to Third-Party Systems
+
+### Mendix Cloud
+
+The Event Broker Bridge feature supports publishing data streams to external systems. Extending the existing bridge implementation to support data streams maintains a consistent integration pattern.
+
+### BYOK Model
+
+For deployments outside Mendix Cloud, bridges are not currently available. To support all deployment models, bridges could be defined in Studio Pro and transformed at build time into configurations for Kafka connector systems (for example, Kafka Connect, Bento, Redpanda Connect). This requires further design work based on customer requirements.
+
+## Non-Functional Requirements
+
+Real-Time Data Sharing addresses the following non-functional requirements:
+
+* **Exactly-Once and Ordered Delivery** 
+* **Elasticity** 
+* **High Throughput and Low Latency** 
+* **Durability**
+* **Data Correctness for Downstream Mendix Systems** 
+
+## Read More
+
+* [Real-Time Data Sharing Configuration](/appstore/services/data-sharing-configuration/)
+* [Mendix Event Broker](/appstore/services/event-broker/)
+* [Event Broker Bridges](/appstore/services/event-broker/#manage-mx-broker-bridge)