Real-Time E-Commerce Event Streaming Pipeline

A local, production-ready event streaming architecture that simulates real-time e-commerce transactions using Apache Kafka (KRaft mode) and Python 3.14.

---

🏗️ Project Structure & Data Flow

kafka-realtime-pipeline/
├── .gitignore               # Ignores virtual environments & cache
├── .python-version          # Pins Python 3.14 runtime environment
├── docker-compose.yml       # Provisions Kafka broker in KRaft mode
├── requirements.txt         # App dependencies (confluent-kafka, streamlit, pandas)
├── README.md                # Project documentation
│
├── producer/
│   └── producer.py          # Continuous transaction event engine
├── consumer/
│   ├── tracker.py           # Real-time backend shipping log processor
│   └── analytics.py         # Terminal-based sales metrics aggregator
└── dashboard/
    └── app.py               # Live web UI script powered by Streamlit

End-to-End Data Lifecycle

  [ Mock Order Generator ]  (producer/producer.py)
             │
             ▼  (Sends JSON payload with User Key)
   ┌────────────────────────────────────────────────────────┐
   │              APACHE KAFKA BROKER (KRaft)               │
   │               Topic: "orders"                          │
   │                                                        │
   │  ┌────────────────┐ ┌────────────────┐ ┌────────────┐  │
   │  │  Partition 0   │ │  Partition 1   │ │ Partition 2│  │
   │  │  [Carlos' mgs] │ │  [Alice's mgs] │ │ [Others]   │  │
   │  └────────────────┘ └────────────────┘ └────────────┘  │
   └─────────────────┬───────────────────────────┬──────────┘
                     │ (Broadcasts Data)         │ (Broadcasts Data)
                     ▼                           ▼
        ┌─────────────────────────┐ ┌─────────────────────────┐
        │  CONSUMER GROUP A       │ │  CONSUMER GROUP B       │
        │  (order-tracker-group)  │ │ (ecommerce-streamlit)   │
        ├─────────────────────────┤ ├─────────────────────────┤
        │ tracker.py              │ │ app.py                  │
        │ 📜 Real-time terminal   │ │ 📊 Real-time Streamlit │
        │    shipping receipt logs│ │    interactive charts   │
        └─────────────────────────┘ └─────────────────────────┘

1. Ingestion & Key Hashing: producer.py attaches a user string as the message key. Kafka hashes this key to guarantee that all orders from the same customer consistently land on the exact same partition, preserving strict chronological processing order.
2. Decoupled Log Storage: The Kafka broker appends events across 3 partitions. Because Kafka utilizes a decoupled publish-subscribe design, data is securely persisted on disk, allowing multiple consumer services to pull data independently at their own pace.
3. Parallel Processing: Consumer Group A (tracker.py) processes the stream to trace partition logistics, while Consumer Group B (app.py) simultaneously processes the exact same data stream to update the visual UI layout.

---

🛠️ Tech Stack & Cluster Management

• Infrastructure & Broker: Docker Desktop | Confluent Kafka v7.8.3
• Runtime & Libraries: Python 3.14.x | confluent-kafka, streamlit, pandas

ZooKeeperless Architecture (KRaft Mode)

This cluster relies entirely on KRaft (Kafka Raft) mode for metadata management, bypassing ZooKeeper entirely.

• Unified Process: Configured via KAFKA_PROCESS_ROLES: broker,controller to handle both storage and internal configuration quorum within a single local container.
• Optimized Failovers: Uses Raft-consensus controller voting to enable nearly instantaneous state synchronization and topic metadata allocation.

---

🚀 How to Run Locally

1. Spin up Infrastructure & Dependencies

Ensure Docker Desktop is active. In your project root folder, run:

docker compose up -d
pip install -r requirements.txt

2. Launch the Ecosystem (Side-by-Side Terminals)

Open three terminal windows simultaneously to watch the data interact in real time:

• Terminal 1 (Producer Engine):

python producer/producer.py

• Terminal 2 (Log Tracker):

python consumer/tracker.py

• Terminal 3 (Web UI Dashboard):

streamlit run dashboard/app.py

---

📺 Live Execution Metrics

Backend Console Activity

# Producer Stream
🚀 Real-time producer is running... Press Ctrl+C to stop.
✅ Sent to Partition [2] at Offset 11 -> {"user": "Deepika", "item": "Pineapple", "quantity": 1}
✅ Sent to Partition [0] at Offset 2  -> {"user": "Carlos", "item": "Mechanical Keyboard", "quantity": 2}

# Order Tracker Stream
🟢 Real-time Tracker Consumer is running... Press Ctrl+C to stop.
📦 Order Received: 1x Pineapple by Deepika ➡️ Partition [2] at Offset 11
📦 Order Received: 2x Mechanical Keyboard by Carlos ➡️ Partition [0] at Offset 2

Streamlit Web UI Rendering (http://localhost:8501)

📊 Real-Time E-Commerce Dashboard
📈 Live Units Sold Matrix
[==================== AUTO-REFRESHING BAR CHART ====================]

🏆 Detailed Sales Leaderboard
Product Name             Total Units Sold
Wireless Mouse           37
Mechanical Keyboard      36

---

🧠 Core Kafka Topologies Practiced

• Pub/Sub Fanout Optimization: Proves system orchestration by allowing completely different operational business logic instances to safely read from identical topics simultaneously without data collisions or lag dependencies.
• Offset Commit Durability: Leverages auto.offset.reset: earliest configuration strategies, tracking explicit offset check-pointing so consumers safely resume calculations from their exact crash-state history upon unexpected restarts.

---

🧹 Clean Up

When you are finished running your local pipelines, you can clean up your local host resources:

• Stop active environment containers: docker compose down
• Stop containers and completely clear stored Kafka volumes: docker compose down -v