DATA605_Spring2026_Real_Time_Stock_Market_Pipeline_Kafka_Spark

[aashishvinod]

Project Description
Author: @aashishvinod
Title: DATA605_Spring2026_Real_Time_Tweet_Sentiment_Analysis_Kafka
Course: DATA605 Spring 2026
Link: https://github.com/gpsaggese/gpsaggese.github.io/blob/master/class_project/data605/Spring2026/projects_descriptions/Apache_Kafka_Project_Description.md

Summary: This project builds a real-time tweet sentiment analysis pipeline using Apache Kafka for stream ingestion and a pre-trained HuggingFace RoBERTa model (cardiffnlp/twitter-roberta-base-sentiment) for sentiment classification. Tweets from the Sentiment140 dataset (1.6M labeled tweets) are streamed through Kafka, classified in real time, and visualized using a live Streamlit dashboard with sentiment trends, comparative model analysis, and anomaly detection.

Planned Workflow:

1. Load and preprocess Sentiment140 dataset (1.6M tweets)
2. Kafka producer setup (publish tweet events to Kafka topic)
3. HuggingFace RoBERTa model loading and classification
4. Kafka consumer setup (consume and classify tweets in real time)
5. Sentiment aggregation (positive, negative, neutral counts and accuracy)
6. Spark SQL analysis (sentiment distribution, high confidence predictions)
7. Comparative model analysis (RoBERTa vs DistilBERT accuracy comparison)
8. Anomaly detection (detect sudden spikes in sentiment using rolling statistics)
9. Live Streamlit dashboard (real-time pie chart, trend chart, metrics)
10. Performance analysis (Kafka producer throughput vs batch size)

Tools: Apache Kafka, HuggingFace Transformers (RoBERTa), Apache Spark (PySpark), Streamlit, Docker, Python, Sentiment140 Dataset

Assigned to: @aashishvinod @gpsaggese @protocorn

[protocorn]

Hi @aashishvinod , it looks like the link currently points to some other project. Could you please check if this is the correct link and update it to your specific project repository if not? Thanks!

[aashishvinod]

Hi @protocorn ,

Thank you so much for taking the time to review my issue and for flagging this!

The link in the issue description points to the official Apache Kafka project description file from the course repository, which is the same reference link for all students who selected this project topic. I added it as per the README guidelines which say to include a link to the project specs document.

My actual project submission and all my code is in PR #478:
#478

My project files are located at:
class_project/data605/Spring2026/projects/UmdTask461_DATA605_Spring2026_Real_Time_Stock_Market_Pipeline_Kafka_Spark/

I hope this clears things up! Please do let me know if there is anything else you would like me to update or fix. I really appreciate your help and feedback!

Thank you,
@aashishvinod

[protocorn]

Hi @aashishvinod,

Just to clarify, that link is not a general reference. It points to a specific project description with a defined objective, which is, real-time sentiment analysis on tweets using Kafka Streams and a pre-trained NLP model. Please open the file and compare it against what you built.

[aashishvinod]

Hi @protocorn ,

Thank you for the clarification and for taking the time to review this carefully.

You are absolutely right. After reading the Apache Kafka project description,
I can see it specifies a sentiment analysis application on tweets, which is
different from what I built.

I built a Real-Time Stock Market Pipeline using Apache Kafka and Spark
Structured Streaming, which I believe covers all the core Kafka concepts
from the description (producers, consumers, topics, partitions, real-time
stream processing) and extends them further with Spark batch processing,
window functions, SQL queries, moving averages, price alert detection,
and volatility analysis across 3,155 events, making it arguably more
complex than the original sentiment analysis objective.

I apologize for not verifying the description more carefully at the start
and I take full responsibility for the mismatch. Could you please let me
know how you would like me to proceed? I am happy to make any changes
based on your guidance and will fix it right away.

Thank you for your patience,
@aashishvinod

[protocorn]

Hi @aashishvinod ,

I can see you've put a lot of quality work into the stock pipeline, but the project topics are set to ensure everyone covers specific skills and follows it. Even though your current build is complex, it doesn't quite hit those specific project goals.

Since you already have the Kafka and Spark infrastructure working, I would strongly recommend you to pivot. You’ll just need to swap the stock data for a text-based stream and add the sentiment analysis logic as originally outlined.

Please update the project to align with the official description before the final submission.

[aashishvinod]

Hi @protocorn,

Thank you for the feedback! I have already taken your advice and pivoted the project. I have built a complete real-time tweet sentiment analysis pipeline as described in the official project description and submitted it in PR #493.

The new project includes:

• Kafka producer streaming tweets from the Sentiment140 dataset
• Pre-trained HuggingFace RoBERTa model for real-time sentiment classification
• Live Streamlit dashboard with sentiment metrics and trend visualization
• Comparative model analysis (RoBERTa vs DistilBERT)
• Anomaly detection for sentiment spikes
• Spark SQL analysis and aggregations
• Working Docker setup

PR: #493

Could you please review PR #493 and let me know if everything looks good?

Thank you so much for your guidance!
@aashishvinod