🚀 "Unveiling My Latest Data Engineering Endeavor: Real-Time Stock Market Analytics Powered by Apache Kafka!"
🎯 Goal Accomplished: Developed an end-to-end pipeline to streamline real-time stock market data for analytics.
🔧 Tools Used: Apache Kafka, AWS EC2 Instance, Jupyter Notebook, Athena, AWS Crawler, Amazon S3
🚧 Challenge Faced: Struggled with incessant errors while launching Java, Kafka, Zookeeper, and topics onto the EC2 Instance.
💡 Learnings Gained: Gained hands-on experience in stream processing using Kafka on the AWS platform. Launched an Apache Kafka cluster using CLI.
🙏 Project Reference and Support: Huge thanks to Darshil Parmar, Srinivasarao Polisetty, and ChatGPT for putting up with my endless questions.
📊 Check out the pipeline diagram and snapshots of the analytics on Athena and more in the document attached!