MLOps Workflow: Comprehensive Full-Stack Deployment

Overview

This project demonstrates how to build and deploy a machine learning model using a modern MLOps pipeline. The workflow is designed to be scalable, reproducible, and maintainable, leveraging the power of cloud services and open-source tools.

The primary objectives of this project include:

• Infrastructure as Code: Using Terraform to provision and manage cloud resources.
• Experiment Tracking: Logging and tracking model experiments with MLflow.
• CI/CD Automation: Automating the deployment pipeline with GitHub Actions.
• Containerization: Dockerizing applications for consistency across environments.
• Model Serving: Deploying the model as a REST API with FastAPI on GCP Cloud Run.
• Data Orchestration: Orchestrating data workflows with Prefect.
• Data Transformation: Utilizing dbt for data transformation in BigQuery.

Architecture

Project Structure

|-- .github/             # GitHub Actions CI/CD workflows
|-- assets/              # Project images
|-- dbt/                 # Data processing scripts and dbt models
|-- fastapi/             # FastAPI application
|-- mlflow/              # MLflow experiment scripts
|-- prefect/             # Prefect flows and tasks
|-- terraform/           # Terraform configuration files
`-- README.md            # Project documentation

Results

The final deployed model is accessible through the API endpoint provided by GCP Cloud Run. Data is automatically fetched and ingested into the GCP BigQuery data warehouse on a daily schedule. Users can review the results of model predictions, along with accuracy and performance metrics, via the MLflow UI.

Future Work

• Model Monitoring: Implement monitoring tools like Prometheus and Grafana for real-time monitoring.
• Hyperparameter Tuning: Integrate tools like Optuna for automated hyperparameter optimization.
• Scalability Improvements: Explore Kubernetes for managing large-scale deployments.

About the Developer

This project is maintained by Sai Nikhil Mattapalli, an AI/ML Engineer with 5 years of experience building and deploying scalable machine learning and generative AI solutions. With expertise in end-to-end ML pipelines, cloud infrastructure, and REST API development, he focuses on creating production-ready systems that bridge the gap between data science and software engineering.

Contact Information:

• Email: ms1104n@gmail.com
• Role: AI/ML Engineer