INGESTION-PIPELINE

Built with the tools and technologies:

Table of Contents

• Overview
• Features
• Project Structure
  • Project Index
• Getting Started
  • Prerequisites
  • Installation
  • Usage
  • Testing
• Project Roadmap
• Contributing
• License
• Acknowledgments

---

Overview

An LLM powered system that detects CVEs in container images and offers a chatbot for insights into vulnerabilities and mitigation strategies

---

Project Structure

└── ingestion-pipeline/
    ├── .github
    │   └── workflows
    ├── ChatCVE_logs.log
    ├── ChatCVE_logs.log.2024-12-10
    ├── README.md
    ├── __pycache__
    │   ├── download.cpython-311.pyc
    │   ├── processor.cpython-311.pyc
    │   └── uploader.cpython-311.pyc
    ├── airflow
    │   ├── Dockerfile
    │   ├── __init__.py
    │   ├── bin
    │   ├── dags
    │   ├── docker-compose.yaml
    │   └── requirements.txt
    ├── artifacts
    │   ├── image-urls.zip
    │   └── image_urls.txt
    ├── chatbot
    │   ├── .env
    │   ├── README.md
    │   ├── __pycache__
    │   ├── llama_index_chatbot.py
    │   └── main.py
    ├── chatbot_pg
    │   ├── .env
    │   ├── __pycache__
    │   ├── db.py
    │   └── main.py
    ├── consumer
    │   ├── .dockerignore
    │   ├── .env
    │   ├── Dockerfile
    │   ├── README.md
    │   ├── __pycache__
    │   ├── consumer.py
    │   ├── main.py
    │   ├── requirements.txt
    │   ├── sample.env
    │   └── snowflake_uploader.py
    ├── demo_examples
    │   ├── Dockerfile_busybox_patch
    │   └── Dockerfile_busybox_vul
    ├── download.py
    ├── extracted_files
    │   └── deltaCves
    ├── images.txt
    ├── lc.py
    ├── llama-chatbot
    │   ├── .env
    │   ├── eval.py
    │   └── main.py
    ├── processor.py
    ├── requirements.txt
    └── scanner
        ├── ChatCVE_logs.log
        ├── __pycache__
        └── scan.py

---

Getting Started

Prerequisites

Before getting started with ingestion-pipeline, ensure your runtime environment meets the following requirements:

• Programming Language: Error detecting primary_language: {'txt': 5, '2024-12-10': 1, 'py': 16, 'yaml': 1, 'json': 714, 'yml': 1}
• Package Manager: Pip
• Container Runtime: Docker

Installation

Install ingestion-pipeline using one of the following methods:

Build from source:

1. Clone the ingestion-pipeline repository:

❯ git clone git@github.com:cve-data-engineering/ingestion-pipeline.git

2. Navigate to the project directory:

❯ cd ingestion-pipeline

3. Install the project dependencies:

❯ pip install -r requirements.txt

---

Contributing

• 💬 Join the Discussions: Share your insights, provide feedback, or ask questions.
• 🐛 Report Issues: Submit bugs found or log feature requests for the ingestion-pipeline project.
• 💡 Submit Pull Requests: Review open PRs, and submit your own PRs.

Contributing Guidelines

1. Fork the Repository: Start by forking the project repository to your LOCAL account.
2. Clone Locally: Clone the forked repository to your local machine using a git client.

   git clone git@github.com:cve-data-engineering/ingestion-pipeline.git

3. Create a New Branch: Always work on a new branch, giving it a descriptive name.

   git checkout -b new-feature-x

4. Make Your Changes: Develop and test your changes locally.
5. Commit Your Changes: Commit with a clear message describing your updates.

   git commit -m 'Implemented new feature x.'

6. Push to LOCAL: Push the changes to your forked repository.

   git push origin new-feature-x

7. Submit a Pull Request: Create a PR against the original project repository. Clearly describe the changes and their motivations.
8. Review: Once your PR is reviewed and approved, it will be merged into the main branch. Congratulations on your contribution!

Contributor Graph

---