Skip to content

jvanheerikhuize/data-collection-framework

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Data Pipeline Framework

A flexible, 3-layer Node.js framework for collecting data from APIs, processing it according to specific use cases, and mapping it to various output formats. It's designed to be entirely configuration-driven, allowing you to create complex data pipelines without writing new code.

✨ Features

  • Layered Architecture: Enforces separation of concerns between data collection, business logic, and presentation.
  • Chainable API Calls: Use data from one API call as input for the next.
  • Configuration Driven: Define entire workflows in simple YAML files.
  • Multiple Output Formats: Easily map processed data to YAML, JSON, or CSV for reports and charts.
  • Secure Credential Handling: Uses environment variables to keep secrets out of configuration files.

📂 Project Structure

The framework is organized into a src directory for the core logic and a usecases directory for configurations.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published