Skip to content

[GSoC 2021] Integrate Apache Airflow to manage Crawler jobs #102

@beshiniii

Description

@beshiniii

Is your feature request related to a problem? Please describe.

Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. It provides following advantages.

  • Easy to manage workflows
  • Guranteed delivery
  • Easy to monitor workflows
  • Easy to schedule Crawler jobs
  • Supports many many Message broker implementations
  • Container native support
  • Dashboard support

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions