repowiseapache/airflow

Architecture of apache/airflow

apache/airflow is 10,273 source files holding 90,080 symbols across 24 modules, written mainly in Python, TypeScript, Shell. The largest module is providers, with 5,887 files, and 909 files are entry points into the dependency graph.

How airflow fits together

Workflows are written as Dags in Python, and the Airflow scheduler runs their tasks on a set of workers while following the declared dependencies. The file airflow-core/src/airflow/__main__.py is the entry point for the airflow-core package, which holds the core Airflow code. airflow-core/src/airflow/api_fastapi/app.py defines the FastAPI application for the Airflow API. airflow-core/src/airflow/jobs/scheduler_job_runner.py contains the scheduler job runner that drives task scheduling. The providers directory holds provider distributions, and the chart directory holds the Helm Chart, which is a dependent project. Progress and results appear in the web user interface, which shows pipelines running in production.

Largest modules

  • providers: 5,887 files and 817,471 lines of code
  • airflow-core: 2,374 files and 378,615 lines of code
  • scripts: 387 files and 39,848 lines of code
  • dev: 308 files and 76,719 lines of code
  • task-sdk: 264 files and 65,281 lines of code
  • chart: 223 files and 43,173 lines of code
  • shared: 125 files and 15,233 lines of code
  • devel-common: 110 files and 13,160 lines of code

Languages

By file count: Python 7,745 files, TypeScript 958 files, Shell 96 files, Go 82 files, JavaScript 58 files, Kotlin 44 files.

Documentation

Repowise documents airflow in 11,386 pages generated from the source.

More on airflow: summary and FAQ, codebase overview, code health.

Related guide: Best dependency graph tools for monorepos