- Create airflow project:
docker compose -f airflow-docker-compose.yaml up -d --buildAfter that, you can see the following structure:
.
├── airflow/ /* airflow folder including dags,.. /*
│ ├── config/ /* contain config files for airflow /*
│ ├── dags/ /* contain all dags in pipeline /*
│ ├── elt_pipeline_dag.py /* dag: extract - load - process (ELT pipeline) and convert to delta lake format /*
│ └── delta_dag.py /* dag: convert parquet to delta lake format /*
│ ├── data/
│ ├── include/
│ ├── plugins/
│ ├── Dockerfile /* build image to run airflow container /*
│ ├── README.md
│ └── requirements.txt /* dependencies for airflow /*- Access Airflow webserver:
You can access at http://localhost:8080/ and login with username && password is airflow
- Run DAG:
You can manually run DAGS by click on Run
Airflow Result
© 2024 NhatNguyen