This project is used as a demo for my talk: Building 3D Trusted Data Pipelines With Dagster, Dbt, and Duckdb at PyCon Australia 2023
The slides for the workshop is on this link.
Data pipelines and architecture
Dagster global assets
git clone git@github.com:danhphan/trusted-data-pipeline.git
cd trusted-data-pipeline
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cd ./dbt_jaffle_shop/
dbt deps
dbt build
dbt run
dbt test
dbt test --select customers
dbt docs generate
dbt docs serve --port 8081
dbt show --inline "select count(*) from {{ ref('stg_customers') }}"
cd ./dagster_dbt_jaffle/
dagster dev # or: dagit