Data Engineering Zoomcamp

See the full course here

Overview

Google Cloud Platform (GCP): Cloud-based auto-scaling platform by Google. Only used in Week 1. For subsequent weeks, Azure is used instead due to personal lack of access.
- Google Cloud Storage (GCS): Data Lake
- BigQuery: Data Warehouse
Azure : Cloud platform developed by Microsoft. Used in Weeks 2-4.
- Azure Data Lake Storage Gen2 (ADLS): Data Lake
- Azure Databricks: Data Processing
- Azure Synapse Analytics (Synapse): Data Warehouse
Terraform: Infrastructure-as-Code (IaC)
Docker: Containerization
SQL: Data Analysis & Exploration
Prefect: Workflow Orchestration
dbt: Data Transformation
Spark: Distributed Processing
Kafka: Streaming

Name		Name	Last commit message	Last commit date
Latest commit History 19 Commits
dbt_synapse		dbt_synapse
flows		flows
logs		logs
resources		resources
spark		spark
terraform_azure		terraform_azure
terraform_gcp		terraform_gcp
visualization		visualization
.gitignore		.gitignore
.prefectignore		.prefectignore
Dockerfile		Dockerfile
README.md		README.md
docker-compose.yaml		docker-compose.yaml
etl_local_flow-deployment.yaml		etl_local_flow-deployment.yaml
etl_parent_flow-deployment.yaml		etl_parent_flow-deployment.yaml
etl_yearly_flow-deployment.yaml		etl_yearly_flow-deployment.yaml
requirements.txt		requirements.txt
synapse_parent_flow-deployment.yaml		synapse_parent_flow-deployment.yaml
week_1_notes.md		week_1_notes.md
week_1_solutions.md		week_1_solutions.md
week_2_notes.md		week_2_notes.md
week_2_solutions.md		week_2_solutions.md
week_3_solutions.md		week_3_solutions.md
week_4_notes.md		week_4_notes.md
week_4_solutions.md		week_4_solutions.md
week_5_solutions.md		week_5_solutions.md