Skip to content

Airflow powered ETL pipeline for moving Near-Earth-Object data from NASA to Google Cloud

License

Notifications You must be signed in to change notification settings

Shegzimus/DE_NASA_NeoW_Pipeline

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

77 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Table of Contents

Motivation and Objectives

Overview

Architecture

New Personal Insights

Prerequisites

System Configuration

  1. Clone the repository

    git clone https://github.com/Shegzimus/DE_NASA_NeoW_Pipeline
  2. Create a virtual environment in your local machine

    python3 -m venv venv
  3. Activate the virtual environment

    source venv/bin/activate
  4. Install dependencies

    pip install -r airflow/requirements.txt
  5. Create directories to store your google credentials

    cd airflow && mkdir -p .google
    
  6. Build the Docker Image

    docker build -d --
  7. Start the Docker containers

    docker-compose up -d
  8. Launch the Airflow web UI

    open http://localhost:8083