Skip to content

A daily feedback dashboard for 311 complaints resolution trend

Notifications You must be signed in to change notification settings

lingyielia/411-4-311

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

40 Commits
 
 
 
 
 
 
 
 

Repository files navigation

411for311

-> Demo -> Slides

demo Figure 1. Final Dashboard on Tableau. In this dashboard, all blue components means recent records, green means historical, purple means the difference between recent and historical. When a particular department being selected, only the data of this department will be rendered on the heat maps, historical line plot, and recent average count.

1. Overview

This is a daily review dashboard showing 311 complaints resolution performance for government agencies. With new complaint records being available via API, the data transform process is scheduled daily to extract the new data and produce a daily review dashboard describing the performance of each agency who is responsible for resolving the requested issues, compared with the historical records.

diagram Figure 2. Data Flow Diagram. For historical data processing, the EMR was utilized once. For the daily updating process, CloudWatch is the trigger for the whole updating process, and data flows are driven by Lambda functions.

Ideas behind the pipeline design

  • Cost efficient. As data are processed once daily, using Lambda functions (pay only for what you use) helps reduce the cost.
  • Handling the API unavailability. As the data source API is under maintenance monthly, the pipeline should 1) not crash when the API is unavailable and 2) be able to process all unprocessed data when the API is back.
  • Extensible and scalable. With Kinesis as the ingestion layer, this pipeline can accept more data producers (e.g. adding other APIs) and more data consumers. Meanwhile, Lambda functions, Kinesis, S3, and Redshift are all easily scalable ready for future growth of data.

2. Requirements

3. Installation

4. Data Source

311 complaints dataset is available in NYC Open Data. Data can be accessed via Socrate API.

5. Getting Started

bash ./src/cloudwatch/run_cloudwatch_trigger.sh

About

A daily feedback dashboard for 311 complaints resolution trend

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published