Merge branch 'release/0.9.3'

azavea · Jun 5, 2023 · 91836f0 · 91836f0
2 parents 1a12761 + ff0f43d
commit 91836f0
Show file tree

Hide file tree

Showing 35 changed files with 669 additions and 77 deletions.
diff --git a/.github/ISSUE_TEMPLATE/release.md b/.github/ISSUE_TEMPLATE/release.md
@@ -31,6 +31,7 @@ assignees: ''
     ```bash
     ./scripts/manage ecsmanage -e production migrate
     ```
+- [ ] If necessary, [publish any new tiles](https://github.com/azavea/iow-boundary-tool/blob/develop/data/README.md#deployment)
 - [ ] Finish and publish the release branch:
     - When prompted, keep default commit messages
     - Use `X.Y.Z` as the tag message

diff --git a/.gitignore b/.gitignore
@@ -43,3 +43,7 @@ src/app/build
 # Config
 .env
 .envrc
+
+# Data
+data/
+!data/README.md
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -19,6 +19,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Security
 
+## [0.9.3] - 2023-06-05
+
+### Added
+
+- Document AWS Architecture [#231](https://github.com/azavea/iow-boundary-tool/pull/231)
+- Document Business Logic [#232](https://github.com/azavea/iow-boundary-tool/pull/232)
+- Add Ingested Tile Support, National Municipal Layer [#234](https://github.com/azavea/iow-boundary-tool/pull/234)
+
 ## [0.9.2] - 2022-12-06
 
 ### Fixed
@@ -144,7 +152,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Show correct utility in NavBar [#218](https://github.com/azavea/iow-boundary-tool/pull/218)
 
 
-[Unreleased]: https://github.com/azavea/iow-boundary-tool/compare/0.9.2...HEAD
-[0.9.2]: https://github.com/azavea/iow-boundary-tool/compare/0.9.0...0.9.2
+[Unreleased]: https://github.com/azavea/iow-boundary-tool/compare/0.9.3...HEAD
+[0.9.3]: https://github.com/azavea/iow-boundary-tool/compare/0.9.2...0.9.3
+[0.9.2]: https://github.com/azavea/iow-boundary-tool/compare/0.9.1...0.9.2
 [0.9.1]: https://github.com/azavea/iow-boundary-tool/compare/0.9.0...0.9.1
 [0.9.0]: https://github.com/azavea/iow-boundary-tool/compare/0a16671...0.9.0
diff --git a/data/README.md b/data/README.md
@@ -0,0 +1,98 @@
+# Application Data
+
+## Tile Data
+
+In cases when we need custom layers on the map that are too large to ship as GeoJSON or raster layers, we ingest and host it ourselves.
+
+### Hosting
+
+The tiles are stored in S3 and served via CloudFront Distributions, which have friendly URLs:
+
+| Environment | S3 Bucket                        | URL                                             |
+|-------------|----------------------------------|-------------------------------------------------|
+| Staging     | `iow-staging-tiles-us-east-1`    | https://tiles.staging.iow.azavea.com/           |
+| Production  | `iow-production-tiles-us-east-1` | https://tiles.boundarysync.internetofwater.app/ |
+
+### Ingest Script
+
+Ensure [`ogr2ogr`](https://gdal.org/programs/ogr2ogr.html) and [`tippecanoe`](https://github.com/mapbox/tippecanoe) are installed:
+
+```bash
+ogr2ogr --version
+GDAL 3.5.3, released 2022/10/21
+```
+```bash
+tippecanoe --version
+tippecanoe v2.26.1
+```
+
+The GeoJSON is converted to a Protobuf Vector Grid using [`scripts/ingest-vector-tiles`](../scripts/ingest-vector-tiles). This script uses `ogr2ogr` to transform the input into EPSG:4326:
+
+```bash
+ogr2ogr \
+    -f GeoJSON "${INPUT_GEOJSON_FILE}_4326.geojson" \
+    -t_srs EPSG:4326 \
+    $(basename -- "$1")
+```
+
+Then it runs `tippecanoe` to generate a directory of Z/X/Y ordered Protobuf tiles:
+
+```bash
+tippecanoe \
+    --maximum-zoom=12 \
+    --output-to-directory="${INPUT_GEOJSON_FILE}" \
+    --layer="${INPUT_GEOJSON_FILE}" \
+    --coalesce-densest-as-needed \
+    --no-tile-compression \
+    "${INPUT_GEOJSON_FILE}_4326.geojson"
+```
+
+We use `--coalesce-densest-as-needed` to group dense features together at low zoom levels to keep the tile sizes small. We also use `--no-tile-compression` which prevents the default GZIP compression, since Leaflet's VectorGrid extension can't read compressed Protobuf tiles. The `--maximum-zoom=12` provides a good balance between file size and detail available to the users while drawing shapes. The value of 12 was chosen after some trial and error to keep the tileset size low while also having enough detail for when users are drawing their shapes. After 12 there are diminishing returns on increased size. At 12 the tileset is about less than 400MB, and costs about $0.05 / month to host with 10K requests, which is quite affordable.
+
+### Ingest Process
+
+To ingest a GeoJSON file into vector tiles, copy the file into this `data` directory, then run the script:
+
+```bash
+../scripts/ingest-vector-data example.geojson
+```
+
+This will create a `example/` directory. Sync this up to AWS:
+
+```bash
+aws --profile=iow-boundary-tool s3 cp example s3://iow-staging-tiles-us-east-1/example/ --recursive
+```
+
+If overwriting an existing tileset, also invalidate the in CloudFront:
+
+```bash
+AWS_CDN=$(aws --profile=iow-boundary-tool cloudfront list-distributions --query "DistributionList.Items[*].{id: Id, origin: Origins.Items[0].Id}[?origin=='S3-iow-staging-tiles-us-east-1'].id" --output text)
+
+aws --profile=iow-boundary-tool cloudfront create-invalidation --distribution-id $AWS_CDN --paths "/example/*"
+```
+
+Add a reference to this in the application, using a tile URL like:
+
+```js
+`https://${window.ENVIRONMENT.IOW_TILES_HOST}/example/{z}/{x}/{y}.pbf`
+```
+
+and test in development / staging.
+
+### Deployment
+
+When ready to push to production, copy over the tiles:
+
+```bash
+aws --profile=iow-boundary-tool s3 cp s3://iow-staging-tiles-us-east-1/example/ s3://iow-production-tiles-us-east-1/example/ --recursive
+```
+
+This command should be run from the Bastion to minimize S3 costs and maximize performance.
+
+If overwriting an existing tileset, also invalidate the in CloudFront:
+
+```bash
+AWS_CDN=$(aws --profile=iow-boundary-tool cloudfront list-distributions --query "DistributionList.Items[*].{id: Id, origin: Origins.Items[0].Id}[?origin=='S3-iow-production-tiles-us-east-1'].id" --output text)
+
+aws --profile=iow-boundary-tool cloudfront create-invalidation --distribution-id $AWS_CDN --paths "/example/*"
+```
diff --git a/deployment/README.md b/deployment/README.md
@@ -60,4 +60,16 @@ This file lives at `s3://iow-boundary-tool-production-config-us-east-1/terraform
 
 GitHub Actions will deploy this project's core infrastructure.  Deploy manually by traversing to the [repo's Actions tab, selecting 'Release'](https://github.com/azavea/iow-boundary-tool/actions/workflows/release.yml), then using the 'Run workflow' dropdown menu to enter in the Short Git commit hash to deploy to production.
 
-Once the release workflow has been kicked off, the deployment can be watched by simply clicking on the run to review summary information.
+Once the release workflow has been kicked off, the deployment can be watched by simply clicking on the run to review summary information.
+
+## Workflows
+
+We use GitHub Actions workflows for deploying to staging and production environments.
+
+Every merge to `develop` will trigger the [`continuous_integration` workflow](../.github/workflows/continuous_integration.yml). This will also be triggered for any branches that start with `release/`, `hotfix/`, or `test/`. Every commit to any such branch will result in a new container image being created and pushed to ECR, and from there deployed to the ECS Staging Cluster:
+
+![AWS Staging Deployment](../doc/diagrams/aws_staging_deployment.png)
+
+To deploy to production, create [a new Release issue](https://github.com/azavea/iow-boundary-tool/issues/new?assignees=&labels=release&template=release.md&title=Release+X.Y.Z), and replace `X.Y.Z` in the issue title and body with the version you intend to release. Then follow the checklist items until the release is finished. This will reuse an existing container image on ECR that has already been deployed to and tested on staging, and deploy it the ECS Production Cluster:
+
+![AWS Production Deployment](../doc/diagrams/aws_production_deployment.png)
diff --git a/deployment/terraform/.terraform.lock.hcl b/deployment/terraform/.terraform.lock.hcl
diff --git a/deployment/terraform/app.tf b/deployment/terraform/app.tf
@@ -125,6 +125,8 @@ resource "aws_ecs_task_definition" "app" {
     # aws_storage_bucket_name = aws_s3_bucket.media.id
     default_from_email = var.default_from_email
 
+    iow_tiles_host = "tiles.${var.r53_public_hosted_zone}"
+
     port = local.django_container_port
 
     project     = var.project
@@ -193,6 +195,8 @@ resource "aws_ecs_task_definition" "app_cli" {
     # aws_storage_bucket_name = aws_s3_bucket.media.id
     default_from_email = var.default_from_email
 
+    iow_tiles_host = "tiles.${var.r53_public_hosted_zone}"
+
     project     = var.project
     environment = var.environment
     aws_region  = var.aws_region

diff --git a/deployment/terraform/cdn.tf b/deployment/terraform/cdn.tf
@@ -88,3 +88,63 @@ resource "aws_cloudfront_distribution" "cdn" {
     Environment = var.environment
   }
 }
+
+resource "aws_cloudfront_origin_access_control" "tiles" {
+  name                              = "${var.environment}-tile-cloudfront-access-control"
+  description                       = "origin access control"
+  origin_access_control_origin_type = "s3"
+  signing_behavior                  = "always"
+  signing_protocol                  = "sigv4"
+}
+
+resource "aws_cloudfront_distribution" "tiles" {
+  comment = "${var.environment} tiles distribution"
+
+  origin {
+    domain_name              = aws_s3_bucket.tiles.bucket_regional_domain_name
+    origin_id                = "S3-${aws_s3_bucket.tiles.id}"
+    origin_access_control_id = aws_cloudfront_origin_access_control.tiles.id
+  }
+
+  enabled         = true
+  is_ipv6_enabled = true
+  http_version    = "http2"
+  aliases         = ["tiles.${var.r53_public_hosted_zone}"]
+
+  default_cache_behavior {
+    target_origin_id       = "S3-${aws_s3_bucket.tiles.id}"
+    viewer_protocol_policy = "redirect-to-https"
+    allowed_methods        = ["GET", "HEAD"]
+    cached_methods         = ["GET", "HEAD"]
+    forwarded_values {
+      query_string = false
+      cookies {
+        forward = "none"
+      }
+    }
+    min_ttl     = 0
+    default_ttl = 3600
+    max_ttl     = 86400
+  }
+
+  logging_config {
+    include_cookies = false
+    bucket          = aws_s3_bucket.logs.bucket_domain_name
+  }
+
+  restrictions {
+    geo_restriction {
+      restriction_type = "none"
+    }
+  }
+
+  viewer_certificate {
+    acm_certificate_arn = module.cert.arn
+    ssl_support_method  = "sni-only"
+  }
+
+  tags = {
+    Project     = var.project
+    Environment = var.environment
+  }
+}
diff --git a/deployment/terraform/dns.tf b/deployment/terraform/dns.tf
@@ -110,3 +110,15 @@ resource "aws_route53_record" "ses_dkim" {
   ttl     = "300"
   records = ["${aws_ses_domain_dkim.app.dkim_tokens[count.index]}.dkim.amazonses.com"]
 }
+
+resource "aws_route53_record" "tiles" {
+  zone_id = aws_route53_zone.external.zone_id
+  name    = "tiles.${var.r53_public_hosted_zone}"
+  type    = "A"
+
+  alias {
+    name                   = aws_cloudfront_distribution.tiles.domain_name
+    zone_id                = aws_cloudfront_distribution.tiles.hosted_zone_id
+    evaluate_target_health = false
+  }
+}
diff --git a/deployment/terraform/storage.tf b/deployment/terraform/storage.tf
@@ -61,3 +61,64 @@ resource "aws_s3_bucket" "data" {
     Environment = var.environment
   }
 }
+
+resource "aws_s3_bucket" "tiles" {
+  bucket = "${lower(replace(var.project, " ", ""))}-${lower(var.environment)}-tiles-${var.aws_region}"
+  acl    = "private"
+
+  cors_rule {
+    allowed_headers = ["*"]
+    allowed_methods = ["HEAD", "GET"]
+    allowed_origins = lower(var.environment) == "staging" ? ["http://localhost:4545", "http://localhost:8181", "https://${var.r53_public_hosted_zone}"] : ["https://${var.r53_public_hosted_zone}"]
+    expose_headers  = ["ETag"]
+    max_age_seconds = 300
+  }
+
+  server_side_encryption_configuration {
+    rule {
+      apply_server_side_encryption_by_default {
+        sse_algorithm = "AES256"
+      }
+    }
+  }
+
+  tags = {
+    Name        = "${lower(replace(var.project, " ", ""))}-${lower(var.environment)}-tiles-${var.aws_region}"
+    Project     = var.project
+    Environment = var.environment
+  }
+}
+
+# Allow tile access only from the Cloudfront CDN
+data "aws_iam_policy_document" "tiles" {
+  statement {
+    principals {
+      type        = "Service"
+      identifiers = ["cloudfront.amazonaws.com"]
+    }
+
+    actions = [
+      "s3:GetObject",
+      "s3:ListBucket",
+    ]
+
+    resources = [
+      aws_s3_bucket.tiles.arn,
+      "${aws_s3_bucket.tiles.arn}/*",
+    ]
+
+    condition {
+      test     = "StringEquals"
+      variable = "AWS:SourceArn"
+
+      values = [
+        aws_cloudfront_distribution.tiles.arn
+      ]
+    }
+  }
+}
+
+resource "aws_s3_bucket_policy" "tiles" {
+  bucket = aws_s3_bucket.tiles.id
+  policy = data.aws_iam_policy_document.tiles.json
+}
diff --git a/deployment/terraform/task-definitions/app.json.tmpl b/deployment/terraform/task-definitions/app.json.tmpl
@@ -54,6 +54,10 @@
       {
         "name": "R53_PUBLIC_HOSTED_ZONE",
         "value": "${r53_public_hosted_zone}"
+      },
+      {
+        "name": "IOW_TILES_HOST",
+        "value": "${iow_tiles_host}"
       }
     ],
     "mountPoints": [],

diff --git a/deployment/terraform/task-definitions/app_cli.json.tmpl b/deployment/terraform/task-definitions/app_cli.json.tmpl
@@ -46,6 +46,10 @@
       {
         "name": "R53_PUBLIC_HOSTED_ZONE",
         "value": "${r53_public_hosted_zone}"
+      },
+      {
+        "name": "IOW_TILES_HOST",
+        "value": "${iow_tiles_host}"
       }
     ],
     "mountPoints": [],

diff --git a/deployment/terraform/versions.tf b/deployment/terraform/versions.tf
@@ -3,7 +3,7 @@ terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 3.73.0"
+      version = "~> 4.63.0"
     }
     template = {
       version = "~> 2.2.0"
-Original file line number
+Diff line change
@@ Expand Up / @@ -43,3 +43,7 @@ src/app/build @@
     # Config
     .env
     .envrc
+    # Data
+    data/
+    !data/README.md