Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure correct footprints and tasks for pre-COGs #5627

Merged
merged 3 commits into from
Sep 16, 2021

Conversation

jisantuc
Copy link
Contributor

@jisantuc jisantuc commented Sep 15, 2021

Overview

Sometimes we need to COGify images on our own, but other times, they're pre-COGs:

When we have COGs already, we don't need to do work to turn the image into a COG. However, we do need to do work to make sure some metadata is correct. In particular, for EPSG:4326 COGs, we were doing a bad job at task side length calculation, and then when I fixed that, we were doing a bad job at footprint calculation.

This PR makes it so that:

  • we use a nice Web Mercator VRT for figuring out the task side length
  • we bail on the complicated footprint calculation and just let the API server handle it
  • we make sure that the scene is ready before we try to use its metadata in the database to calculate annotation project footprints

Checklist

  • Description of PR is in an appropriate section of the changelog and grouped with similar changes if possible

Demo

For my local copy of a poison tiff:

$ time ./scripts/process-upload 3c5df351-5868-4fbb-883b-48a96ce32cf4
...
________________________________________________________
Executed in   20.92 secs   fish           external
   usr time  1108.50 millis  170.00 micros  1108.33 millis
   sys time  248.30 millis  119.00 micros  248.18 millis

Here's the log stream for trying to process the same tif in staging. You can see that 24 minutes in it dies with some kind of SQL error.

Testing Instructions

  • Upload the poison tif from the linked issue in staging and wait for the job to die
  • build the batch container docker-compose build batch
  • make sure your servers are up-to-date just as a precaution ./sbt ';api/assembly;backsplash-server/assembly;batch/assembly'
  • bring up servers ./scripts/server
  • bring up GroundWork pointed to local servers
  • upload the same poision tif
  • process it ./scripts/process-upload <your id>
  • note that it doesn't take 24 minutes and die, but instead takes a short amount of time, succeeds, and you eventually get a footprint matching the imagery 🥖

Closes azavea/raster-foundry-platform#1328

Copy link
Contributor

@pcaisse pcaisse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a ton of context around RF image processing, but I can confirm that the poison TIFF poisoned my computer real good before these changes and with these changes image processing was nice and fast, and after processing I can view the image in GroundWork successfully.

app-tasks/rf/src/rf/utils/io.py Outdated Show resolved Hide resolved
@jisantuc jisantuc merged commit 2497581 into develop Sep 16, 2021
@jisantuc jisantuc deleted the bugfix/js/always-use-webmercator-edge-lengths branch September 16, 2021 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants