Make export configurable with asset types #5619
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
since custom types in arrays aren't supported by JDBC
instead of an enum type which isn't supported by doobie. See: https://tpolecat.github.io/doobie/docs/15-Extensions-PostgreSQL.html#array-types
Only download scene layer items if export definition includes that as an asset type.
if the original filename is missing
The STAC link doesn't actually have the S3 URI to the TIFF on it which is on the STAC item.
Without this, COGs with the same name (eg. cog.tif) will overwrite each other.
This function seems to not do anything for two reasons: 1. It doesn't look like there's a data key for scene item assets. There is for label items so it's possible this was copied over from that. 2. The maybeSignUri function is never passed in a whitelist so it looks like it would never have an effect and would just return the URI, though it doesn't seem like it's getting that far.
pcaisse
commented
Sep 1, 2021
| .transact(xa) | ||
| .unsafeToFuture | ||
| } | ||
| case StacExport.CampaignExport(_, _, _, _, campaignId) => |
There was a problem hiding this comment.
This was the only line I changed in this file, FYI. The rest is wacky formatting changes from scalafmt.
jisantuc
suggested changes
Sep 1, 2021
There was a problem hiding this comment.
Two things that are no longer correct are:
- the README links to items at the location where they used to live, instead of at the new location (e.g.
images/6c73b56a-6fd1-4c02-888b-320f0545fd87.jsoninstead ofimages/6c73b56a-6fd1-4c02-888b-320f0545fd87/item.json) - the label items' source links do the same (
../images/415a9426-671d-4a89-b0fd-cb2e4fe4660b.jsoninstead of../images/415a9426-671d-4a89-b0fd-cb2e4fe4660b/item.json)
Other than that though looks good 😎 Nice work on something that turned out to be way larger than advertised
Comment on lines
+173
to
+180
| fs2.io | ||
| .readInputStream( | ||
| IO(s3Object.getObjectContent().getDelegateStream), | ||
| // Chunk size is set to default used for S3 CLI multipart (8MB) | ||
| // See: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html#multipart-chunksize | ||
| 1024 * 8, | ||
| ExportData.fileBlocker | ||
| ) |
There was a problem hiding this comment.
Nice, I remember trying to get the s3 client and fs2 to cooperate forever ago and I never figured out the magic combination of accesses on s3Object to get it to work 😎
| Decoder.decodeString.emap { str => | ||
| Either | ||
| .catchNonFatal(unsafeFromString(str)) | ||
| .leftMap(_ => "ExportAssetType") |
There was a problem hiding this comment.
You might want to provide a nicer error message here, since when bad strings are passed the user-facing message just says DecodingFailure at .exportAssetTypes[0]: ExportAssetType
pcaisse
commented
Sep 1, 2021
This is superceded by nelExportAssetTypesGet and nelExportAssetTypesPut
jisantuc
approved these changes
Sep 1, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Brief description of what this PR does, and why it is needed.
Checklist
BUILDCONFIG.APP_NAMEDemo
Notes
cog.tif) so the files would overwrite each other in the export since they're all in the same directory as it was previously organized (see 6302e11). To overcome this, I grouped item JSON and the corresponding COG (if configured) within a directory named after the item id to ensure uniqueness.export_asset_typesis just avarchar[]since fancier array types aren't supported by the JDBC driver as I understand it. I think enforcing everything via Scala types at the application and batch layers should get us enough safety in any case.Testing Instructions
Reassemble API and batch and restart the server.
Log into Groundwork as dev@rasterfoundry.com and get a JWT. Export it so
jwt-httpie-authcan findit:
Create an export via API request (this assumes you have dev data loaded because it depends on a
campaign with this id being in the database to work):
Ensure that only those two strings are accepted for the export asset type, list cannot be empty, and property is not required (can be null or missing).
Run a STAC export using the export id returned from the response:
Download the catalog at
s3://rasterfoundry-development-data-us-east-1/stac-exports/<export_id>/catalog.zipand unzip.Inspect the catalog to make sure the collection JSON and the item JSON reference each other correctly via links and that the images come down in the appropriate folder grouped by item id.
Ensure the assets on each item include both the signed URL with its expiration date and the COG with appropriate metadata.
Also check the STAC assets and links to make sure the link and media types are correct.
Closes #1352