Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make export configurable with asset types #5619

Merged
merged 43 commits into from
Sep 1, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
6651568
Add asset types to STAC export
pcaisse Aug 16, 2021
32dffb7
Add conversion between list and NEL in database
pcaisse Aug 20, 2021
099bf67
Make asset types NEL of strings
pcaisse Aug 20, 2021
72d1302
Update migration to be an array of strings
pcaisse Aug 20, 2021
f508583
Always include tile layer
pcaisse Aug 20, 2021
15a22d6
Allow for getting/putting export asset types directly
pcaisse Aug 20, 2021
69fe843
Use export definition when downloading images
pcaisse Aug 23, 2021
e1d1cc7
Use annotation project name for file
pcaisse Aug 23, 2021
a01fb0c
Add asset for signed URL
pcaisse Aug 25, 2021
e4c5ad0
Refactor to have single function for assets
pcaisse Aug 25, 2021
fc71d8a
Always sign URI
pcaisse Aug 25, 2021
b5cff96
Use one S3 client for campaign export
pcaisse Aug 25, 2021
a05ad1b
Add COG asset
pcaisse Aug 25, 2021
c8b8b66
Add COG link to item
pcaisse Aug 27, 2021
5af8368
Write COG to export
pcaisse Aug 30, 2021
18b14e0
Increment migration version number after rebase
pcaisse Aug 30, 2021
b3b106c
Add logging and fix COG link
pcaisse Aug 31, 2021
f5928ed
Change export asset type name
pcaisse Aug 31, 2021
6908b4f
Remove defunct STAC export v1 code
pcaisse Aug 31, 2021
c31f34b
Add more logging and TODO
pcaisse Aug 31, 2021
42c7c65
Fix writing COG
pcaisse Aug 31, 2021
5b361ed
Increase chunk size
pcaisse Aug 31, 2021
be68c1f
Use constants for asset keys
pcaisse Aug 31, 2021
3ddf460
Use filename for name of COG file in export
pcaisse Aug 31, 2021
1fd5424
Add signed URL expiration to asset description
pcaisse Aug 31, 2021
3fd7794
Write STAC item files independently
pcaisse Aug 31, 2021
66a27d8
Log when writing text files
pcaisse Aug 31, 2021
6302e11
Group item JSON and images in folder by id
pcaisse Aug 31, 2021
8d6f4c5
Remove no-op withAsset
pcaisse Aug 31, 2021
8659ab6
Add missing closed paren
pcaisse Aug 31, 2021
3832506
Fix parent links for imagery items
pcaisse Aug 31, 2021
f932209
Format
pcaisse Aug 31, 2021
3fbf22b
Remove other unused STAC export v1 files
pcaisse Aug 31, 2021
1e85807
Reinstate necessary function from deleted file
pcaisse Aug 31, 2021
eebb733
Add optional param in API route
pcaisse Aug 31, 2021
610102f
Update changelog
pcaisse Aug 31, 2021
fa3fdb0
Run scalafix
pcaisse Sep 1, 2021
07a1982
Remove unused implicit
pcaisse Sep 1, 2021
61914a4
Add generator for export asset type
pcaisse Sep 1, 2021
a2a4a4d
Fix whitespace
pcaisse Sep 1, 2021
190f142
Provide better error message for ExportAssetType decoder error
pcaisse Sep 1, 2021
38e95d1
Set correct item path for label source links and README
pcaisse Sep 1, 2021
8a153e2
Fix again
pcaisse Sep 1, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Indexed all foreign key references to users [#5611](https://github.com/raster-foundry/raster-foundry/pull/5611)
- Use original filename for COG [#5613](https://github.com/raster-foundry/raster-foundry/pull/5613)
- Extended backsplash healthcheck timeout to 10 seconds, which is longer than the ALB healthcheck timeout [#5616](https://github.com/raster-foundry/raster-foundry/pull/5616)
- Make export configurable with asset types [#5619](https://github.com/raster-foundry/raster-foundry/pull/5619)

### Fixed
- Updated some setup steps and Auth0 interaction for more convenient external user use [#5612](https://github.com/raster-foundry/raster-foundry/pull/5612)
Expand Down
207 changes: 111 additions & 96 deletions app-backend/api/src/main/scala/stac/Routes.scala
Original file line number Diff line number Diff line change
Expand Up @@ -72,122 +72,137 @@ trait StacRoutes
}
}

def listStacExports: Route = authenticate { user =>
authorizeScope(ScopedAction(Domain.StacExports, Action.Read, None), user) {
(withPagination & stacExportQueryParameters) {
(page: PageRequest, params: StacExportQueryParameters) =>
complete {
StacExportDao
.list(page, params, user)
.map(
p =>
PaginatedResponse[StacExport.WithSignedDownload](
p.count,
p.hasPrevious,
p.hasNext,
p.page,
p.pageSize,
p.results.map(signExportUrl(_))
)
)
.transact(xa)
.unsafeToFuture
}
}
}
}

def createStacExport: Route = authenticate { user =>
authorizeScope(ScopedAction(Domain.StacExports, Action.Create, None), user) {
entity(as[StacExport.Create]) { newStacExport =>
newStacExport match {
case StacExport
.AnnotationProjectExport(_, _, _, annotationProjectId) =>
authorizeAsync {
AnnotationProjectDao
.authorized(
user,
ObjectType.AnnotationProject,
annotationProjectId,
ActionType.View
)
.map(_.toBoolean)
def listStacExports: Route =
authenticate { user =>
authorizeScope(
ScopedAction(Domain.StacExports, Action.Read, None),
user
) {
(withPagination & stacExportQueryParameters) {
(page: PageRequest, params: StacExportQueryParameters) =>
complete {
StacExportDao
.list(page, params, user)
.map(
p =>
PaginatedResponse[StacExport.WithSignedDownload](
p.count,
p.hasPrevious,
p.hasNext,
p.page,
p.pageSize,
p.results.map(signExportUrl(_))
))
.transact(xa)
.unsafeToFuture
}
case StacExport.CampaignExport(_, _, _, campaignId) =>
authorizeAsync {
CampaignDao
.authorized(
user,
ObjectType.Campaign,
campaignId,
ActionType.View
)
.map(_.toBoolean)
}
}
}

def createStacExport: Route =
authenticate { user =>
authorizeScope(
ScopedAction(Domain.StacExports, Action.Create, None),
user
) {
entity(as[StacExport.Create]) { newStacExport =>
newStacExport match {
case StacExport
.AnnotationProjectExport(_, _, _, annotationProjectId) =>
authorizeAsync {
AnnotationProjectDao
.authorized(
user,
ObjectType.AnnotationProject,
annotationProjectId,
ActionType.View
)
.map(_.toBoolean)
.transact(xa)
.unsafeToFuture
}
case StacExport.CampaignExport(_, _, _, _, campaignId) =>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the only line I changed in this file, FYI. The rest is wacky formatting changes from scalafmt.

authorizeAsync {
CampaignDao
.authorized(
user,
ObjectType.Campaign,
campaignId,
ActionType.View
)
.map(_.toBoolean)
.transact(xa)
.unsafeToFuture
}
}
{
onSuccess(
StacExportDao
.create(newStacExport, user)
.transact(xa)
.unsafeToFuture
) { stacExport =>
kickoffStacExport(stacExport.id)
complete((StatusCodes.Created, stacExport))
}
}
{
onSuccess(
StacExportDao
.create(newStacExport, user)
.transact(xa)
.unsafeToFuture
) { stacExport =>
kickoffStacExport(stacExport.id)
complete((StatusCodes.Created, stacExport))
}
}
}
}
}

def getStacExport(id: UUID): Route = authenticate { user =>
authorizeScope(ScopedAction(Domain.StacExports, Action.Read, None), user) {
authorizeAsync {
StacExportDao
.isOwnerOrSuperUser(user, id)
.transact(xa)
.unsafeToFuture
} {
rejectEmptyResponse {
complete {
StacExportDao
.getById(id)
.map {
case Some(export) =>
Some(
signExportUrl(export)
)
case _ => None
}
.transact(xa)
.unsafeToFuture
def getStacExport(id: UUID): Route =
authenticate { user =>
authorizeScope(
ScopedAction(Domain.StacExports, Action.Read, None),
user
) {
authorizeAsync {
StacExportDao
.isOwnerOrSuperUser(user, id)
.transact(xa)
.unsafeToFuture
} {
rejectEmptyResponse {
complete {
StacExportDao
.getById(id)
.map {
case Some(export) =>
Some(
signExportUrl(export)
)
case _ => None
}
.transact(xa)
.unsafeToFuture
}
}
}
}
}
}

def deleteStacExport(id: UUID): Route = authenticate { user =>
authorizeScope(ScopedAction(Domain.StacExports, Action.Delete, None), user) {
authorizeAsync {
StacExportDao
.isOwnerOrSuperUser(user, id)
.transact(xa)
.unsafeToFuture
} {
onSuccess(
def deleteStacExport(id: UUID): Route =
authenticate { user =>
authorizeScope(
ScopedAction(Domain.StacExports, Action.Delete, None),
user
) {
authorizeAsync {
StacExportDao
.delete(id)
.isOwnerOrSuperUser(user, id)
.transact(xa)
.unsafeToFuture
) { count: Int =>
complete((StatusCodes.NoContent, s"$count stac export deleted"))
} {
onSuccess(
StacExportDao
.delete(id)
.transact(xa)
.unsafeToFuture
) { count: Int =>
complete((StatusCodes.NoContent, s"$count stac export deleted"))
}
}
}
}
}
}
94 changes: 0 additions & 94 deletions app-backend/batch/src/main/scala/stacExport/DatabaseIO.scala

This file was deleted.

20 changes: 0 additions & 20 deletions app-backend/batch/src/main/scala/stacExport/ExportData.scala

This file was deleted.

Loading