Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add search filtering for search extension #29

Merged
merged 10 commits into from
Dec 12, 2019

Conversation

notthatbreezy
Copy link

@notthatbreezy notthatbreezy commented Dec 5, 2019

Overview

This adds the STAC search API extension support for filtering based on query parameters and a POST request with a json body.

There are a few other changes as well.

The biggest change I think is the collapse of the subprojects into a single project. The thinking here is that this makes development a little easier and since any library-like functionality should be moved into stac4s we don't lose anything in portability or reduced artifact size. Reducing to a single project makes it easier to identify packages that should be removed if they are no longer necessary.

Everything is changed to use bloop and run commands through that - this aligns with our giter8 template at Azavea.

Additionally, the DAO from RF is imported here. At least until that functionality gets moved out into its own artifact the functionality is copied over here. Still works out nicely.

./scripts/test now does way more things - checking formatting, running scapegoat, etc.

Notes

Right now the tests confirm that the search service doesn't explode, but I think ideally we have some more specific tests to ensure that the tests function as we expect with some fixtures -- I opened an issue for that.

A few generators are copied over from stac4s - I opened an issue over there to publish those. Once that happens we can work on this issue.

Testing

  • Run ./scripts/setup
  • If you haven't imported data, import the Panama dataset
  • Start the server: ./scripts/server
  • Make some requests to verify that filtering works. Here are some examples:
http :9090/stac/search datetime==2019-10-01T18:37:11Z/2019-12-01T18:37:11Z limit==2
echo '{"datetime":"2019-10-01T18:37:11Z/2019-12-01T18:37:11Z","items":[],"collections":[]}' | http :9090/stac/search

Closes #3

@notthatbreezy notthatbreezy changed the title Add search filtering for search extension [WIP] Add search filtering for search extension Dec 6, 2019
@notthatbreezy notthatbreezy changed the title [WIP] Add search filtering for search extension Add search filtering for search extension Dec 9, 2019
}
}

implicit val searchFilter: Filterable[Any, SearchFilters] =
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converts SearchFilters to a fragment, this didn't turn out to be too bad

import io.circe.generic.semiauto._
import scala.util.Try

final case class Page(limit: Option[Int], next: Option[String]) {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Page is weird because the semantics on what next can be are pretty loose in the spec, in our case we interpret it as "page number", but we have to be careful for how we parse it, handle errors, and deal with defaults. In this case we don't raise if it's something we can't interpret, just assume that it's None.

Also, I don't know the best way to pass in a default limit, so for now 20 is a magical number.

scapegoatVersion in ThisBuild := Versions.ScapegoatVersion,
scalacOptions := Seq(
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced with sbt-tpolecat

build.sbt Show resolved Hide resolved
build.sbt Show resolved Hide resolved
docker-compose -f docker-compose.yml \
up api
echo -e "\e[32m[franklin] Starting API service\e[0m"
bloop run application -- serve --db-host=localhost
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switches to using bloop for everything here.

Neat thing I learned is that everything after -- gets interpreted as arguments to run which makes our usage of decline not a problem.


./sbt "; scalafix --check; test; docs/mdoc --check"
echo -e "\e[32m[franklin] Execute Scalafix\e[0m"
./sbt ";scalafix --check;scalafmtCheck;scalafmtSbtCheck;scapegoat;undeclaredCompileDependenciesTest;unusedCompileDependenciesTest;test"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing all the things

} else {
collection
}
collection.copy(
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The distinction between public/private was lost after upgrading gt server

Copy link
Contributor

@jisantuc jisantuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thing I'm most unsure about is the caching strategy. Other than that, everything that I'd have commented on you've opened issues for, so I think this is mostly good to go.

build.sbt Show resolved Hide resolved
project/Versions.scala Outdated Show resolved Hide resolved
.circleci/config.yml Outdated Show resolved Hide resolved
build.sbt Show resolved Hide resolved
project/Versions.scala Outdated Show resolved Hide resolved
@jisantuc
Copy link
Contributor

Oops forgot to mention -- also bloop stuff in the project plugins!

@notthatbreezy notthatbreezy mentioned this pull request Dec 11, 2019
@notthatbreezy notthatbreezy merged commit 06c7994 into master Dec 12, 2019
@notthatbreezy notthatbreezy deleted the feature/cmb/add-search-params branch December 12, 2019 17:56
@jcahail jcahail added this to the RF Sprint 11/27 - 12/12 milestone Dec 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add STAC Search Endpoints
3 participants