Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GDALRasterSources performance #183

Open
pomadchin opened this issue May 30, 2019 · 3 comments
Open

GDALRasterSources performance #183

pomadchin opened this issue May 30, 2019 · 3 comments
Milestone
GT 3.0

Comments

@pomadchin
Copy link
Member

pomadchin commented May 30, 2019 •
edited
Loading

During the RasterSources benchmark https://github.com/pomadchin/vlm-performance/ on EMR we received no performance improvements with GDAL usage. On the contrary, it created more complexity in jobs configuration. Without tuning GDAL_CAHEMAX options and wo (for warp operations), EMR nodes are dying / hanging and => tasks are failing. This is also can be the problem of the GDAL 2.3 usage on yarn and according to OSGeo/gdal@5351335 GDAL doesn't respect cgroups. But it is unclear how yarn limits resources, it may be not using cgroups to limit CPU / RAM and this can be useless in this case.

Related to https://github.com/azavea/raster-foundry-platform/issues/743
and to https://github.com/azavea/geotrellis/issues/68
@pomadchin pomadchin added this to the GT 3.0 milestone May 30, 2019
@rossbernet
Copy link

@notthatbreezy
@jcahail
@echeipesh

We talked about this yesterday in cross team planning.

But we wanted to confirm the expected outcome, timeline, and priority for RF.

Does it make sense to wait until https://github.com/azavea/raster-foundry-platform/issues/743 would be resolved, because this has the potential to fix the RF problems.

It's an option to spend at least two points this sprint, but we confirm with you all.

@notthatbreezy
Copy link

There are a few issues being worked on that probably need to be resolved for us to get more information to make working on GDAL related performance issues easier as far as RF goes.

I think at least the first two need to be done before we can start looking into garbage collection more otherwise we won't really have enough information to determine if we're improving things.

@pomadchin
Copy link
Member Author

pomadchin commented Jun 4, 2019 •
edited
Loading

RPMs for the EMR test are here: s3://geotrellis-test/rpms/v1.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
GT 3.0
Development

No branches or pull requests
3 participants
@notthatbreezy @pomadchin @rossbernet