Supplying a large number to a distributed object store weighting consumes large amounts of ram #6988
Labels
Comments
hexylena
added
the
help wanted
|
apparently in stdlib as soon as we're py3 only :) https://docs.python.org/3.7/library/random.html#random.choices |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Say you're configuring your distributed object_store_conf.xml file and you want to be really sure that data only goes to one backend and you don't trust your eyeballs that the other ones say
weight="0". So you fill in a very large number likeweight="1000000000".Unfortunately this creates that many duplicates of the object, consuming ~30+gb of ram very quickly.
This is an easy "bug" to fix and would be a good beginner bug. Just need to refactor https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/objectstore/__init__.py#L615-L619 to use a weighted random choice rather than building an array of all backends + randomly choosing from that.
The text was updated successfully, but these errors were encountered: