-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stream the adhoc query results #2429
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gz
force-pushed
the
streaming-yo
branch
2 times, most recently
from
September 5, 2024 09:39
22339bc
to
ee2153a
Compare
ryzhyk
approved these changes
Sep 6, 2024
gz
force-pushed
the
streaming-yo
branch
3 times, most recently
from
September 7, 2024 08:56
5e8f5d2
to
1487edf
Compare
This prevents the manager from crashing on machines with many cores. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
If an adhoc query is sent it will be handled in the same tokio runtime that is running the webserver. This raises the questions of how many cores the runtime should have. Datafusion will need one task for each partition we're reading from. And we have as many partitions as dbsp workers. So maybe we should've just change HTTP_NUM_WORKERS from 4 to `config.global.runtime_workers`. Except, this doesn't quite help because if your query is very expensive like `select * from table_with_100m`, all it's task run for quite some time and starve out the HTTP request. So now the UI no longer updates. Ok, so maybe it should just be `workers*2`, but if workers is set near the amount of CPU cores on the machine then we'll overprovision the tokio runtime and that will slow things down too. So for now we just set this to #cores on the machine by not configuring it (this is the default). Note aside, ideally we can just swap out the actix runtime with `dbsp::runtime::TOKIO` which is what we use everywhere else and this already uses the #cores in the runtime to run the tasks. However, this also doesn't work because the actix API has a stupid bug in the interface. I submitted a PR to fix this here: actix/actix-net#599 So for now, we'll just have two tokio runtimes, with Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Also add ability to abort request with Ctrl+C. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
As opposed to sending them only once everything is computed, which can take a long time. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.