feat: add benchmarks_entrypoint.py
#34495
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
McPatate
force-pushed
the
feat/allow_new_benchmarks
branch
2 times, most recently
from
41487ff to
49dcf70
Compare
| @@ -82,7 +40,7 @@ def collect_metrics(benchmark_id, continue_metric_collection): | |||
| conn.close() | |||
|
|
|||
|
|
|||
| def run_benchmark(branch: str, commit_id: str, commit_msg: str, num_tokens_to_generate=100): | |||
| def run_benchmark(logger: Logger, branch: str, commit_id: str, commit_msg: str, num_tokens_to_generate=100): | |||
There was a problem hiding this comment.
is there a way to only pass the logger and a class that wrapps around your continue_metric_collection .
This way the only thing people need when adding a new file with run_benchmark is:
- Call start() on the class
- compute some stuff
- call .record() (for example) which take the dict of inputs, and adds them to the data, commit and close!
completely up to debate, try to find the simplest way to do it! 🤗
There was a problem hiding this comment.
Yes that's probably the best path forward! Will iterate and let you know when I have something nicer
There was a problem hiding this comment.
lmk if the changes look good to you
McPatate
force-pushed
the
feat/allow_new_benchmarks
branch
from
f7d0286 to
796affb
Compare
ArthurZucker
approved these changes
Nov 5, 2024
|
I'll update the README and then we can merge! |
McPatate
force-pushed
the
feat/allow_new_benchmarks
branch
from
3190b19 to
9c493ff
Compare
Adding `benchmarks_entrypoint.py` file, which will be run from the benchmarks CI. This python script will list all python files from the `benchmark/` folder and run the included `run_benchmark` function, allowing people to add new benchmarks scripts.
McPatate
force-pushed
the
feat/allow_new_benchmarks
branch
from
afa207f to
c2d4b82
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding
benchmarks_entrypoint.pyfile, which will be run from the benchmarks CI.This python script will list all python files from the
benchmark/folder and run the includedrun_benchmarkfunction, allowing people to add new benchmarks scripts.