Implementing BDN Perf pipeline #758
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…king. Also, a fix in ClusterBench for and renamed Lua Runner benchmarking
… build statement in YML
…vs previous BDN perf numbers
… push collisions with GitHub
…etimes is in 40%+ range of difference.
…ntinuous benchmarking and also changed to where script is only checking allocated values and let github-action-benchmark compare mean values
badrishc
reviewed
Oct 31, 2024
test/BDNPerfTests/ConfigFiles/CI_CONFIG_BDN_Benchmark_Cluster.ClusterOperations.json
Outdated
Show resolved
Hide resolved
badrishc
reviewed
Oct 31, 2024
test/BDNPerfTests/ConfigFiles/CI_CONFIG_BDN_Benchmark_Cluster.ClusterOperations.json
Outdated
Show resolved
Hide resolved
badrishc
reviewed
Oct 31, 2024
test/BDNPerfTests/ConfigFiles/CI_CONFIG_BDN_Benchmark_Operations.ObjectOperations.json
Outdated
Show resolved
Hide resolved
badrishc
reviewed
Oct 31, 2024
test/BDNPerfTests/ConfigFiles/CI_CONFIG_BDN_Benchmark_Lua.LuaScripts.json
Outdated
Show resolved
Hide resolved
badrishc
reviewed
Oct 31, 2024
test/BDNPerfTests/ConfigFiles/CI_CONFIG_BDN_Benchmark_Lua.LuaScripts.json
Outdated
Show resolved
Hide resolved
badrishc
reviewed
Oct 31, 2024
test/BDNPerfTests/ConfigFiles/CI_CONFIG_BDN_Benchmark_Operations.BasicOperations.json
Outdated
Show resolved
Hide resolved
badrishc
reviewed
Oct 31, 2024
have shorter names for values not have configuration or framework (moved to ps1 file) not have windows / linux differnce in expected Moved acceptable allocated range to script modified the script: to only load values for the specific test instead of looking for all values changes to match the changes in config (shorter names, etc)
…into darrenge/BDNPerf
…into darrenge/BDNPerf
…es not need to be updated when new BDNs are added.
|
This was a sizable push so code review again. Also - might want to NOT merge yet as I am looking at the feature where BDN Benchmark compares to a specific commit instead of previous commit |
…loy website if we want
…into darrenge/BDNPerf
…s we can raise it back. Also, removed darrenge from the alert CC line as no need for that.
…l will get summary data in the specific workflow run and still uploads data to charts (when pushing to main).
badrishc
reviewed
Dec 10, 2024
badrishc
approved these changes
Dec 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are three areas of this PR:
a) Runs individual bdn test based on the associated config file (in /config) directory
b) Compares "allocated" value found from the run to the expected value in config file and if it more than 10% from expected, it will pop a fail
a) Goes through the matrix of Linux, WIndows and runs each BDN test on those two (total of 10 runs)
b) Each run saves results file and error log to GH Actions artifacts (for debugging if needed)
c) Each run takes the BDN results (from the ps1 file run) and compares MEAN to previous and if changes by 50% or more (represented as 150% in yml file) then it will comment in commit
d) Each run has task that adds to the "Summary" part of the GH Action run (table that shows previous run to current run)
e) Each run adds to charts (data is stored in continuousbenchmarking branch in the website/static/charts folder of that branch)
a) Name of BDN filter it is running
b) Expected "ground truth" for Allocated values
b) Acceptable range % for allocated ("acceptableAllocatedRange": 10) before flags a fail