Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

benchmark: include go and grpc version in result files #3242

Merged
merged 2 commits into from
Dec 12, 2019
Merged

benchmark: include go and grpc version in result files #3242

merged 2 commits into from
Dec 12, 2019

Conversation

adtac
Copy link
Contributor

@adtac adtac commented Dec 10, 2019 •
edited
Loading

Closes #3235

@easwars
Copy link
Contributor

easwars commented Dec 11, 2019

Could you provide the output of running benchresult against benchmark outputs from different Go versions or gRPC versions. Thanks.

@adtac
Copy link
Contributor Author

adtac commented Dec 11, 2019 

 @adtac.svl.corp.google.com  ~/src/google.golang.org/grpc $ xvbench 5s 1 1 unary
go1.13.4/grpc1.26.0-dev
unary-networkMode_none-bufConn_false-keepalive_false-benchTime_5s-trace_false-latency_0s-kbps_0-MTU_0-maxConcurrentCalls_1-reqSize_1B-respSize_1B-compressor_off-channelz_false-preloader_false:
50_Latency: 150.3820µs	90_Latency: 186.2530µs	99_Latency: 274.7360µs	Avg_Latency: 155.3150µs	Bytes/op: 9602.819306390802	Allocs/op: 230.36014084068177	
Histogram (unit: µs)
Count: 32093  Min:  62.3  Max: 4898.9  Avg: 155.32
------------------------------------------------------------
[     62.253000,      62.254000)      1    0.0%    0.0%  
[     62.254000,      62.258530)      0    0.0%    0.0%  
[     62.258530,      62.283581)      0    0.0%    0.0%  
[     62.283581,      62.422114)      2    0.0%    0.0%  
[     62.422114,      63.188205)     20    0.1%    0.1%  
[     63.188205,      67.424699)     73    0.2%    0.3%  
[     67.424699,      90.852598)    213    0.7%    1.0%  
[     90.852598,     220.409334)  30900   96.3%   97.2%  ##########
[    220.409334,     936.860599)    876    2.7%  100.0%  
[    936.860599,    4898.850000)      7    0.0%  100.0%  
[   4898.850000,            inf)      1    0.0%  100.0%  
Number of requests:  32093	Request throughput:  51348.8 bit/s
Number of responses: 32093	Response throughput: 51348.8 bit/s

 @adtac.svl.corp.google.com  ~/src/google.golang.org/grpc $ cp /tmp/benchmain/bench-goversion /tmp/defgo
 @adtac.svl.corp.google.com  ~/src/google.golang.org/grpc $ xvcbench 5s 1 1 unary
devel +63546a03cc Wed Nov 6 21:40:43 2019 +0000/grpc1.26.0-dev
unary-networkMode_none-bufConn_false-keepalive_false-benchTime_5s-trace_false-latency_0s-kbps_0-MTU_0-maxConcurrentCalls_1-reqSize_1B-respSize_1B-compressor_off-channelz_false-preloader_false:
50_Latency: 134.9510µs	90_Latency: 166.0310µs	99_Latency: 256.3330µs	Avg_Latency: 139.6570µs	Bytes/op: 9394.65889278206	Allocs/op: 208.04838121934128	
Histogram (unit: µs)
Count: 35675  Min:  53.3  Max: 11981.8  Avg: 139.66
------------------------------------------------------------
[      53.252000,       53.253000)      1    0.0%    0.0%  
[      53.253000,       53.258113)      0    0.0%    0.0%  
[      53.258113,       53.289374)      0    0.0%    0.0%  
[      53.289374,       53.480488)      2    0.0%    0.0%  
[      53.480488,       54.648851)      2    0.0%    0.0%  
[      54.648851,       61.791601)     97    0.3%    0.3%  
[      61.791601,      105.458563)   2670    7.5%    7.8%  #
[     105.458563,      372.415060)  32720   91.7%   99.5%  #########
[     372.415060,     2004.444590)    180    0.5%  100.0%  
[    2004.444590,    11981.802000)      2    0.0%  100.0%  
[   11981.802000,             inf)      1    0.0%  100.0%  
Number of requests:  35675	Request throughput:  57080 bit/s
Number of responses: 35675	Response throughput: 57080 bit/s

 @adtac.svl.corp.google.com  ~/src/google.golang.org/grpc $ cp /tmp/benchmain/bench-goversion /tmp/defgo
 @adtac.svl.corp.google.com  ~/src/google.golang.org/grpc $ go run google.golang.org/grpc/benchmark/benchresult /tmp/benchmain/bench-goversion /tmp/defgo        
unary-networkMode_none-bufConn_false-keepalive_false-benchTime_5s-trace_false-latency_0s-kbps_0-MTU_0-maxConcurrentCalls_1-reqSize_1B-respSize_1B-compressor_off-channelz_false-preloader_false
               Title       Before        After Percentage
            TotalOps        35675        32093   -10.04%
             SendOps            0            0      NaN%
             RecvOps            0            0      NaN%
            Bytes/op      9394.66      9602.82     2.21%
           Allocs/op       208.05       230.36    10.57%
             ReqT/op     57080.00     51348.80   -10.04%
            RespT/op     57080.00     51348.80   -10.04%
            50th-Lat    134.951µs    150.382µs    11.43%
            90th-Lat    166.031µs    186.253µs    12.18%
            99th-Lat    256.333µs    274.736µs     7.18%
             Avg-Lat    139.657µs    155.315µs    11.21%

The first benchmark is default go1.13.4 compiler installed in my system. The second (devel +63546a03cc Wed Nov 6 21:40:43 2019 +0000) is a custom go version that I locally compiled -- this is loosely based on 1.13.4 with some experimental work that I did.

@adtac
Copy link
Contributor Author

adtac commented Dec 11, 2019

@easwars ^

also, xvbench and xvcbench are zsh aliases to go run google.golang.org/grpc/benchmark/benchmain/main.go with some parameters filled in.

@adtac
Copy link
Contributor Author

adtac commented Dec 11, 2019

I can run this across grpc versions too, if you like.

@easwars
Copy link
Contributor

easwars commented Dec 11, 2019

Thanks for the outputs.
If it's not a lot of work, could you also update benchresult to display the go-versions and gRPC versions of the After and Before in its output. This would be useful because most of the time, we just send the output of benchresult while showing the perf impact of any PR.

@adtac
Copy link
Contributor Author

adtac commented Dec 11, 2019

Done:

unary-networkMode_none-bufConn_false-keepalive_false-benchTime_5s-trace_false-latency_0s-kbps_0-MTU_0-maxConcurrentCalls_1-reqSize_1B-respSize_1B-compressor_off-channelz_false-preloader_false
               Title       Before        After Percentage
            TotalOps        32093        35675    11.16%
             SendOps            0            0      NaN%
             RecvOps            0            0      NaN%
            Bytes/op      9602.82      9394.66    -2.17%
           Allocs/op       230.36       208.05    -9.55%
             ReqT/op     51348.80     57080.00    11.16%
            RespT/op     51348.80     57080.00    11.16%
            50th-Lat    150.382µs    134.951µs   -10.26%
            90th-Lat    186.253µs    166.031µs   -10.86%
            99th-Lat    274.736µs    256.333µs    -6.70%
             Avg-Lat    155.315µs    139.657µs   -10.08%
           GoVersion     go1.13.4 devel +63546a03cc Wed Nov 6 21:40:43 2019 +0000
         GrpcVersion   1.26.0-dev   1.26.0-dev

the ugly, long devel... string would be much nicer when working with real go versions (like comparing the results across go1.11 and go1.13, for example -- I intend to do this later btw).

@easwars easwars added the Type: Internal Cleanup Refactors, etc label Dec 11, 2019
@easwars easwars added this to the 1.26 Release milestone Dec 11, 2019
@easwars easwars self-assigned this Dec 11, 2019
@dfawley
Copy link
Member

dfawley commented Dec 12, 2019

the ugly, long devel... string would be much nicer when working with real go versions (like comparing the results across go1.11 and go1.13, for example -- I intend to do this later btw).

Why does it only show one GoVersion in the output?

@dfawley
Copy link
Member

dfawley commented Dec 12, 2019

Why does it only show one GoVersion in the output?

@easwars has informed me that "devel..." is the "After" version. I was assuming "Go 1.13.4 devel..." was the "Before" version and there was no "After" version. LGTM.

@dfawley dfawley merged commit cb0e11b into grpc:master Dec 12, 2019
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 25, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@easwars easwars easwars approved these changes

Assignees

@easwars easwars

Labels
Type: Internal Cleanup Refactors, etc
Projects
None yet
Milestone
1.26 Release
Development

Successfully merging this pull request may close these issues.

benchmain: print and store go version
3 participants
@adtac @easwars @dfawley