Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a x perf command for integrating bootstrap with rustc-perf #126318

Merged
merged 5 commits into from
Jun 22, 2024

Conversation

Kobzol
Copy link
Contributor

@Kobzol Kobzol commented Jun 12, 2024

This PR adds a new x perf command to bootstrap. The idea is to let rustc developers profile (profile_local) and benchmark (bench_local) a stage1/stage2 compiler directly from within rust.

Before, if you wanted to use rustc-perf, you had to clone it, set it up, copy the rustc sysroot after every change to rust etc. This is an attempt to automate that.

I opened this PR mostly for discussion. My idea is to offer an interface that looks something like this (a random sample of commands):

x perf --stage 2 profile eprintln
x perf --stage1 profile cachegrind
x perf benchmark --id baseline
x perf benchmark --id after-edit
x perf cmp baseline after-edit

In this PR, I'd like to only implement the simplest case (profile_local (eprintln)), because that only requires a single sysroot (you don't compare anything), and it's relatively easy to set up. Also, I'd like to avoid forcing developers to deal with the rustc-perf UI, so more complex use-cases (like benchmarking two sysroots and comparing the results) should probably wait for rust-lang/rustc-perf#1734 (which is hopefully coming along soon-ish).

I'm not sure if it's better to do this in bootstrap directly, or if I should create some shim tool that will receive a rustc sysroot, and offer a simplified CLI on top of rustc-perf.

Why is a separate CLI needed?

We definitely need to add some support to bootstrap to automate preparing rustc-perf and the rustc sysroot, but in theory after that we could just let people invoke rustc-perf manually. While that is definitely possible, you'd need to manually figure out where is your sysroot located, which seems annoying to me. The rustc-perf CLI is also relatively complex, and for this use-case it makes sense to only use a subset of it. So I thought that it would be better to offer a simplified interface on top of it that would make life easier for contributors. But maybe it's not worth it.

CC @onur-ozkan

@rustbot
Copy link
Collaborator

rustbot commented Jun 12, 2024

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

@rustbot
Copy link
Collaborator

rustbot commented Jun 12, 2024

⚠️ Warning ⚠️

  • These commits modify submodules.

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 12, 2024
@Kobzol
Copy link
Contributor Author

Kobzol commented Jun 12, 2024

Damn, forgot to set r? ghost. Well, then:

r? @onur-ozkan

@onur-ozkan
Copy link
Member

Based on #126306 and #126312.

#126312 is already in the bors queue, you can revert that commit from this PR.

@Kobzol Kobzol marked this pull request as ready for review June 12, 2024 19:06
@rustbot
Copy link
Collaborator

rustbot commented Jun 12, 2024

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

@Kobzol
Copy link
Contributor Author

Kobzol commented Jun 12, 2024

@rustbot ready

@Kobzol
Copy link
Contributor Author

Kobzol commented Jun 12, 2024

To clarify: this PR now hardcodes a single command (profile_local eprintln). I'd like to add more commands, but I'm not sure yet which approach to use to prepare the corresponding CLI.

@Kobzol
Copy link
Contributor Author

Kobzol commented Jun 14, 2024

I realized that having the CLI as a separate tool probably won't work (x run), because we would need to pass it paths from bootstrap, and I'm not sure if x run is prepared for that. So probably having the CLI directly in bootstrap is the easier solution.

@onur-ozkan
Copy link
Member

I realized that having the CLI as a separate tool probably won't work (x run), because we would need to pass it paths from bootstrap, and I'm not sure if x run is prepared for that. So probably having the CLI directly in bootstrap is the easier solution.

The PR title says x perf. I thought we already implementing this directly in bootstrap without involving x run.

@Kobzol
Copy link
Contributor Author

Kobzol commented Jun 14, 2024

Yeah, I started with this implementation, since it seemed more natural, but as the PR description said, it was up for discussion :) But I think that indeed x perf is the better option.

Copy link
Member

@onur-ozkan onur-ozkan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unsure how the rustc-perf tool works as I haven't used it locally. When I run x perf, I expect it to work, but it only creates empty files currently:

  $ echo -n | sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -
  $ sha256sum build/rustc-perf/*
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Check-Full
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Check-IncrFull
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Check-IncrPatched0
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Check-IncrUnchanged
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Debug-Full
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Debug-IncrFull
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Debug-IncrPatched0
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Debug-IncrUnchanged
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Opt-Full
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Opt-IncrFull
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Opt-IncrPatched0
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-Opt-IncrUnchanged
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  build/rustc-perf/eprintln-Id-helloworld-tiny-Opt-Full

src/bootstrap/src/core/build_steps/tool.rs Outdated Show resolved Hide resolved
src/bootstrap/src/core/build_steps/perf.rs Outdated Show resolved Hide resolved
@onur-ozkan onur-ozkan added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 22, 2024
@Kobzol
Copy link
Contributor Author

Kobzol commented Jun 22, 2024

I'm unsure how the rustc-perf tool works as I haven't used it locally. When I run x perf, I expect it to work, but it only creates empty files currently:

This PR currently uses the eprintln profiler, which is the simplest one. It simply runs the compiler and stores all stderr into a file. Since the compiler didn't print anything to stderr, it is empty. Normally, you would modify the compiler, add some eprintln calls, then run the benchmark and check if some codepaths were taken, or how many times they were taken. This can be then combined with tools like counts to gather statistics. It is described here (you can search for eprintln).

@rustbot ready

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 22, 2024
@onur-ozkan
Copy link
Member

@bors r+

@bors
Copy link
Contributor

bors commented Jun 22, 2024

📌 Commit 0bd58d8 has been approved by onur-ozkan

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 22, 2024
bors added a commit to rust-lang-ci/rust that referenced this pull request Jun 22, 2024
…iaskrgr

Rollup of 3 pull requests

Successful merges:

 - rust-lang#126140 (Rename `std::fs::try_exists` to  `std::fs::exists` and stabilize fs_try_exists)
 - rust-lang#126318 (Add a `x perf` command for integrating bootstrap with `rustc-perf`)
 - rust-lang#126552 (Remove use of const traits (and `feature(effects)`) from stdlib)

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 162120b into rust-lang:master Jun 22, 2024
6 checks passed
@rustbot rustbot added this to the 1.81.0 milestone Jun 22, 2024
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Jun 22, 2024
Rollup merge of rust-lang#126318 - Kobzol:bootstrap-perf, r=onur-ozkan

Add a `x perf` command for integrating bootstrap with `rustc-perf`

This PR adds a new `x perf` command to bootstrap. The idea is to let rustc developers profile (`profile_local`) and benchmark (`bench_local`) a stage1/stage2 compiler directly from within `rust`.

Before, if you wanted to use `rustc-perf`, you had to clone it, set it up, copy the `rustc` sysroot after every change to `rust` etc. This is an attempt to automate that.

I opened this PR mostly for discussion. My idea is to offer an interface that looks something like this (a random sample of commands):
```bash
x perf --stage 2 profile eprintln
x perf --stage1 profile cachegrind
x perf benchmark --id baseline
x perf benchmark --id after-edit
x perf cmp baseline after-edit
```

In this PR, I'd like to only implement the simplest case (`profile_local (eprintln)`), because that only requires a single sysroot (you don't compare anything), and it's relatively easy to set up. Also, I'd like to avoid forcing developers to deal with the rustc-perf UI, so more complex use-cases (like benchmarking two sysroots and comparing the results) should probably wait for rust-lang/rustc-perf#1734 (which is hopefully coming along soon-ish).

I'm not sure if it's better to do this in bootstrap directly, or if I should create some shim tool that will receive a `rustc` sysroot, and offer a simplified CLI on top of `rustc-perf`.

## Why is a separate CLI needed?
We definitely need to add some support to bootstrap to automate preparing `rustc-perf` and the `rustc` sysroot, but in theory after that we could just let people invoke `rustc-perf` manually. While that is definitely possible, you'd need to manually figure out where is your sysroot located, which seems annoying to me. The `rustc-perf` CLI is also relatively complex, and for this use-case it makes sense to only use a subset of it. So I thought that it would be better to offer a simplified interface on top of it that would make life easier for contributors. But maybe it's not worth it.

CC `@onur-ozkan`
@Kobzol Kobzol deleted the bootstrap-perf branch June 22, 2024 20:56
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jun 29, 2024
…-ozkan

Implement `x perf` as a separate tool

Continues work from rust-lang#126318, adds a CLI for running `rustc-perf` profiling commands through a new `rustc-perf-wrapper` tool. The CLI is in a separate tool to enable experimentation outside of `bootstrap`.

This is probably most of what we can do so far, I'll add support for benchmarking once `rustc-perf` gets a terminal output for comparing benchmark results.

r? `@onur-ozkan`
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jun 29, 2024
…-ozkan

Implement `x perf` as a separate tool

Continues work from rust-lang#126318, adds a CLI for running `rustc-perf` profiling commands through a new `rustc-perf-wrapper` tool. The CLI is in a separate tool to enable experimentation outside of `bootstrap`.

This is probably most of what we can do so far, I'll add support for benchmarking once `rustc-perf` gets a terminal output for comparing benchmark results.

r? ``@onur-ozkan``
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Jun 30, 2024
Rollup merge of rust-lang#127002 - Kobzol:bootstrap-perf-tool, r=onur-ozkan

Implement `x perf` as a separate tool

Continues work from rust-lang#126318, adds a CLI for running `rustc-perf` profiling commands through a new `rustc-perf-wrapper` tool. The CLI is in a separate tool to enable experimentation outside of `bootstrap`.

This is probably most of what we can do so far, I'll add support for benchmarking once `rustc-perf` gets a terminal output for comparing benchmark results.

r? ``@onur-ozkan``
jaisnan pushed a commit to jaisnan/rust-dev that referenced this pull request Jul 29, 2024
Update Rust toolchain from nightly-2024-06-22 to nightly-2024-06-23
without any other source changes.
This is an automatically generated pull request. If any of the CI checks
fail, manual intervention is required. In such a case, review the
changes at https://github.com/rust-lang/rust from
rust-lang@c1b336c
up to
rust-lang@3cb521a.
The log for this commit range is:
rust-lang@3cb521a434 Auto merge of
rust-lang#126761 - GuillaumeGomez:unsafe_extern_blocks, r=spastorino
rust-lang@a0f01c3c10 Auto merge of
rust-lang#126838 - matthiaskrgr:rollup-qkop22o, r=matthiaskrgr
rust-lang@dc9a08f535 Rollup merge of
rust-lang#126552 - fee1-dead-contrib:rmfx, r=compiler-errors
rust-lang@162120b4fa Rollup merge of
rust-lang#126318 - Kobzol:bootstrap-perf, r=onur-ozkan
rust-lang@f3ced9d540 Rollup merge of
rust-lang#126140 - eduardosm:stabilize-fs_try_exists, r=Amanieu
rust-lang@f944afe380 Auto merge of
rust-lang#116113 - kpreid:arcmut, r=dtolnay
rust-lang@88c3db57e4 Generalize
`{Rc,Arc}::make_mut()` to unsized types.
rust-lang@a9a4830d25 Replace
`WriteCloneIntoRaw` with `CloneToUninit`.
rust-lang@ec201b8650 Add
`core::clone::CloneToUninit`.
rust-lang@81da6a6d40 Make `effects` an
incomplete feature
rust-lang@ac47dbad50 Auto merge of
rust-lang#126824 - GuillaumeGomez:rollup-sybv8o7, r=GuillaumeGomez
rust-lang@d265538016 Rollup merge of
rust-lang#126823 - GuillaumeGomez:migrate-run-make-inline-always-many-cgu,
r=Kobzol
rust-lang@25bcc7d130 Rollup merge of
rust-lang#126731 - Kobzol:bootstrap-cmd-refactor, r=onur-ozkan
rust-lang@399c5cabdd Rollup merge of
rust-lang#126723 - estebank:dot-dot-dot, r=Nadrieril
rust-lang@3ed2cd74b5 Rollup merge of
rust-lang#126686 - fmease:dump-preds-n-item-bounds, r=compiler-errors
rust-lang@07e8b3ac01 Rollup merge of
rust-lang#126555 - beetrees:f16-inline-asm-arm, r=Amanieu
rust-lang@d03d6c0fea Auto merge of
rust-lang#126750 - scottmcm:less-unlikely, r=jhpratt
rust-lang@e7dfd4a913 Migrate
`run-make/inline-always-many-cgu` to `rmake.rs`
rust-lang@d9962bb4d8 Make `read_dir`
method take a mutable callback
rust-lang@f1b0d54ca9 Auto merge of
rust-lang#126816 - weihanglo:update-cargo, r=weihanglo
rust-lang@0bd58d8122 Apply review
comments.
rust-lang@250586cb2e Wrap std `Output` in
`CommandOutput`
rust-lang@f0aceed540 Auto merge of
rust-lang#126817 - workingjubilee:rollup-0rg0k55, r=workingjubilee
rust-lang@38bd7a0fcb Add
`#[rustc_dump_{predicates,item_bounds}]`
rust-lang@1916b3d57f Rollup merge of
rust-lang#126811 - compiler-errors:tidy-ftl, r=estebank
rust-lang@539090e5cd Rollup merge of
rust-lang#126809 - estebank:wording-tweak, r=oli-obk
rust-lang@b9ab6c3501 Rollup merge of
rust-lang#126798 - miguelfrde:master, r=tmandry
rust-lang@9498d5cf2f Rollup merge of
rust-lang#126787 - Strophox:get-bytes, r=RalfJung
rust-lang@1f9793f1aa Rollup merge of
rust-lang#126722 - adwinwhite:ptr_fn_abi, r=celinval
rust-lang@84b0922565 Rollup merge of
rust-lang#126712 - Oneirical:bootest-constestllation, r=jieyouxu
rust-lang@e7956cd994 Rollup merge of
rust-lang#126530 - beetrees:f16-inline-asm-riscv, r=Amanieu
rust-lang@10e1f5d212 Auto merge of
rust-lang#124101 - the8472:pidfd-methods, r=cuviper
rust-lang@2c65a24b8c Update cargo
rust-lang@fcae62649e Auto merge of
rust-lang#126758 - spastorino:avoid-safe-outside-unsafe-blocks, r=compiler-errors
rust-lang@ffd72b1700 Fix remaining cases
rust-lang@ea681ef281 Add a tidy rule to
make sure that diagnostics don't end in periods
rust-lang@8abf149bde to extract a pidfd
we must consume the child
rust-lang@0787c7308c Add PidFd::{kill,
wait, try_wait}
rust-lang@5d5892e966 Remove stray `.`
from error message
rust-lang@d94a40516e
[fuchsia-test-runner] Remove usage of kw_only
rust-lang@771e44ebd3 Add `f16` inline ASM
support for RISC-V
rust-lang@753fb070bb Add `f16` inline ASM
support for 32-bit ARM
rust-lang@22831ed117 Do not allow safe
usafe on static and fn items
rust-lang@a6a83d3d4e bless tests
rust-lang@b512bf6f77 add as_ptr to trait
AllocBytes, fix 2 impls; add pub fn get_bytes_unchecked_raw in
allocation.rs; add pub fn get_alloc_bytes_unchecked_raw[_mut] in
memory.rs
rust-lang@02aaea1803 update intrinsic
const param counting
rust-lang@3b14b756d8 Remove
`feature(effects)` from the standard library
rust-lang@a314f7363a Stop using
`unlikely` in `strict_*` methods
rust-lang@225796a2df Add method to get
`FnAbi` of function pointer
rust-lang@630c3adb14 Add regression test
for `unsafe_extern_blocks`
rust-lang@bb9a3ef90c Implement
`unsafe_extern_blocks` feature in rustdoc
rust-lang@3c0a4bc915 rewrite
crate-name-priority to rmake
rust-lang@bc12972bcd Slightly refactor
the dumping of HIR analysis data
rust-lang@3fe4d134dd Appease `clippy`
rust-lang@c15293407f Remove unused import
rust-lang@5c4318d02c Implement `run_cmd`
in terms of `run_tracked`
rust-lang@0de7b92cc6 Remove
`run_delaying_failure`
rust-lang@e933cfb13c Remove
`run_quiet_delaying_failure`
rust-lang@949e667d3f Remove `run_quiet`
rust-lang@a12f541a18 Implement new
command execution logic
rust-lang@9fd7784b97 Fix `...` in
multline code-skips in suggestions
rust-lang@f22b5afa6a rewrite
error-writing-dependencies to rmake
rust-lang@75ee1d74a9 rewrite
relocation-model to rmake
rust-lang@87d2e61428 Add `x perf` command
for profiling the compiler using `rustc-perf`
rust-lang@fd44aca2aa Copy `rustc-fake`
binary when building the `rustc-perf` tool
rust-lang@9e0b76201b Add `RustcPerf`
bootstrap tool
rust-lang@9ec178df0b Add `cargo_args` to
`ToolBuild`
rust-lang@6a04dfe78c Rename
`std::fs::try_exists` to `std::fs::exists` and stabilize fs_try_exists

Co-authored-by: celinval <35149715+celinval@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants