build-std: always

[davidtwco]

Add a new Cargo configuration option, build-std = "always|never", which will unconditionally rebuild standard library dependencies. The set of standard library dependencies can optionally be customised with a new build-std-crates option. It also describes how Cargo (or external tools) should build the standard library crates on stable (i.e., which flags to pass and features to enable).

This proposal limits the ways the built standard library can be customised (such as by settings in the profile) and intends that the build standard library matches the prebuilt one (if available) as closely as possible.

This RFC is is part of the build-std project goal and a series of build-std RFCs:

1. build-std context (build-std: context #3873)
2. build-std="always" (this RFC)
3. Explicit standard library dependencies (build-std: explicit dependencies #3875)
4. build-std="compatible" (RFC not opened yet)
5. build-std="match-profile" (RFC not opened yet)

There is also a Zulip channel where you can ask questions about any of the build-std RFCs. This series of RFCs was drafted over many months with the help of stakeholders from many Rust project teams, we thank them for their help!

Rendered

[epage]

This has moved to a more official place

Suggested change

	[Opaque dependencies]: https://hackmd.io/@epage/ByGfPtRell
	[Opaque dependencies]: https://github.com/rust-lang/cargo/issues/3573#issuecomment-3498262549

[epage]

@nicoburns brought up on reddit the idea of cargo clean not affecting opaque dependencies, requiring some other step to clean them (maybe a cargo clean -p std). Alternatively, we may get this "for free" with rust-lang/cargo#5931.

I lean towards keeping cargo clean the same for now but might be worth calling this question out somewhere.

[Skgland]

crates being plural but only taking a single string argument rather than a list/table reads a bit odd. Not sure what to do about that given that only specifying the "max" crate to be build and implying dependencies sounds reasonable.

[clarfonthey]

Maybe build-std.up-to = "std"? Since crates = "std" is the default, it makes a lot of sense for people to say, for example, build-std = { when = "always", up-to = "alloc" }.

[ehuss]

As a heads up, we are planning to start an FCP proposal on this RFC shortly after #3873 (build-std context) has been approved.

[bjorn3]

I went through all build scripts in dependencies we have. The libc crate detects both the rustc version and if it is a nightly version. It seems like it doesn't actually use the latter for anything though. The object crate also detects rustc version. Nothing else seems to be detecting the rustc version that I could see.

[bjorn3]

Until external-mem automatically gets enabled for all targets with a libc implementation, this inversion of the mem feature would cause a silent performance hit for those targets, while previously you would get a loud failure, which IMO is preferable.

[bjorn3]

--sysroot /dev/null is already effectively that except for that also blocking discovery of self-contained artifacts.

[bjorn3]

I fully agree we should be vendoring the standard library dependencies. Actually implementing that is non-trivial however: #t-cargo > vendoring for -Zbuild-std Setting the vendoring using .cargo/config.toml doesn't work and I couldn't figure out how to make -Zbuild-std respect it. Next thing I want to try is patching library/Cargo.toml to add a [patch.crates-io] section. Perhaps the standard library workspace is still considered enough of a workspace by -Zbuild-std for that to work.

[bjorn3]

An alternative design would be to have a rustc flag which makes it build all requested standard library crates and put them in a specified directory. And then another flag to load standard library crates from said directory when compiling user code. Rustc would then be responsible for all orchestration and we wouldn't require every build system to implement a Cargo.toml parser and build.rs output parser. Said parser would need to be updated every time we use a new cargo feature in the standard library, which would mean we can't use any new cargo features ever without breaking some build system if said build system is responsible for orchestrating the compilation of the standard library.

Handling this all inside rustc itself would allow much more freedom for us (both using newer cargo features and reorganizing the standard library) and be easier for alternative build systems. If we don't want all this logic to be inside rustc itself, we could also put the build logic in a separate rustc-sysroot-build binary (which could for example use cargo as library or binary to build the standard library as regular cargo project and then copy out the libraries into the output directory) and only keep the flag to use the built standard library inside rustc, with regular cargo invoking rustc-sysroot-build first.

[bjorn3]

To me the fact that profiler-builtins is even a crate at all rather than a C static library is a hack to enable building the profiler-rt C library within the context of a regular cargo build of the standard library. If we give up on supporting that, we should just get rid of profiler-builtins as a crate entirely and have rustc directly load a .a file. That would also simplify some logic inside rustc I think. Currently we have to inject a dependency on profiler-builtins quite early already rather than delaying this to the linker code.

[bjorn3]

I'm pretty sure -Cprefer-dynamic would fallback to static linking rather than failing compilation.