feat: add the %{jobs} variable

[MisterDA]

%{jobs} is the maximum number of concurrent jobs Dune has been allowed to have (see https://dune.readthedocs.io/en/latest/reference/config/jobs.html). An initial value of auto for the config variable expands to the auto-detected number of cores.
See Dune, cmake and -j on Discuss (cc @tbrk).

A grep for %{jobs} returned a few matches, that are direct translations of opam jobs variable. I don't understand whether it was actually supported somehow or not, it seems not, but maybe I've missed something.

test/blackbox-tests/test-cases/pkg/opam-var/opam-var-global.t
23:       (run echo %{jobs})

test/blackbox-tests/test-cases/pkg/partial-filter-evaluation.t
43:        %{jobs}
77:      (run dune build -p %{pkg-self:name} -j %{jobs} --foobar @install @doc)))))

test/blackbox-tests/test-cases/pkg/extra-sources.t
122:       (run dune build -p %{pkg-self:name} -j %{jobs}))))))

test/blackbox-tests/test-cases/pkg/convert-opam-commands.t
29:  >   [make "-j%{jobs}%"]
107:       (run dune build -p %{pkg-self:name} -j %{jobs} @install))))))
121:       (run %{make} -j%{jobs}))))))

I wrote on Discuss:

Actually, when I use Dune’s foreign build sandboxing, it happens often that Dune waits for the build of the foreign library to finish, with just one job in flight. In that case, the foreign build system could freely use all of the possible cores, since Dune doesn’t execute any job in parallel with the foreign build system.

It's just important not to run an unbounded number of jobs.

[rgrinberg]

Could you modify this test to censor out the resulting j value? Otherwise this test will fail anywhere -j isn't set to 16.

[MisterDA]

yeah this is a bit silly ;)

[rgrinberg]

Just a heads up: this will feature make it so that your make rule will pointlessly re-run whenever you change the number of jobs. If you're fine with that, then it's fine. But we should really not be recommending this pattern.

[MisterDA]

Just a heads up: this will feature make it so that your make rule will pointlessly re-run whenever you change the number of jobs. If you're fine with that, then it's fine.

Hmm, I haven't thought of that, thanks.
Changing the number of jobs will re-trigger a make run, but it won't invalidate the make run, right? If the foreign build system run is idempotent, there's just the cost of an empty run?
I don't see why the number of CPUs would change in this context.

But we should really not be recommending this pattern.

Ok, I'll add a caveat post to the Discuss thread. Do you see a tweak (besides implementing the whole GNU jobserver) so that changing the number of jobs wouldn't re-trigger the rule? A weak reference, of sorts? I suppose setting an env var will also re-trigger the rules?

[rgrinberg]

Changing the number of jobs will re-trigger a make run, but it won't invalidate the make run, right? If the foreign build system run is idempotent, there's just the cost of an empty run?
I don't see why the number of CPUs would change in this context.

It depends if the build is sandboxed. If it is, make will not have access to its previous build artifacts will do a clean build. Sandboxing of course is much safer and in general all non trivial and manually written makefiles do not specify their dependencies correctly.

Do you see a tweak (besides implementing the whole GNU jobserver) so that changing the number of jobs wouldn't re-trigger the rule? A weak reference, of sorts? I suppose setting an env var will also re-trigger the rules?

The alternative are all fundamentally unsafe and rely on actions being co-operative. One alternative might be dune export its j value in some sort of environment variable that is visible to actions. That will not affect the digest of the action.

[MisterDA]

It depends if the build is sandboxed. If it is, make will not have access to its previous build artifacts will do a clean build. Sandboxing of course is much safer

Thanks for the explanations!

in general all non trivial and manually written makefiles do not specify their dependencies correctly.

True.

One alternative might be dune export its j value in some sort of environment variable that is visible to actions.

Both GNU Make and BSD Make recognize the MAKEFLAGS environment variable. Dune could recognize %{make} and add -jN to the MAKEFFLAGS it passes to the Make process. I'd like to propose this, but I also know I've added the .NOTPARALLEL target (available in BSD Make and GNU Make) to at least 3 different projects this month to disable parallel Make builds in projects that don't support it. I usually set export MAKEFLAGS=-j$(nproc) in my shell's rc.

Another way to alleviate the problem would be for the foreign build system to auto-detect the number of cores; Ninja (used as a backend for Meson, and an optional backend for CMake) does this.

I can document all of that regardless of whether we accept the %{jobs} variable. If we do accept it, I can also document the caveat that changing the number of jobs will invalidate a sandboxed build.