refactoring: add dune_opam library for opam file parsing

[samoht]

This PR creates a new dune_opam library that sits between dune_lang and dune_pkg in the dependency hierarchy.

As discussed with @Alizter and @art-w, this patch extracts the opam build and opam install Dune rules into their own libraries so they can be reused, for instance by the scope stanza.

Apologies in advance for the code drop... I'll have limited availability in the following days, so please feel free to take over that PR if you think that's useful.

[rgrinberg]

Is this for the ability the ability to vendor opam packages inside inside dune workspaces?

[samoht]

yes -- and also to separate the "locking" rules from the "building" rules. They don't have to be so closely intertwined (they are separate features).

[rgrinberg]

I see. Off the top of my head, I don't see how it helps since the rules can already use dune_pkg. I also expect that this feature would work independently from the scope stanza so that more users could benefit from it. Meanwhile, all of these additional libraries are slowing down builds as we're starting to realize.

Let's keep this in mind while we're trying to make this feature work. I'd like to see more concrete benefits before we commit to this split.

Something that will definitely need to be split for this feature to work is some of the code in pkg_rules.ml. We need a set of functions that will build an opam package without looking it up in the lock directory.

[rgrinberg]

Here's an outline for how to implement this feature:

1. Introduce a stanza to tell dune that an opam package is to be built using its rules. E.g. something like (opam_package (dir foo))

2. Modify the package rules to look up such stanzas in addition to lock directories. We'll need a package type to distinguish the two types of packages I suppose

3. Build the new type of packages using the package rules but modified to read the information directly from the opam file instead of the lock directory and set the source to the stanza's dir field

[samoht]

In your mind, is this an incremental path to a point where opam packages that use Dune in their lock file are built with dune directly rather than via opam rules?

I'm still not very fond of mixing the notions of locks and opam packages. It's perfectly fine to lock dune packages (like opam-monorepo does). And in that case, it's really odd to have pkg_rules intertwining both opam rules and lock rules.

tbh, I'd really prefer if locks were just a light way to create a workspace that then uses normal dune rules (that know nothing about the lock). You could, for instance, always promote the opam-repo metadata to the package (ie, copy the opam file back in the source/build tree where the source of that package builds) so they are standalone and there is no distinction between "lock" package and normal packages stanza.

[rgrinberg]

At the moment, building an opam package with dune is something that we already have in pkg_rules. So I'm of course eager to reuse that code as much as possible because it already does all the hard work of integrating with the rest of the rules. Without using this module, the entire idea would be hopeless because of the massive amount of work that it would require. So it really is the path of least resistance by a wide margin.

Now it's true that this module has awareness of a bunch of things that you might not care about (lock dirs, autolocking, dev tools), but that's a question of organization that I'd rather address after the feature is in a working state. I much prefer refactoring after we have a working prototype rather than before.

But moreover, I would like this feature to integrate with package management even if not everybody intends to use it that way. I would for example prefer that a vendored package can be used to build a package inside the lock dir and vice versa.

tbh, I'd really prefer if locks were just a light way to create a workspace that then uses normal dune rules (that know nothing about the lock). You could, for instance, always promote the opam-repo metadata to the package (ie, copy the opam file back in the source/build tree where the source of that package builds) so they are standalone and there is no distinction between "lock" package and normal packages stanza.

I think you need to clarify what you mean here. Do you mean just copy opam packages along with their sources into some directory that dune knows how to use?

[samoht]

Without using this module, the entire idea would be hopeless because of the massive amount of work that it would require. So it really is the path of least resistance by a wide margin.
Now it's true that this module has awareness of a bunch of things that you might not care about (lock dirs, autolocking, dev tools), but that's a question of organisation that I'd rather address after the feature is in a working state.

To be clear, I really don't intend to reimplement it at all from scratch! Lots of work has already been put into that feature, so it's just about extracting/reorganising it. And I do think doing this sooner rather than later will help with testing and proper design, instead of having a very large file where everything is tightly coupled and hard to (unit) test.

Anyway, I don't have strong opinions either way. I created this PR to open the discussion, I didn't intend to have it merged straight away :-)

[samoht]

I think you need to clarify what you mean here. Do you mean just copy opam packages along with their sources into some directory that dune knows how to use?

Somehow my mental model is this one:

• The solver tells you what packages/libraries you need to make visible in your workspace (i.e., your source tree + maybe some scratch pad for fetched sources that is probably even better to always promote to the source tree to make your editor tools happy -- so that "jump to defintion" just jump into your source tree instead of hidden read-only files). To make sure only the solver recommendations are visible,e you need an extended notion of scope (that is addressed in feat(scope): add scope stanza for library visibility control #13337)
• You also need to encode in the workspace how you build those packages. For a package that uses dune, you do nothing special: dune alreadys knows how to build those :-) For a package that doesn't work, you can rely on the opam build/install instructions. A good way to store those instructions is to promote the opam-repositiry opam file into the fetched source: those opam-repo opam files are more up-to-date, are guaranteed to build (thanks to opam-repo CI), etc
• Once your workspace is set up, Dune works as usual. There is no notion of lockfiles, or locks or whatever.

[rgrinberg]

Yeah, that's basically the opam-monorepo model but with dune filling in first class support for non dune packages. I don't think it works for everybody, but I don't really see the tension in the dune code base to support both workflows. I think you already have your own solver integration in opam-monorepo to select version and fetch sources. Dune just needs to take it from there.

[samoht]

Why would this not work for everybody ?

Also I don't understand the rationale for the current pkg strategy : why you are trying to wrap dune builds inside opam build inside dune builds ? This is very inefficient and you loose all the good properties of Dune). Seems more reliable to just use opam at this point :-)

[rgrinberg]

Why would this not work for everybody ?

I don't want external sources polluting my workspace. I want to reuse dune's cache to store and deduplicate sources.

why you are trying to wrap dune builds inside opam build inside dune builds ? This is very inefficient and you loose all the good properties of Dune).

That is just a temporary limitation that will be lifted eventually.

Seems more reliable to just use opam at this point

But then your feature request to build opam packages in dune wouldn't have any legs to stand on :).

[samoht]

I want to reuse dune's cache to store and deduplicate sources.

So jump-to-definition will jump into the cache? :-)

[rgrinberg]

Yes, right in the _build directory. What's the downside? That we can't edit our dependencies? That's not really all that common. Moreover, any such patches would need to be formalized with something like pins or composition otherwise the next time the deps are updated by whatever tool re-downloads the sources they would be blown away.