New monaco-vscode-api / vite 8 beta 12 + esbuild-import-meta-url-plugin removal

[kaisalmen]

@CGNonofr vite@8.0.0-beta.11 handles import.meta.url properly now. Two related issues remain that I want to discuss with you. The problem may be resolved in monaco-vscode-api.

On this branch I disabled @codingame/esbuild-import-meta-url-plugin and the following two files are problematic as far as I can see. I manually changed the import to see if I can make it work (see commented code and following line). When this is in place all test pass and textmate / highlighting is working properly afterwards.

The following files are problematic:
@codingame/monaco-vscode-textmate-service-override/vscode/src/vs/workbench/services/textMate/browser/textMateTokenizationFeatureImpl.js ll. 313 following:

 async _loadVSCodeOnigurumaWASM() {
  if (isWeb) {
    const createData = {
      grammarDefinitions: this._grammarDefinitions,
      // onigurumaWASMUri: ( new URL('vscode-oniguruma/release/onig.wasm', import.meta.url)).href,
      onigurumaWASMUri: (new URL('../../../../../../../../../../vscode-oniguruma/release/onig.wasm', import.meta.url)).href
};

@codingame/monaco-vscode-textmate-service-override/vscode/src/vs/workbench/services/textMate/browser/backgroundTokenization/threadedBackgroundTokenizerFactory.js ll. 109 following:

async _createWorkerProxy() {
    const createData = {
        grammarDefinitions: this._grammarDefinitions,
        // onigurumaWASMUri: ( new URL('vscode-oniguruma/release/onig.wasm', import.meta.url)).href,
        onigurumaWASMUri: (new URL('../../../../../../../../../../vscode-oniguruma/release/onig.wasm', import.meta.url)).href
    };

import.meta.url refers to the js file and therefore the url is wrongly resolved by vite without the change. The problem is that a non-exported sub-package / file is tried to be loaded. That is actually fishy. I know you already changed that with a patch from what originally is done by VS Code.
Wouldn't it be best if vscode-onigurama supplies a function that allows to load or just supply the bundled onig.wasm? I don't know the reason why there is no such utility.
Referring to another node_module by a relative path (like I did) is not a good solution, I know. Can't we do that on our level? Or just copy the file to @codingame/monaco-vscode-textmate-service-override, so we can control it / expose a utility functions that ensure proper loading onig.wasm into an array buffer? Do you have other ideas?

[kaisalmen]

CC @cpojer regarding import.meta.url Vite 8 Beta 11 looks good so far. The problem above is not a vite issue as far as I can tell. needs to be discussed.

[CGNonofr]

Isn't the issue that @codingame/esbuild-import-meta-url-plugin was able to resolve like node, while vite 8.0.0-beta.11 is only able to resolve relative paths?

[kaisalmen]

Isn't the issue that @codingame/esbuild-import-meta-url-plugin was able to resolve like node

Something like that yes! But is new URL('vscode-oniguruma/release/onig.wasm', import.meta.url)) really a proper definition? If I understand https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta#url correctly it should not work in both environments (web or node). Does @codingame/esbuild-import-meta-url-plugin do to much magic here? Or am I understanding something incorrect.

Old vite (before vite 8 beta.11) did not load sub-module exports properly (e.g. workers) and also there were issues with VS Code theme resources not being resolved. This is all working now.
I know that Turbopack/Next is unable to handle the onig.wasm as well, so if we can supply a better URL (not cross-module dependent) or a loading functions (or even an inlined onig wasm), we get rid of a pain point. 🙂 WDYT?

[CGNonofr]

Something like that yes! But is new URL('vscode-oniguruma/release/onig.wasm', import.meta.url)) really a proper definition? If I understand https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta#url correctly it should not work in both environments (web or node). Does @codingame/esbuild-import-meta-url-plugin do to much magic here? Or am I understanding something incorrect.

If new URL('vscode-oniguruma/release/onig.wasm', import.meta.url)) is not a valid definition, then I don't see how import 'vscode-oniguruma' can be valid either 🤔 they obviously can't be resolved from the browser itself and need both a transformation to resolve them

if we can supply a better URL (not cross-module dependent) or a loading functions (or even an inlined onig wasm), we get rid of a pain point. 🙂 WDYT?

I'm not sure what you mean by any of the suggestion 🤔 Also inlining the wasm is a shame since it's pretty big and the inlining (in base64 I guess) with add a ~30% overhead

[kaisalmen]

I'm not sure what you mean by any of the suggestion 🤔 Also inlining the wasm is a shame since it's pretty big and the inlining (in base64 I guess) with add a ~30% overhead

Yeah, inlining is not a good idea. What I don't understand is why vscode-oniguruma ships the wasm, but only provides a method(s) to load a buffer. Why don't they provide a method for loading what is contained (like loadDefault)? I just had a look at the repo. v2 is from 2023 and not even used by VS Code. oniguriuma itself is archived. 👀 Don't know if raising issues helps there.

What I meant is to contain onig.wasm in @codingame/monaco-vscode-textmate-service-override as asset, then the resolution is module internal.

If new URL('vscode-oniguruma/release/onig.wasm', import.meta.url)) is not a valid definition, then I don't see how import 'vscode-oniguruma' can be valid either 🤔 they obviously can't be resolved from the browser itself and need both a transformation to resolve them

It would be a valid if defined as sub-export, wouldn't it? Like you do with the workers. Or does vite treat js differently than wasm with regard to URL resolution?

[CGNonofr]

What I don't understand is why vscode-oniguruma ships the wasm, but only provides a method(s) to load a buffer. Why don't they provide a method for loading what is contained (like loadDefault)?

They have the same problem as ours, how do you expect them to load it?

I just had a look at the repo. v2 is from 2023 and not even used by VS Code. oniguriuma itself is archived. 👀 Don't know if raising issues helps there.

That's concerning 🤔 probably not

What I meant is to contain onig.wasm in @codingame/monaco-vscode-textmate-service-override as asset, then the resolution is module internal.

One solution is to make vscode-textmate and oniguruma as internal in the monaco-vscode-api rollup config indeed 🤔 it's probably a good idea as no one will likely be using it by hands anyway 🤔 OR publish both packages: monaco-vscode-textmate-service-override and monaco-vscode-textmate-internalized-service-override or something 🤔

It would be a valid if defined as sub-export, wouldn't it? Like you do with the workers. Or does vite treat js differently than wasm with regard to URL resolution?

vscode-oniguruma doesn't define any sub-export so you are able to import any file in it by its path

[kaisalmen]

One solution is to make vscode-textmate and oniguruma as internal in the monaco-vscode-api rollup config indeed 🤔 it's probably a good idea as no one will likely be using it by hands anyway

If this removes pain points with bundlers I vote for such a solution. We don't need different packages, IMO. Also, vscode-oniguruma and vscode-textmate need to be optimized currently. This need would disappear as well.

[CGNonofr]

CodinGame/monaco-vscode-api#764 should make it

[CGNonofr]

CodinGame/monaco-vscode-api#764 should make it

Hum no, not working yet 🤔

[kaisalmen]

@CGNonofr awesome work. I confirm it is working nicely now.

[kaisalmen]

@CGNonofr this is ready now