[FEATURE] Move rimage toml config files to SOF repository

[kv2019i]

Is your feature request related to a problem? Please describe.
The toml files in sof/rimage/config/*.toml have recently started covered properties of SOF modules that are no longer static properties of a platform, but rather are describiting attributes that can change when code is changed in SOF main.

This highly problematic if a code to optimize a module's performance is merged to SOF main, a matching update is needed to rimage toml files (for all affected platforms), plus a separate sof/west.yml update to bring that in to SOF builds (with Zephyr where west is used).

Describe the solution you'd like
Rimage repository should only host the tool, not the actual platform-specific content.

Describe alternatives you've considered
Moving only IPC4 specific data is one option, but due to design of rimage, this seems rather complex and in the end the easier solution would be to move all of toml config files.

Additional context

• Recent rimage pull request history -> a sharp uptick in activity with IPC4 platforms:
  https://github.com/thesofproject/rimage/pulls?q=is%3Apr+is%3Aclosed+sort%3Aupdated-desc
• [BUG] sof_ri_info.py fails on Intel MTL+ binaries #8073

[lgirdwood]

@mwasko @marcinszkudlinski fyi - obvious effort reduction here to keep toml manifest configs alongside FW code.

[marc-hb]

The toml files in sof/rimage/config/*.toml have recently started covered properties of SOF modules that are no longer static properties of a platform, but rather are describiting attributes that can change when code is changed in SOF main.

Wait: the static properties still exist AFAIK. So shouldn't rimage be able to read from more than one .toml file / from both places? Would simply concatenating the files in the order given be enough? This would mean moving fewer files / less content and not breaking backwards compatibility, probably easier git bisect etc.

PS: I think zephyrproject-rtos/zephyr#54700 should go first. Working on it. EDIT: done.

EDIT: this seems highly relevant:

• rimage: add ipc4 extended module config introduction sof-docs#451

[kv2019i]

@marc-hb wrote:

Wait: the static properties still exist AFAIK. So shouldn't rimage be able to read from more than one .toml file / from both places? Would simply concatenating the files in the order given be enough? This would mean moving fewer files / less content

But why should rimage (the tool) repo host even the static details? I agree we could split, but I'm not sure if there is any benefit to hosting in multiple repos. These are essentially the platforms we maintain in SOF repository. I.e. shouldn't rimage repo be more like a repo for gcc/clang in that it provides a tool that takes X in and spews Y out, but the input data X is hosted in other repos.

[marc-hb]

After taking a look at thesofproject/sof-docs#451 I wonder why we're even building the extended manifest using C code in 2023?

https://github.com/thesofproject/rimage/issues?q=crash

AFAIK the extended manifest is merely concatenated to the "main" .ri rimage file, so they seem fairly independent from each other. So why stick to obsolete and crashing rimage "technology" for the extended manifest?

EDIT: C re-implementation of elfutils submitted in

• New functions to read elf files and parse modules (SP3) rimage#155

[pjdobrowolski]

I was just about closing my issue with crashing. Last changes in rimage improved significantly stability of that tool. @marc-hb I know that @aborisovich is huge advocate changing language of that tool but currently @softwarecki invest huge amount of work to make that tool a bit more bugfree. Soooo If you want to discuss future of that tool Andrey is right person ;)

[aborisovich]

@pjdobrowolski not exactly, I'm not a person responsible for such strategical decisions.
I had started recently a substitute project in C++, but @mmaka1 convinced me that developing such tool in this languages is not the right way to proceed. We also should consider possible development in Python.

After @softwarecki fixes the tool stopped crashing on Windows and looks much more stable.

[lgirdwood]

After taking a look at thesofproject/sof-docs#451 I wonder why we're even building the extended manifest using C code in 2023?

@marc-hb lets keep on topic. Verification of hashing/signing is faster in C and hence that's why rimage is in C today.

[softwarecki]

@lgirdwood I think @marc-hb was referring to the extended manifest created from toml files. It isn't signed, unlike the module manifest. So it can be generated by another tool and inserted at the beginning of the file.

For the toml files, I see the following problems with them:

• No support for file inclusion. We have a separate toml file for each platform. If a module is used by multiple platforms, we have an identical description of the module's capabilities/configuration duplicated in multiple files. A solution could be to separate module descriptions into separate files and include them in platform files.
• Lack of support for labels. At this moment, some fields of the module's configuration is represented as hexadecimal bit masks that are completely unreadable by humans. Textual value labels should be introduced to make the configuration more readable.
• Enabling/disabling building of a given module requires correcting toml files.

My idea to solve these problems is to move the information from the toml files to the source code of the modules. While leaving in them only some basic description of the platform and its address space.
We can prepare a structure with the information necessary to generate the extended manifest and put it in a separate section of the elf file. It would be read by rimage (or any other tool in the future). In this way, we already provide some informations now. This is the module manifest (struct sof_man_module_manifest) in the ".module" section and the extended manifest (duplicated name, this is different information than in toml files), (struct ext_man_header) section ".fw_metadata". Module manifest for all modules we can easily move to source code, because its reading is already implemented in rimage. The ext_man_header structure was designed to be extensible, so it could be used to pass the module's pin configuration, etc. to rimage. The advantage of this approach is that rimage (or separate tool) would build all manifests using only a compilation artifacts. Adding/removing a module would not involve updating multiple toml files for different platforms.
When creating a loadable library, we eliminate the risk of taking a toml file that does not match the compiled module. The module metadata would be included in the module file.