Canonical Payload Encoding

[XAMPPRocky]

Currently the standard is pretty open on how the payload can be encoded and provides the meta section for handling the sections encoded at different offsets and sizes. While this allows it to work with a variety of different systems and implementations, I think it would be beneficial if there also was a canonical representation for the payload (meaning the same data always produces the same payload).

At the very least it would make it easy to share a test suite of canonical data between implementations that could be validated, but would also make it easier to allow for payloads to be signed and shared.

This wouldn't have to come at the expense of the current non-canonical representation, it could just be a separate set of restrictions listed in addition that are required to make the encoded payload canonical and implementors could choose whether they want to make a canonical encoder or not.

Reading the spec I can think of the following restrictions being needed to make it canonical.

• The meta section is always an empty map
• The encoding is always meta, main, aux (the current spec doesn't restrict you from having it be meta, aux, main).
• Keys in maps must be in ascending order.
• enum_array must encode the values in ascending order.
• color_rgba must always be encoded in 4 bytes with RGBA.
• byte strings should be defined as always one kind (definite or indefinite).

number types is probably the trickiest with the current spec because it allows for all possible kinds of numbers. I honestly think it would be more useful if number was split up into three types (signed, unsigned, real) because it's not particularly useful for example that fields like filament_diameter or the weight fields can be negative and be considered valid tags but that's probably a separate issue.

Number could be made canonical with following rules

• All positive integers must be encoded as unsigned integers
• All negative integers must be encoded as signed integers
• All real numbers within ±65,504 must be encoded as half floats.
• All real numbers exceeding ±65,504 must be encoded as floats.

[CZDanol]

Hello,
Thanks for the suggestion :) We will take it into consideration, it might indeed be practical for some use cases to have this.

FYI, there is a canonical CBOR that already covers some of your points (number encoding, map fields ordering).

Some notes why we decided not to enforcfe something like this:

1. OPT is also intended for embedded devices, where ordering of the fields would cost resources
2. OPT aux region is designed to be updated "en situ" (when you want to update used filament, you're not supposed to write the whole tag again, but just write the diffed bytes in the aux region memory). Adding new fields to aux region at the end of the map will then produce smaller update diff than if the new field would have to be canonically sorted in to the middle.
3. Storing all the data canonically would be impossible to enforce and most implementations would probably not do it anyway.

The meta section is always an empty map

I'm a bit confused why you would want this - this would mean that there could not be any aux section/data. And it goes against your 'meta, main, aux'.

color_rgba must always be encoded in 4 bytes with RGBA.

Fair point here.

filament_diameter or the weight fields can be negative and be considered valid tags

Hmm, I'd say it might be more versatile if we allow specifying the range of permitted values instead of just differentiating between signed and unsigned. Diameter of 65 km should probably not be valid as well :)

[XAMPPRocky]

FYI, there is a canonical CBOR that already covers some of your points (number encoding, map fields ordering).

Some notes why we decided not to enforcfe something like this:

Oh yeah, to be clear my ask is not for this to be the only representation, but for it to be a representation. In my experience writing a framework for ASN.1 (a suite of binary wire formats for embedded and desktop devices) most of those formats offer both canonical and non-canonical variants for exactly the same reasons you provide. Canonical representation often comes at a performance cost to get reproducibility.

Storing all the data canonically would be impossible to enforce and most implementations would probably not do it anyway.

At the spec level yes, but at the individual library author/vendor level it would be possible and desirable. For example; ensuring there's no regressions between library versions in how something is encoded is a lot easier if there's a canonical format. Validating that the payload hasn't been corrupted is a lot easier and quicker if you can just do a byte match on the section rather than needing to decode and match each entry individually to check that they are semantically equivalent.

I'm a bit confused why you would want this - this would mean that there could not be any aux section/data. And it goes against your 'meta, main, aux'.

I misread a bit of the spec (forgetting the "Its presence is indicated by the aux_region_offset field." part), but from my understanding setting the offset wouldn't actually be required to have an aux section in the wire format. Since the payload is essentially a sequence of three maps (if you don't offset them from each other), you could just read the bytes immediately following the main section to get the aux section, since the sections are already delimited by 0xff break codes.

Hmm, I'd say it might be more versatile if we allow specifying the range of permitted values instead of just differentiating between signed and unsigned. Diameter of 65 km should probably not be valid as well :)

I agree, I think you want both as I think you'd want to differentiate between wire format and valid representation. Otherwise you would need custom tags for each unique range of values which could hinder compatibility with existing cbor libraries which may not support custom tags (or not support them well). So you'd probably want to be able to say for example filament_diameter is a real with a range of 0–1000.

[CZDanol]

I misread a bit of the spec (forgetting the "Its presence is indicated by the aux_region_offset field." part), but from my understanding setting the offset wouldn't actually be required to have an aux section in the wire format.

Technically, not, but not requiring it would mean that the user would first need to parse the whole main section to be able to determine where the aux region starts - not desirable. But in the canonical format, we might simply say that the aux region is required to start right after the main one, so the aux_section_offset should have the exact value to reflect that.

[XAMPPRocky]

Yeah, that makes sense. I think that's a better solution.

[XAMPPRocky]

Another restriction I just noticed.

• enum_array types must only contain unique values.

Currently you could fill it with duplicate values which seems antithetical to its purpose since there's nothing that says that you should treat [glitter, glitter] differently from [glitter].