Double precision float threshold value for checking if quat is near identity

[dimvoly]

Issue

The single precision default threshold value is float threshold_angle = 0.00284714461F. There is commentary about the logic used to arrive at that value. This assumes an "error threshold" of 1.e-6.

The double precision default threshold value is the same value double threshold_angle = 0.00284714461. The comments reference the single precision case, but no comment is made on the possible differences.

It would seem, that if you follow similar logic for the double precision version, you'll arrive at an "error threshold" of 1.e-15, and thus a threshold_angle of 8.940696716308595e-8. This is just above the presumably smallest detectable angle of

0.99999999999999988897769753748434595763683319091796875.acos() * 2.0_f64 = 2.9802322387695313e-8

Alternatively, if that's too close for comfort, you could use an "error threshold" of 1.e-14 and get a threshold_angle of 2.8272965492323474e-7.

Caveats

The above calculation was done in rust (I don't know cpp), so I'm not sure on what the acos() would actually give me if using the scalar_acos() from this library. Also, mathematically, I'm not too knowledgeable on how these operations came about, I'm merely comparing the logic in the single precision case vs the double precision case. Commenting on this library, as the glam library uses similar logic and references this library.

Summary Question

Should the double precision version follow a tighter threshold? And if so, should it be based on the "error threshold" of 1.e-14, yielding a threshold_angle of 2.8272965492323474e-7?

[nfrechette]

Hello and thanks for reaching out.
I believe your analysis is correct and indeed the float64 threshold could be tighter at 1e-14 as you mention.

For float32, there is barely enough precision to be able to represent small angles with a quaternion representation and things deteriorate significantly if your quaternion isn't normalized (e.g. if you wanted to encode multiple 'turns' in the rotation motion). In practice, this means that quaternions near the identity can sometimes represent valid values which can lead to a measurable impact, visually, whether or not they are treated as being equivalent to the identity. The safest thing there would probably be to use a 0.0 threshold for float32 but that seemed extremely restrictive and so I made a judgment call to derive the value from the smallest computable angle using float32 precision. (Fun fact, even 1bit away from 1.0 on the quat W component can lead to a measurable impact at human scale with float32, 0.99999964 or whatever that value is)

Float64 has no such issues with their increased precision, at least when it comes to practical use cases in realtime applications (e.g. gamedev). When it came time to port the API to keep it symmetrical, I simply kept the same value (aka I was lazy). You'll see other matching thresholds in the float64 API.

I've been working my way towards completing the next release. I'm still holding hope that it can land this year, but perhaps in the first half of next year is more realistic. I'll include a fix for this in then.

Thanks for reporting this and the details you've provided. With RTM, I tried my best to explain where every 'magic' constant comes from in contrast to other math libs; glad to see it was useful :)

[dimvoly]

Thank you for the commentary, helps me and others learn. Yes I have encountered the single float rotation upon rotation issue you describe and these checks for identity are quite a helpful function. Great work on the library.