Segfault on pytest exit in CI against pybind11 3.0.1

[HinTak]

We have a segfault on exiting pytest in CI from all builds when built against pybind11 3.0.1 . I cannot reproduce it locally, but it is blocking newer releases (I am somewhat reluctant to put in a pybind11 version limit).

In CI, it seems that pytest is successfully completed, then segfaults. I cannot get this behavior locally.

My guess is it is probably something like this:
pybind/pybind11#5827

@kyamagu just to sit on this a bit to see if pybind11 3.0.2 etc knows better.