8373096: JFR leak profiler: path-to-gc-roots search should be non-recursive

[tstuefe]

A customer reported a crash when producing a JFR recording with path-to-gc-roots=true. It was a native stack overflow that occurred during the recursive path-to-gc-root search performed in the context of PathToGcRootsOperation.

We try to avoid this by limiting the maximum search depth (DFSClosure::max_dfs_depth). That solution is brittle, however, since recursion depth is not a good proxy for thread stack usage: it depends on many factors, e.g., compiler inlining decisions and platform specifics. In this case, the VMThread's stack was too small.

This RFE changes the algorithm to be non-recursive.

Note that as a result of this change, the order in which oop maps are walked per oop is reversed : last oops are processed first. That should not matter for the end result, however. The search is still depth-first.

Note that after this patch, we could easily remove the max_depth limitation altogether. I left it in however since this was not the scope of this RFE.

Testing:

• Tested manually with very small (256K) thread stack size for the VMThread - the patched version works where the old version crashes out
• Compared JFR recordings from both an unpatched version (with a large enough VMThread stack size) and a patched version; made sure that the content of "Old Object Sample" was identical
• Ran locally all jtreg tests in jdk/jfr
• GHAs

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8373096: JFR leak profiler: path-to-gc-roots search should be non-recursive (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28659/head:pull/28659
$ git checkout pull/28659

Update a local copy of the PR:
$ git checkout pull/28659
$ git pull https://git.openjdk.org/jdk.git pull/28659/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28659

View PR using the GUI difftool:
$ git pr show -t 28659

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28659.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@tstuefe The following label will be automatically applied to this pull request:

• hotspot-jfr

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (5ac152db)

[tstuefe]

Ping @egahlin