8373100: Genshen: Control thread can miss allocation failure notification

[earthling-amzn]

In some cases, the control thread may fail to observe an allocation failure. This results in the thread which failed to allocate waiting forever for the control thread to run a cycle. Depending on which thread fails to allocate, the process may not make progress.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8373100: Genshen: Control thread can miss allocation failure notification (Bug - P4)

Reviewers

• Kelvin Nilsen (@kdnilsen - Committer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28665/head:pull/28665
$ git checkout pull/28665

Update a local copy of the PR:
$ git checkout pull/28665
$ git pull https://git.openjdk.org/jdk.git pull/28665/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28665

View PR using the GUI difftool:
$ git pr show -t 28665

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28665.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back wkemper! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@earthling-amzn The following labels will be automatically applied to this pull request:

• hotspot-gc
• shenandoah

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (1081f21e)
• 00: Full (89af1701)

[kdnilsen]

Thanks.

[earthling-amzn]

These methods were the root cause here. ShenandoahHeap::_canceled_gc is read/written atomically, but ShenandoahGenerationalControlThread::_requested_gc_cause is read/written under a lock. These notify_cancellation methods did not update _requested_gc_cause at all. So, in the failure I observed we had:

1. Control thread finishes cycle and sees no cancellation is requested (no lock used).
2. Mutator thread fails allocation, cancels GC (again, no lock used), and does not change _requested_gc_cause.
3. Control thread takes _control_lock and checks _requested_gc_cause and sees _no_gc (because notify_cancellation didn't change it) and waits forever now.

The fix here is to replace notify_cancellation with notify_control_thread which serializes updates to _requested_gc_cause under _control_lock.

[kdnilsen]

Thanks for diligent testing and analysis. Subtle code here.