8352116: Deadlock with GCLocker and JVMTI after JDK-8192647

[albertnetymk]

Using a new lock (JNICritical_lock) in GCLocker::block to resolve a deadlock issue. The root cause of the deadlock is that holding Heap_lock while waiting in GCLocker::block is unsafe.

The new lock is held from the start of GCLocker::block to the end of GCLocker::unblock. This requires adjusting Heap_lock's rank to allow acquiring Heap_lock while holding JNICritical_lock. The most important changes are in gcVMOperations.cpp and mutexLocker.cpp.

Test: tier1-8; verified failure can be observed 2/2000 and pass 8000 iterations.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8352116: Deadlock with GCLocker and JVMTI after JDK-8192647 (Bug - P2)

Reviewers

• Erik Österlund (@fisk - Reviewer) Review applies to bd3aa431
• Kim Barrett (@kimbarrett - Reviewer)
• Thomas Schatzl (@tschatzl - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/24407/head:pull/24407
$ git checkout pull/24407

Update a local copy of the PR:
$ git checkout pull/24407
$ git pull https://git.openjdk.org/jdk.git pull/24407/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 24407

View PR using the GUI difftool:
$ git pr show -t 24407

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/24407.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back ayang! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@albertnetymk This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8352116: Deadlock with GCLocker and JVMTI after JDK-8192647

Reviewed-by: kbarrett, tschatzl, eosterlund

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 39 new commits pushed to the master branch:

• d63b561: 8353188: C1: Clean up x86 backend after 32-bit x86 removal
• d1e91fc: 8353344: RISC-V: Detect and enable several extensions for debug builds
• 6abf4e6: 8353568: SEGV_BNDERR signal code adjust definition
• 6d9ece7: 8351949: RISC-V: Cleanup and enable store-load peephole for membars
• 97ed536: 8346989: C2: deoptimization and re-execution cycle with Math.*Exact in case of frequent overflow
• 660b17a: 8350852: Implement JMH benchmark for sparse CodeCache
• 6d37e63: 8353753: Remove unnecessary forward declaration in oop.hpp
• 9bb804b: 8338554: Fix inconsistencies in javadoc/doclet/testLinkOption/TestRedirectLinks.java
• e17c399: 8353201: Open source Swing Tooltip tests - Set 2
• 9fcb06f: 8352997: Open source several Swing JTabbedPane tests
• ... and 29 more: https://git.openjdk.org/jdk/compare/f7a94feedd63775a09d0bcb9ef3313972e2a5d69...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[openjdk]

@albertnetymk The following label will be automatically applied to this pull request:

• hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[albertnetymk]

/cc hotspot-gc

[openjdk]

@albertnetymk
The hotspot-gc label was successfully added.

[fisk]

Looks good. Would be nice to refactor the if (UseSerialGC || UseParallelGC) code to something that explains why it's there (those are the GCs that use the new improved GC locker). But that's pre existing so I don't mind if it's split to a separate RFE.

[mlbridge]

Webrevs

• 00: Full (b225cdb7)

[kimbarrett]

Looks good.

[albertnetymk]

Thanks for review.

/integrate

[openjdk]

Going to push as commit 39549f8.
Since your change was applied there have been 39 commits pushed to the master branch:

• d63b561: 8353188: C1: Clean up x86 backend after 32-bit x86 removal
• d1e91fc: 8353344: RISC-V: Detect and enable several extensions for debug builds
• 6abf4e6: 8353568: SEGV_BNDERR signal code adjust definition
• 6d9ece7: 8351949: RISC-V: Cleanup and enable store-load peephole for membars
• 97ed536: 8346989: C2: deoptimization and re-execution cycle with Math.*Exact in case of frequent overflow
• 660b17a: 8350852: Implement JMH benchmark for sparse CodeCache
• 6d37e63: 8353753: Remove unnecessary forward declaration in oop.hpp
• 9bb804b: 8338554: Fix inconsistencies in javadoc/doclet/testLinkOption/TestRedirectLinks.java
• e17c399: 8353201: Open source Swing Tooltip tests - Set 2
• 9fcb06f: 8352997: Open source several Swing JTabbedPane tests
• ... and 29 more: https://git.openjdk.org/jdk/compare/f7a94feedd63775a09d0bcb9ef3313972e2a5d69...master

Your commit was automatically rebased without conflicts.

[openjdk]

@albertnetymk Pushed as commit 39549f8.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.