Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better, faster, more debugging info #56

Merged
merged 1 commit into from
Feb 4, 2025
Merged

Better, faster, more debugging info #56

merged 1 commit into from
Feb 4, 2025

Conversation

dpp
Copy link
Contributor

@dpp dpp commented Feb 4, 2025

💻 Description of Change(s) (w/ context)

Added lock contention detection with additional context to figure out the context (depth of container descent)

Improved the way back-references (contained by, etc.) handled to make things much more efficient.

🧠 Rationale Behind Change(s)

There were a bunch of issues around handling large sets of JARs that contained a lot of the same class files.
The number of Items per minute would decline as there were more Items processed.

So, I added detection for row-level lock contention.

Turns out there was a lot of lock contention, although lock contention should have been rare.

But there was mistake in the way back-references were handled.

Back-references were done after an Item was merged with the Item in storage. This means that
if a .class file was contained in a lot of different JAR files, each time the .class file was processed,
it updated all the references to all the JAR files that it had previously been referenced.

This was causing lock contention and a lot of extra processing.

This change should speed Goat Rodeo by 2x or more.

📝 Test Plan

Existing tests pass

📜 Documentation

Scala docs

💣 Quality Control

(All items must be checked before a PR is merged)
Did you…

  • Mention an issue number in the PR title?
  • Update the version # in the build file?
  • Create new and/or update relevant existing tests?
  • Create or update relevant documentation and/or diagrams?
  • Comment your code?
  • Fix any stray verbose logging (removing, or moving to debug / trace level)?

Before Merging…

  • Make sure the Quality Control boxes are all ticked
  • Make sure any open comments or conversations on the PR are resolved

…torage. Added support for detecting and alerting on Storage lock contention. Improved performance by not repeatedly updating every back-reference

Signed-off-by: David Pollak <[email protected]>
@dpp dpp requested review from aredridel and earldouglas February 4, 2025 22:09
@dpp dpp merged commit a116735 into main Feb 4, 2025
2 checks passed
@dpp dpp deleted the better_faster branch February 12, 2025 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants