Better, faster, more debugging info #56
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
💻 Description of Change(s) (w/ context)
Added lock contention detection with additional context to figure out the context (depth of container descent)
Improved the way back-references (contained by, etc.) handled to make things much more efficient.
🧠 Rationale Behind Change(s)
There were a bunch of issues around handling large sets of JARs that contained a lot of the same class files.
The number of Items per minute would decline as there were more Items processed.
So, I added detection for row-level lock contention.
Turns out there was a lot of lock contention, although lock contention should have been rare.
But there was mistake in the way back-references were handled.
Back-references were done after an Item was merged with the Item in storage. This means that
if a
.class
file was contained in a lot of different JAR files, each time the.class
file was processed,it updated all the references to all the JAR files that it had previously been referenced.
This was causing lock contention and a lot of extra processing.
This change should speed Goat Rodeo by 2x or more.
📝 Test Plan
Existing tests pass
📜 Documentation
Scala docs
💣 Quality Control
(All items must be checked before a PR is merged)
Did you…
Before Merging…