[SPARK-53898][CORE] Shuffle cleanup should not clean MapOutputTrackerMaster.shuffleStatuses in local cluster #52606
Conversation
| case _ => | ||
| } | ||
| // Shuffle cleanup should not clean up shuffle metadata on the driver | ||
| assert(mapOutputTrackerMaster.shuffleStatuses.nonEmpty) |
There was a problem hiding this comment.
Tests fails this assert before the fix.
| // through `ContextCleaner` when the shuffle is considered no longer referenced anywhere. | ||
| // Otherwise, we might hit exceptions if there is any subsequent access (which still | ||
| // reference that shuffle) to that shuffle metadata in `MapOutputTrackerMaster`. E.g., | ||
| // an ongoing subquery could access the same shuffle metadata which could have been |
There was a problem hiding this comment.
I am not sure if we should mention this, since the ideal behavior would be to terminate any subqueries when the main query completes.
There was a problem hiding this comment.
There could be a race even if we terminating the subquery?
There was a problem hiding this comment.
The main query has ended already. The subqueries results are anyway not going to be used (and is also a waste of resource in allowing it to continue).
There was a problem hiding this comment.
I understand we should cancel the running subquery and we should do it. My point is the running subquery could still access MapOutputTrackerMaster even if we cancel it right after the main query ends due to the race between them. So I think it's fine to mention it here.
| case RemoveShuffle(shuffleId) => | ||
| doAsync[Boolean](log"removing shuffle ${MDC(SHUFFLE_ID, shuffleId)}", context) { | ||
| if (mapOutputTracker != null) { | ||
| if (mapOutputTracker != null && !mapOutputTracker.isInstanceOf[MapOutputTrackerMaster]) { |
There was a problem hiding this comment.
Should this be in check be in unregistershuffle if we dont expect this to be called in master.
There can be someone else calling this same method on local mode in future?
There was a problem hiding this comment.
I was thinking about that. That way requires us to pass isLocal into MapOutputTrackerMaster, involves more changes. But I also agree it's safer.
There was a problem hiding this comment.
So with local mode, we can't clean up shuffle files only?
There was a problem hiding this comment.
I think shuffle cleanup still happens in local.
But the shuffle metadata cleanup only happens from ContextCleaner in driver.
|
cc: @cloud-fan |
| // an ongoing subquery could access the same shuffle metadata which could have been | ||
| // cleaned up after the main query completes. Note this currently only happens in local | ||
| // cluster where both driver and executor use the `MapOutputTrackerMaster`. | ||
| mapOutputTracker.unregisterShuffle(shuffleId) |
There was a problem hiding this comment.
so for non-local cluster, mapOutputTracker is at executors and unregisterShuffle only unregister the shuffle at executor side?
There was a problem hiding this comment.
On executors, it would call the MapOutputTrackerWorker#unregisterShuffle, where the shuffle status are not cleaned up unlike in MapOutputTrackerMaster#unregisterShuffle
In local, we just have MapoutputTrackerMaster and ends up cleaning the shufflestatuses
MapOutputTrackerMaster
def unregisterShuffle(shuffleId: Int): Unit = {
shuffleStatuses.remove(shuffleId).foreach { shuffleStatus =>
shuffleStatus.invalidateSerializedMapOutputStatusCache()
shuffleStatus.invalidateSerializedMergeOutputStatusCache()
}
}
MapOutputTrackerWorker
def unregisterShuffle(shuffleId: Int): Unit = {
mapStatuses.remove(shuffleId)
mergeStatuses.remove(shuffleId)
shufflePushMergerLocations.remove(shuffleId)
}
There was a problem hiding this comment.
so how do we clean up shuffle files with local mode?
There was a problem hiding this comment.
The shuffle files are cleaned by rpc RemoveShuffle, sent from driver to executors and handled by BlockManagerStorageEndpoint that deletes the files on the disk. And the rpc RemoveShuffle nowadays can be raised in two ways: 1) ContextCleaner 2) Shuffle Cleanup feature at the end of a SQL query. These are the same for all the modes.
There was a problem hiding this comment.
ah, so shuffle files are already cleaned up before we reach here?
There was a problem hiding this comment.
One idea: shall we add a new method clearShuffleStatusCache and call it here? The executor side shuffle status is more like a cache and the driver side one is single source of truth. MapOutputTrackerMaster#clearShuffleStatusCache is noop.
There was a problem hiding this comment.
ah, so shuffle files are already cleaned up before we reach here?
No. We're handling the RemoveShuffle rpc in BlockManagerStorageEndpoint right at this point. The files are deteleted at the line 72 by shuffleManager.unregisterShuffle(shuffleId).
There was a problem hiding this comment.
One idea: shall we add a new method clearShuffleStatusCache and call it here? The executor side shuffle status is more like a cache and the driver side one is single source of truth.
Actaully we already do this in non-local mode. In non-local mode, we call MapOutputTrackerWorker. unregisterShuffle() to clean up the statues (that's exactly what line 68 does). In local mode, we don't use MapOutputTrackerWorker so there is no cached statuses to clean.
There was a problem hiding this comment.
yea it's the same thing, but clearShuffleStatusCache is more explicit and clear than skipping unregisterShuffle if the instance is MapOutputTrackerWorker.
3 participants
What changes were proposed in this pull request?
This PR fixes a bug where
MapOutputTrackerMaster.shuffleStatusesis mistakenly cleaned up by Shuffle Cleanup feature in local cluster. The fix is done by avoid invokingmapOutputTracker.unregisterShuffle()inBlockManagerStorageEndpointwhenmapOutputTrackerisMapOutputTrackerMasteras it only happens in local cluster (non-local cluster should useMapOutputTrackerWorkerinstead).Why are the changes needed?
MapOutputTrackerMaster.shuffleStatusesshould only be cleaned whenContextCleanerconsiders the shuffle is no longer referenced anywhere. Otherwise, any subsequent access (which still reference that shuffle) to the same shuffle metadata inMapOutputTrackerMastercan lead toSparkExceptionand crash theSparkContext. Note this currently only happens in local cluster due to both driver and executor use theMapOutputTrackerMaster. E.g., an ongoing subquery could access the same shuffle metadata which could have been removed after the main query completes. See the detailed discussion at #52213 (comment).Does this PR introduce any user-facing change?
No.
How was this patch tested?
Updated the existing tests.
Was this patch authored or co-authored using generative AI tooling?
No.