You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we keep every single history even though they are thousands of epochs earlier.
We don't need to keep all of them in-memory. We can discard 9/10 of them, for instance.
If a gleaner is lagging behind, it might have to scan more logs due to the history compaction, so we have to find a right balance between wasted memory and wasted work.
Or, write the history out to a file.
related code:
void LoggerRef::add_epoch_history(const EpochMarkerLogType& epoch_marker) {
...
if (control_block_->epoch_history_count_ >= LoggerControlBlock::kMaxEpochHistory) {
// TASK(Hideaki) To avoid this, we should maintain sparse history, like one history per
// tens of epochs. So far kMaxEpochHistory is enough big, so it won't happen.
LOG(FATAL) << "Exceeded kMaxEpochHistory. Unexpected.";
}
The text was updated successfully, but these errors were encountered:
Currently we keep every single history even though they are thousands of epochs earlier.
We don't need to keep all of them in-memory. We can discard 9/10 of them, for instance.
If a gleaner is lagging behind, it might have to scan more logs due to the history compaction, so we have to find a right balance between wasted memory and wasted work.
Or, write the history out to a file.
related code:
void LoggerRef::add_epoch_history(const EpochMarkerLogType& epoch_marker) {
...
if (control_block_->epoch_history_count_ >= LoggerControlBlock::kMaxEpochHistory) {
// TASK(Hideaki) To avoid this, we should maintain sparse history, like one history per
// tens of epochs. So far kMaxEpochHistory is enough big, so it won't happen.
LOG(FATAL) << "Exceeded kMaxEpochHistory. Unexpected.";
}
The text was updated successfully, but these errors were encountered: