Debounce Content Publish Event Manifest Refreshes in C# Microservices

[mhijaziB]

Reason for the Feature

There is an existing TODO in the C# microservice content publish handling path suggesting that content publish events may need to be debounced:

// TODO: Maybe this should get debounced? If a flood of events started pouring in for whatever reason, this could kill us.

Location:

microservice/microservice/Content/ContentService.cs

Inside:

HandleContentPublish(ContentManifestUpdateEvent evt)

Currently, every content publish/update event triggers:

GetManifestEntry().RefreshManifest();

If multiple content events arrive close together, the microservice may repeatedly invalidate/refresh the manifest cache. This can create unnecessary churn and increase load during content publish flows, especially when downstream code also resolves content entries after receiving a content update event.

This is related to, but separate from, transient 503 failures during ClientContentInfo.Resolve(). The 503 resilience issue should handle failed content fetches. This ticket focuses specifically on reducing repeated manifest refresh work caused by bursts of content publish events.
Ticket 4599

Suggested Requirements

• Investigate the current HandleContentPublish(ContentManifestUpdateEvent evt) behavior in:

  microservice/microservice/Content/ContentService.cs
  

• Add debounce or rate-limiting behavior around manifest refresh after content publish/update events.

• If multiple content update events arrive within a short window, collapse them into a single manifest refresh.

• Ensure the latest manifest is eventually refreshed after the debounce window.

• Avoid blocking the content event handler for long-running work.

• Make debounce timing configurable if appropriate.

• Add logging or metrics so we can tell when events were collapsed/debounced.

• Add tests covering:

  • a single content update event still refreshes the manifest
  • multiple rapid events result in one refresh
  • events separated by more than the debounce window trigger separate refreshes
  • manifest refresh still happens after the final event in a burst

• Ensure this does not break existing content cache invalidation semantics.

Alternatives or Workarounds

Clients can currently reduce the impact by avoiding heavy content processing directly inside content update handlers, or by adding their own batching/delay logic.

However, the TODO already identifies this as a platform-level concern. Each microservice should not need to implement its own debounce layer around content publish events.

Additional Context

Existing TODO:

microservice/microservice/Content/ContentService.cs

protected virtual void HandleContentPublish(ContentManifestUpdateEvent evt)
{
    GetManifestEntry().RefreshManifest();
    // the event data isn't super useful on its own, so just use it to know that our cache is invalid.
    // TODO: Maybe this should get debounced? If a flood of events started pouring in for whatever reason, this could kill us.
}

This ticket should be tracked separately from the content resolution retry/backoff issue. That issue addresses transient failures when fetching content objects. This issue addresses repeated manifest refreshes caused by bursts of content publish/update events.

[cdhanna]

First, we should validate this is actually a problem by force pumping a lot of manifest updates into a realm with a microservice; and see if it breaks. It probably will, but before we fix anything, we should know what symptom we are fixing.