High Memory Usage Incident: contoso-chat-net - System.Byte[] Heap Growth

[sanchitmehta]

Incident Summary

• App Name: contoso-chat-net
• Incident Time: 2025-05-08T15:14:00Z to 2025-05-08T15:43:00Z
• Alert: 0% availability and memory spike detected

Memory Analysis

• Type occupying most memory: System.Byte[] (2,878,627 KB, 15,701 objects)
• Largest GC root chain:
  • System.Object[] → System.Object[] → ContosoChat2.Customers+Processor → ContosoChat2.Customers+CustomerCache → System.Collections.Generic.List<ContosoChat2.Customers+Customer> → ContosoChat2.Customers+Customer[] → ContosoChat2.Customers+Customer → System.Byte[] (30 objects, 30,725 KB)
• Other notable GC roots:
  • Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.CoreEventSource (1 object, 62 KB)
  • Microsoft.AspNetCore.Builder.WebApplication (4 objects, 0.2 KB)
  • System.Threading.Tasks.TplEventSource (4 objects, 0.2 KB)
  • System.Net.Http.Headers.KnownHeader (5 objects, 0.16 KB)

Steps Taken

1. Detected 0% availability and memory spike >10%.
2. Collected memory dump and performed heap analysis.
3. Scaled up app service plan to mitigate immediate impact.
4. Notified user and opened this issue for code review.

Resolution Guidance

• Investigate the referenced chains, especially the CustomerCache and Processor objects, for potential memory leaks or unbounded object retention.
• Consider implementing cache eviction or memory management strategies.

---

This issue was automatically created by the operations agent in response to a production incident.
Tracked by the SRE agent here

Here