Problem
`AbstractIndexingAdvice` catches `RuntimeException` thrown by a serializer's `populate(...)` per-entity and logs at WARN, then continues. The intent (fault isolation across a batch of saves) is sound, but the consequences are:
- Ops dashboards rarely page on WARN. The failure is effectively silent.
- There's no metric. Operators have no way to know "5% of saves on this resource type are failing serialization for the last 30 days."
- The bad rows are missing from the index. Consumers see partial data and have no signal that anything is wrong.
This drives defensive coding on every serializer: in the billing slice we accepted (with explicit comments) that we needed null-guards on every dereferenced field because an NPE would silently drop the document. See ADR D19 in openmrs/openmrs-module-billing#176 for the analysis. But defensive coding only catches the foreseeable nulls; logic errors, validator gaps, and upstream-model changes still produce silent drift.
Distinct from #18
Issue #18 covers durability of the post-commit dispatch (the serializer produced a document, but the backend write failed). This issue covers durability of the serialize step itself (the serializer threw before producing a document). They share the same observability gap but are otherwise independent failure surfaces:
Both should produce a metric and an alertable signal. They could share the same infrastructure but represent different failure causes that operators care to distinguish.
Suggested approach
Three steps:
-
ERROR-level log on per-entity serialize failure (currently WARN). Include the resource type, resource UUID, and exception class. The current WARN level is below most production alert thresholds.
-
Micrometer counter `querystore.index.serialize.failure` tagged by `resource_type` and `exception_class`. Lets operators dashboard the failure rate per resource type and alert when it crosses a threshold.
-
Optional: a configurable strict-mode per resource type that re-throws the exception (failing the originating transaction) rather than swallowing. Useful for resource types where a missing document is worse than a failed save. Default remains "swallow + log + count."
(1) is essentially free. (2) is the principled fix. (3) is opt-in for callers who can tolerate the latency / availability trade-off.
Failure mode prevented
A logic bug in a serializer (e.g., a typo that NPEs on a rare code path) ships to production, silently drops 3% of documents for the affected resource type, and the failure surfaces only when a dashboard query returns suspiciously low numbers — usually weeks later, with no easy way to identify which rows are missing or re-index them.
Related: #16, #17, #18, #27, #28.
Problem
`AbstractIndexingAdvice` catches `RuntimeException` thrown by a serializer's `populate(...)` per-entity and logs at WARN, then continues. The intent (fault isolation across a batch of saves) is sound, but the consequences are:
This drives defensive coding on every serializer: in the billing slice we accepted (with explicit comments) that we needed null-guards on every dereferenced field because an NPE would silently drop the document. See ADR D19 in openmrs/openmrs-module-billing#176 for the analysis. But defensive coding only catches the foreseeable nulls; logic errors, validator gaps, and upstream-model changes still produce silent drift.
Distinct from #18
Issue #18 covers durability of the post-commit dispatch (the serializer produced a document, but the backend write failed). This issue covers durability of the serialize step itself (the serializer threw before producing a document). They share the same observability gap but are otherwise independent failure surfaces:
Both should produce a metric and an alertable signal. They could share the same infrastructure but represent different failure causes that operators care to distinguish.
Suggested approach
Three steps:
ERROR-level log on per-entity serialize failure (currently WARN). Include the resource type, resource UUID, and exception class. The current WARN level is below most production alert thresholds.
Micrometer counter `querystore.index.serialize.failure` tagged by `resource_type` and `exception_class`. Lets operators dashboard the failure rate per resource type and alert when it crosses a threshold.
Optional: a configurable strict-mode per resource type that re-throws the exception (failing the originating transaction) rather than swallowing. Useful for resource types where a missing document is worse than a failed save. Default remains "swallow + log + count."
(1) is essentially free. (2) is the principled fix. (3) is opt-in for callers who can tolerate the latency / availability trade-off.
Failure mode prevented
A logic bug in a serializer (e.g., a typo that NPEs on a rare code path) ships to production, silently drops 3% of documents for the affected resource type, and the failure surfaces only when a dashboard query returns suspiciously low numbers — usually weeks later, with no easy way to identify which rows are missing or re-index them.
Related: #16, #17, #18, #27, #28.