Add failed-job email notifications using CDA user lists

[krowvin]

Summary

• add durable, office-scoped failed-job notification templates and per-script alert rules
• resolve current recipients from CDA user lists and expose list selection in Scripts Manager
• enqueue rendered email envelopes for a separate local/AWS notification worker, with retry/DLQ-compatible behavior
• document the notification contract, local flow, and future producer path

Why this follow-up is needed

PR #160 added API-key authentication for clients calling Batch Events, but it did not implement notification configuration, recipient resolution, queueing, delivery, or the management UI. This PR provides that missing notification workflow for part of issue #157.

Recipient lists remain owned and maintained by CDA to avoid duplicating user membership in Batch Events. This work therefore depends on the office-scoped user-list API and UI in USACE/cwms-data-api#1734.

User impact

Script administrators can enable email-on-failure for an individual script, select a reusable email template, and select an office-scoped CDA user list. Batch Events resolves current list membership when the failure occurs and queues a rendered message for delivery. The queue contract is intentionally generic enough for later authorized ad-hoc or event-driven email producers.

Validation

• python -m pytest -q — 133 passed
• npm run lint — passed
• npm run build — passed (existing unresolved government-banner SVG and large-chunk warnings remain)

[krowvin]

I caught a few things i'd like to improve on in the UI. Will request review again when i get those in with some screenshots!

[krowvin]

Screenshots

/scripts-manager - Script Details tab

/scripts-manager - Script Failure Email (On)

/scripts-manager - Script Failure Email (Off)

/scripts-manager - Making a new alert for that script

Manual email

CDA User List

Empty Office / one that we do not have permissions to create lists for

When you click "manage user lists" it takes you to the new CDA UI tool for creating these managing them. Screenshots here:
USACE/cwms-data-api#1734 (comment)

/scripts-manager - Preview email/selected template

/events/setup (Edit button above will target a URI to get you to that template)

[krowvin]

Updated the way the manage user list button looks

[MikeNeilson]

what's with the numbering?

sorry, might make sense if I saw the other ones, just too used to doing calendar versioning in these.

[krowvin]

Brandon had setup the repo to be

V - Version migration
1_02_04 - semver
__ - sep the version from description
etc - the rest is any human readable note

My understanding is that flyway uses the values to apply the migrations in order and record them in the schema history.

Looks like he got it from a flyway migration template that CWBI had

• Implement PostgreSQL as job metadata database #30

[MikeNeilson]

Oh yeah, my comment was on the specific version number, not that it was a version.

[MikeNeilson]

Okay names work they are just... weird. but the thing is live so we live with it.

[MikeNeilson]

Might want to put a comment in here that while this by definition has to be "wrong" (the valid forms of email being rather near infinite), that it will match > 99% of what we would normally expect.... you know for the next person like me who will point out these never fully work.

[krowvin]

Indeed it's a fools errand

MDN takes a crack at it too

the parser is a modified version of what MDN has

[krowvin]

Done here:
183a9d6

[MikeNeilson]

still don't like .env files being in the repo... I've said it, that is all.

[krowvin]

As far as I can tell it's common practice with vite https://vite.dev/guide/env-and-mode#env-files

To have the env files committed but to make sure you prefix with VITE_ to ensure there's a clear line between frontend env variables and otherwise.

But to your point I try not to commit a ".env" file.

Not saying don't feel that way, just pointing this out!

[MikeNeilson]

Okay names work they are just... weird. but the thing is live so we live with it.