skip db user actions when its secret failed to sync on update

[FxKu]

Bug report

On UPDATE events the secrets sync can exit early on error but the database role sync will still happen. This will potentially override database passwords with what was generated in the initUsers step. Environment variables don't get overridden and Patroni can still operate, but apps (and even the operator when it happens to postgres user) will get "locked out".

Proposed solution

We could decide to not call syncRoles if the secret sync fails. We already do so if the initUsers step fails. During cluster creation and sync any error in SyncSecrets stops the entire sync loop as well. Only during update we care to process resources as much as possible.

Therefore, I chose a more fine-grained approach:

1. still try to sync roles when the secret sync worked for them.
2. Instead of existing the function errors are collected and returned at the end.
3. I've added a new field in the pgUser struct to remember if the user is degraded or not.
4. This logic will also be applied for cluster CREATE and SYNC events, meaning no database user would get created if the secret creation failed.

More minor changes

• The updateSecret function now returns the updated secret so callers can decide to update the cluster secret map.
• The sync test got extended with a case that would lead to the bug where syncSecrets is quit too soon.

[idanovinda]

👍

[FxKu]

👍