Add cleanup-on-delete ArgoCD PreDelete hook for RHOAI 2.x and 3.x

[lautou]

Summary

• Adds a cleanup-on-delete Kustomize Component to both instance-2.x/components/ and instance-3.x/components/ that implements an ArgoCD PreDelete hook running the official Red Hat RHOAI uninstall procedure
• Enables the component by default in all instance overlays (11 × 2.x, 1 × 3.x) — cleanup is not optional since stuck namespaces and orphaned webhooks affect every user who deletes the RHOAI Application
• Zero runtime footprint: all resources carry the PreDelete hook annotation and are only created/executed at Application deletion time

Problem

Without this component, ArgoCD deletes RHOAI manifests in arbitrary order, which consistently results in:

• Namespaces (redhat-ods-operator, redhat-ods-applications, redhat-ods-monitoring) stuck in Terminating
• Orphaned ValidatingWebhookConfiguration and MutatingWebhookConfiguration resources that block future reinstalls

How it works

A PreDelete Job runs in openshift-gitops (not in the RHOAI namespaces, so it survives their deletion) and executes a 10-step cleanup:

1. Delete DataScienceCluster CR (triggers operator cleanup)
2. Wait 60 s for operator cleanup to complete
3. Delete operator Subscription
4. Delete operator ClusterServiceVersion
5. Wait for operator pods to terminate
6. Delete OperatorGroup
7. Delete all RHOAI ValidatingWebhookConfiguration / MutatingWebhookConfiguration resources
8. Delete RHOAI namespaces (redhat-ods-operator, redhat-ods-applications, redhat-ods-monitoring)
9. Delete RHOAI CRDs (opendatahub.io, kubeflow.org) with finalizer fallback
10. Verify cleanup and report status

The Job exits 0 even when warnings occur, so ArgoCD always proceeds with Application deletion. backoffLimit: 0 means no retry; any remaining resources are cleaned up by ArgoCD's normal deletion.

Resources created

All are PreDelete hooks with hook-delete-policy: HookSucceeded,HookFailed:

Resource	Namespace	sync-wave
ServiceAccount/rhoai-cleanup	openshift-gitops	-1
ClusterRole/rhoai-cleanup	cluster-scoped	-1
ClusterRoleBinding/rhoai-cleanup	cluster-scoped	-1
Job/delete-rhoai-resources	openshift-gitops	0

Test plan

• Installed RHOAI 3.3.1 via fast-3.x channel on OCP 4.20 (fresh cluster)
• Verified all 3 RHOAI namespaces active and DataScienceCluster Ready
• Applied cleanup-on-delete resources simulating ArgoCD PreDelete hook
• Job completed in ~3m11s with exit 0 — all namespaces, CRDs, and webhooks removed
• dscinitializations CRD finalizer fallback triggered and handled correctly
• All 260 kustomization.yaml files pass format validation (check-kustomization-format.sh)
• All manifests build successfully (validate_manifests.sh)

🤖 Generated with Claude Code

[lautou]

@strangiato can you review and merge?

[strangiato]

To be honest the approach here scares me.

There are a lot of very unsafe practices that I would advise against.

I would highly recommend simplifying the uninstall to cover as basic uninstall as covered by the official docs here:
https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.4/html-single/installing_and_uninstalling_openshift_ai_self-managed/index#uninstalling-openshift-ai-self-managed-using-cli_uninstalling-openshift-ai-self-managed

There appears to be a ton of brute force that is happening in the script that should not be necessary and I am concerned that was applied because of other unsafe practices.

In case you are curious, the following is a set of instructions I put together to do a complete manual uninstall of 2.25 for a customer:
https://github.com/redhat-ai-services/ai-accelerator/blob/main/documentation/rhoai-2.25-uninstall.md

[lautou]

Thanks for the detailed feedback and the uninstall doc — very helpful. You're right, the brute force (forced deletion, finalizer patching) was a symptom of deleting things in the wrong order. I'll rework the script to follow the official procedure: delete user workload CRs first, then DSC, then subscription/CSV, then explicit CRD list by name. No grep, no force. Will push an update shortly.

[lautou]

Thanks again for the detailed review — every point was valid and I've reworked the entire script based on your feedback.

What changed:

• Script structure: moved to a separate delete-rhoai-resources.sh file mounted via configMapGenerator, following the aws-gpu-machineset pattern you referenced
• Deletion order fixed: added Step 0 that deletes all user workload CRs across all namespaces first (InferenceServices, Notebooks, RayJobs, Kueue workloads, TrustyAI, etc.) before touching the DSC — this is the root cause of why the original version needed force deletion and finalizer patching
• Explicit CRD list: replaced the grep-based approach with an explicit list taken directly from your rhoai-2.25-uninstall.md — thank you for that reference, it was exactly what was needed
• No more force deletion: removed --grace-period=0 --force
• No more finalizer patching: removed completely — as you said, if you need it, you've done something else wrong
• No more arbitrary sleep: removed the 60s grace period

Tested end-to-end on OCP 4.20:

• RHOAI 3.3.1 (fast-3.x): clean uninstall in 55s ✓
• RHOAI 3.3.2 (stable-3.x): clean uninstall in 51s ✓

All namespaces, CRDs, and webhooks removed cleanly with no forced operations.

[lautou]

@strangiato can you review?

[lautou]

@strangiato can you kindly review?