Configure automatic restart for OOM-terminated pods in JupyterHub and Jobs

[Copilot]

Pods terminated by OOM errors in JupyterHub and batch workloads did not restart automatically, requiring manual intervention.

Changes

JupyterHub singleuser pods (MAIA_scripts/MAIA_create_JupyterHub_config.py)

• Set restartPolicy: OnFailure via extraPodConfig in generated JupyterHub values
• Notebook pods now restart automatically on OOM or other failures

MAIAKubeGate Job workloads (charts/maiakubegate/)

• Made restartPolicy and backoffLimit configurable
• Defaults: restartPolicy: OnFailure, backoffLimit: 3
• User workloads retry up to 3 times on failure

Kaniko build Jobs (charts/maiakubegate-kaniko/)

• Made restart behavior configurable
• Defaults: restartPolicy: Never, backoffLimit: 0 (no retries for builds)

Configuration example:

# JupyterHub values now include:
"singleuser": {
    "extraPodConfig": {
        "restartPolicy": "OnFailure"
    },
    ...
}

# Helm values for Jobs:
restartPolicy: OnFailure  # or Never
backoffLimit: 3           # retry attempts

All changes backward compatible via Helm default value patterns. Documentation added in docs/OOM_RESTART_CONFIGURATION.md.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Pods terminated with OOM should restart automatically (JupyterHub/Kubeflow)</issue_title>
<issue_description>## Problem
Currently, when a pod started through JupyterHub or Kubeflow in the MAIA platform runs out of memory (OOM), it does not restart automatically. This results in the need for manual intervention to restart the affected pods, interrupting user workflows and reducing reliability.

Expected Behavior

Pods that are terminated due to OOM (Out Of Memory) errors should be automatically restarted by Kubernetes without requiring manual action from users or administrators.

Steps to Reproduce

1. Launch a pod via JupyterHub or Kubeflow in the MAIA cluster.
2. Perform memory-intensive operations until the pod exceeds its memory limit and is terminated with an OOM error.
3. Observe that the pod does not automatically restart and remains in a failed state.

Desired Solution

• Configure MAIA platform pod templates, deployment, or JupyterHub/Kubeflow integrations so that OOM pods always restart automatically.
• Potentially add or adjust the restartPolicy (should be Always for most interactive workloads).
• Validate expected behavior by forcing several OOM events and confirming pods are restarted.

Additional Context

• Applies to pods started both from JupyterHub and via Kubeflow pipelines/interactive sessions.
• Consider if default behavior differs between Deployment vs Job resources, and document where relevant.

---

Please investigate and update the configuration as needed so that user pods do not require manual restart after OOM events.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Pods terminated with OOM should restart automatically (JupyterHub/Kubeflow) #107

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.