Better disk limit strategy

[rgaudin]

We are currently solely relying on local hostPath to store application data. So app data is stored on the k8s nodes and applications are thus tied to a specific node.

Because k8s is not meant to be used this way (it contradicts the core principle of k8s or moving pods across nodes) k8s is very aggressive when disk pressure is detected (90% of disk usage I think) and kills the running pods.

Our strategy so far is:

• have plenty of extra disk space on each node (based on expected/guessed data usage)
• have an image prune policy to purge OCI images stuff when disk usage reaches a threshold
• manually look at disk space every week in the routine.

Given we accidentally triggered DiskPressure while restoring a backup the other day (and k8s killed all pods and the ingress did not restart for a different reason), we should start discussing better strategies.

[benoit74]

Quite important indeed, probably even urgent

[siddheshwar-9897]

We can replace the current hostPath-based storage with a distributed object storage solution like MinIO, which is S3-compatible and deployable within our Kubernetes cluster.

[kelson42]

I see following alternatives:

• Object storage, via s3fs
• Block storage via a SAN
• Network file system, via NFS or CIFS

Do we have possible solutions for all these alternatives?

[benoit74]

None of these strategies are "k8s native", I would prefer to look at something more common in the k8s ecosystem unless we have strong arguments to believe these are better.