Support a configmap-backed config dir for configuring the bootc-operator

[jlebon]

We're adding a bunch of CLI switches to the operator which would require users to edit the manifest. And then those manifests would need to be reconciled during updates.

Let's support a config dir (e.g. at /etc/bootc-operator/) which supports the same switches as key/vals, and then add to the manifests optional configmap volume mounts for a e.g. bootc-operator-config configmap.

Then, admins can do e.g.

apiVersion: v1
kind: ConfigMap
metadata:
  name: bootc-operator-config
data:
  allow-insecure-registry: true
  ...

to configure the operator.

[jlebon]

I'm sure there's prior art for other ways to solve the config separation problem. Worth doing a sanity-check around to make sure there isn't a cleaner more established pattern we should follow instead.

[alicefr]

We will need to avoid that the operator patches its own deployment in order to apply the new flags.

Additionally, instead of using configmap, I would introduce a CRD, in this way, it can be versioned

[jlebon]

We will need to avoid that the operator patches its own deployment in order to apply the new flags.

Not sure I follow this. The configmap would be a way to get away from patching entirely.

Additionally, instead of using configmap, I would introduce a CRD, in this way, it can be versioned

Hmm, maybe. It would also give us type enforcement. This is where looking at prior art would be helpful. Though configmaps are pretty much made for this and well-understood. Re. versioning, I think the main stability promise is through the bootcnodepool. For CLI flags/configmap keys, I think we can more aggressively deprecate and phase things out as needed.

[alicefr]

We will need to avoid that the operator patches its own deployment in order to apply the new flags.

Not sure I follow this. The configmap would be a way to get away from patching entirely.

I mean that when the configmap changes we shouldn't patch the controller for updating of the corresponding fields on the command line. Probably, once we introduce the configmap/crd we should deprecate the flags in order to have a single way to configure the controller.

Additionally, instead of using configmap, I would introduce a CRD, in this way, it can be versioned

Hmm, maybe. It would also give us type enforcement. This is where looking at prior art would be helpful. Though configmaps are pretty much made for this and well-understood. Re. versioning, I think the main stability promise is through the bootcnodepool. For CLI flags/configmap keys, I think we can more aggressively deprecate and phase things out as needed.

We should investigate but I do prefer an explicit API that we can read in the yaml instead of a way to parse the config map data content into the code.

[jlebon]

I mean that when the configmap changes we shouldn't patch the controller for updating of the corresponding fields on the command line. Probably, once we introduce the configmap/crd we should deprecate the flags in order to have a single way to configure the controller.

Right yeah, how this would work is that at startup we combine both CLI flags and configmaps args to determine the overall config we run with. Changes to the configmap require restarting the operator.

We should investigate but I do prefer an explicit API that we can read in the yaml instead of a way to parse the config map data content into the code.

Yeah, that's a good point. (I miss Rust's clap...).

[jlebon]

One easy approach is to actually support env vars for every CLI flag we support, similar to what we do in bink. Then in the manifests, we project configmap keys to those env vars (but again, marked optional). I think that would require minimal changes to the operator itself.

Edit: see the env example in https://kubernetes.io/docs/concepts/configuration/configmap/#configmaps-and-pods

[alicefr]

I mean that when the configmap changes we shouldn't patch the controller for updating of the corresponding fields on the command line. Probably, once we introduce the configmap/crd we should deprecate the flags in order to have a single way to configure the controller.

Right yeah, how this would work is that at startup we combine both CLI flags and configmaps args to determine the overall config we run with. Changes to the configmap require restarting the operator.

That's what I would like to avoid. Who should restart the controller? A controller patching its own deployment sounds like an anti-pattern. On the the other hand, it could simply watch for the new CRD and if update, then reloading the configuration at runtime

[jlebon]

That's what I would like to avoid. Who should restart the controller? A controller patching its own deployment sounds like an anti-pattern.

I don't think anything would restart the controller. It would be up to the user to restart it if they changed the configmap.

But I think #92 (comment) is probably the simplest path forward. Needs some playing around with that idea to see how it feels.