[FIX] Terraform provider version inconsistency within stages

[smokestacklightnin]

Reference Issues or PRs

Fixes #2614

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

How to test this PR?

Any other comments?

The main change in this PR is that Terraform/OpenTofu versions and required providers are removed from the stage templates and instead injected via stages.tf_objects with version data from constants.py. stages.tf_objects.NebariOpentofuRequiredVersion and stages.tf_objects.NebariOpentofuRequiredProvider were added.

Now all subclasses of NebariTerraformStage present with a tf_objects method except terraform_state will call include the result of super().tf_objects in their return value:

class FooStage(NebariTerraformStage):
    def tf_objects(self) -> List[Dict]:
        return [
            *super().tf_objects(),
            ...,
        ]    

This way we can also inject NebariOpentofuRequiredVersion into NebariTerraformStage and have subclasses inherit it.

[smokestacklightnin]

I've executed the changes I intended to make, however, when running Nebari, submodules are not inheriting required providers from the parent module. In particular, this is happening in some of the following modules:

./kubernetes_ingress/template/modules/kubernetes/ingress/
./kubernetes_services/template/modules/kubernetes/cephfs-mount/
./kubernetes_services/template/modules/kubernetes/
./kubernetes_services/template/modules/kubernetes/nfs-mount/
./kubernetes_services/template/modules/kubernetes/forwardauth/
./kubernetes_services/template/modules/kubernetes/nfs-server/
./kubernetes_services/template/modules/kubernetes/services/jupyterhub-ssh/
./kubernetes_services/template/modules/kubernetes/services/jupyterhub/
./kubernetes_services/template/modules/kubernetes/services/keycloak-client/
./kubernetes_services/template/modules/kubernetes/services/
./kubernetes_services/template/modules/kubernetes/services/redis/
./kubernetes_services/template/modules/kubernetes/services/monitoring/
./kubernetes_services/template/modules/kubernetes/services/monitoring/loki/
./kubernetes_services/template/modules/kubernetes/services/postgresql/
./kubernetes_services/template/modules/kubernetes/services/rook-ceph/
./kubernetes_services/template/modules/kubernetes/services/dask-gateway/
./kubernetes_services/template/modules/kubernetes/services/minio/
./kubernetes_services/template/modules/kubernetes/services/argo-workflows/
./kubernetes_services/template/modules/kubernetes/services/conda-store/
./nebari_tf_extensions/template/modules/nebariextension/
./nebari_tf_extensions/template/modules/helm-extensions/
./kubernetes_initialize/template/modules/
./kubernetes_initialize/template/modules/initialization/
./kubernetes_initialize/template/modules/cluster-autoscaler/
./kubernetes_initialize/template/modules/traefik_crds/
./kubernetes_initialize/template/modules/extcr/
./kubernetes_initialize/template/modules/nvidia-installer/
./kubernetes_keycloak/template/modules/kubernetes/keycloak-helm/

Just to see if it would work, I tried adding the correct required providers to those locations. It worked, so this suggest that it might be a good option to be able to dynamically inject required providers from constants.py this would be generalizing what is currently done at the root module to submodules as well.

@viniciusdc Do you have any input?

CC: @marcelovilla

[smokestacklightnin]

My honest opinion is that it is easier to inject submodule required providers manually now and change to using inherited required providers from parent modules in the future if/when OpenTofu supports this, rather than waiting for another project to merge our changes. I also don't know Go, but am activetly trying to learn.

The code I have written would be easy to remove for the injection process in submodules if OpenTofu started supporting version requirement inheritance in the future. This is because it requires minimal additional machinery, just _tf_objects_required_providers.

We need this feature soon because other issues depend on updating versions (#2806, #2870, #2872).

Ideas? @marcelovilla @viniciusdc @dcmcand

[viniciusdc]

I will have a look, we need to make the issues with the double version we noticed before are not happening anymore. Thanks for all the fantastic work so far @smokestacklightnin

[smokestacklightnin]

I will have a look, we need to make the issues with the double version we noticed before are not happening anymore. Thanks for all the fantastic work so far @smokestacklightnin

Pinging @viniciusdc

[viniciusdc]

To be included in 2025.1.2 release, for ref. check the parent issue.

[viniciusdc]

I am reviewing this now

[viniciusdc]

I am encountering an error runing this localyly

[tofu]: Providers are signed by their developers.
[tofu]: If you'd like to know more about provider signing, you can read about it here:
[tofu]: https://opentofu.org/docs/cli/plugins/signing/
[tofu]: ╷
[tofu]: │ Error: Failed to query available provider packages
[tofu]: │ 
[tofu]: │ Could not retrieve the list of available versions for provider
[tofu]: │ hashicorp/keycloak: provider registry registry.opentofu.org does not have a
[tofu]: │ provider named registry.opentofu.org/hashicorp/keycloak
[tofu]: │ 
[tofu]: │ Did you intend to use mrparkers/keycloak? If so, you must specify that
[tofu]: │ source address in each module which requires that provider. To see which
[tofu]: │ modules are currently depending on hashicorp/keycloak, run the following
[tofu]: │ command:
[tofu]: │     tofu providers
[tofu]: │ 
[tofu]: │ If you believe this provider is missing from the registry, please submit a
[tofu]: │ issue on the OpenTofu Registry
[tofu]: │ https://github.com/opentofu/registry/issues/new/choose

but I remember this been addressed on another PR a while ago, so maybe the same fix needs to be moved here as well. It might be related to how versions were handled currently. Overall all constraints are there.

Also, there are no tests to make sure we are actually getting the correct versions and this code is doing what's supposed to do, so I will include then later this week -- since CI runs a local deployment with local terraform state, we can parse the state files when they exists and compare the installed version with it's expected constraints.

[viniciusdc]

If I understood correctly, this was included since we also wanted to render the _tf.json files for the module versions right? I've been testing this locally so far today, and even without this the actual versions have been correctly used by each stage and its underlying modules. I will follow on what we discussed back then to refresh my head around why we went this route, but we might not need ti anymore :)

[viniciusdc]

I think originally, I had in mind a *RequiredProviders class for accommodating all providers for each stage in a single sweep, but this also does the job.