Open
Description
What happened?
I'm trying to setup a k8s cluster via KubeOne with the vSphere cloudProvider.
The setup of the VMs is done via Terraform, see the output of the command terraform output -json > tf.json below:
# tf.json
{
"kubeone_api": {
"sensitive": false,
"type": [
"object",
{
"apiserver_alternative_names": [
"list",
"string"
],
"endpoint": "string"
}
],
"value": {
"apiserver_alternative_names": [],
"endpoint": "x.x.x.x"
}
},
"kubeone_hosts": {
"sensitive": false,
"type": [
"object",
{
"control_plane": [
"object",
{
"bastion": "string",
"bastion_host_key": "string",
"bastion_port": "number",
"bastion_user": "string",
"cloud_provider": "string",
"cluster_name": "string",
"hostnames": [
"list",
"string"
],
"private_address": [
"tuple",
[]
],
"public_address": [
"tuple",
[
"string",
"string",
"string"
]
],
"ssh_agent_socket": "string",
"ssh_hosts_keys": [
"list",
"string"
],
"ssh_port": "number",
"ssh_private_key_file": "string",
"ssh_user": "string"
}
]
}
],
"value": {
"control_plane": {
"bastion": "",
"bastion_host_key": null,
"bastion_port": 22,
"bastion_user": "",
"cloud_provider": "vsphere",
"cluster_name": "kkp-test",
"hostnames": [
"kkp-test-cp-1",
"kkp-test-cp-2",
"kkp-test-cp-3"
],
"private_address": [],
"public_address": [
"x.x.x.x",
"x.x.x.x",
"x.x.x.x"
],
"ssh_agent_socket": "env:SSH_AUTH_SOCK",
"ssh_hosts_keys": null,
"ssh_port": 22,
"ssh_private_key_file": "",
"ssh_user": "kubeone"
}
}
},
"kubeone_workers": {
"sensitive": false,
"type": [
"object",
{
"kkp-test-pool1": [
"object",
{
"providerSpec": [
"object",
{
"annotations": [
"object",
{
"cluster.k8s.io/cluster-api-autoscaler-node-group-max-size": "string",
"cluster.k8s.io/cluster-api-autoscaler-node-group-min-size": "string",
"k8c.io/operating-system-profile": "string"
}
],
"cloudProviderSpec": [
"object",
{
"allowInsecure": "bool",
"cluster": "string",
"cpus": "number",
"datacenter": "string",
"datastore": "string",
"datastoreCluster": "string",
"diskSizeGB": "number",
"folder": "string",
"memoryMB": "number",
"resourcePool": "string",
"templateVMName": "string",
"vmNetName": "string"
}
],
"operatingSystem": "string",
"operatingSystemSpec": [
"object",
{
"distUpgradeOnBoot": "bool"
}
],
"sshPublicKeys": [
"tuple",
[
"string"
]
]
}
],
"replicas": "number"
}
]
}
],
"value": {
"kkp-test-pool1": {
"providerSpec": {
"annotations": {
"cluster.k8s.io/cluster-api-autoscaler-node-group-max-size": "2",
"cluster.k8s.io/cluster-api-autoscaler-node-group-min-size": "2",
"k8c.io/operating-system-profile": ""
},
"cloudProviderSpec": {
"allowInsecure": true,
"cluster": "CLUSTER",
"cpus": 2,
"datacenter": "DATACENTER",
"datastore": "DATASTORE",
"datastoreCluster": "",
"diskSizeGB": 10,
"folder": "/Customers/TEST/kubermatic/kubeone",
"memoryMB": 2048,
"resourcePool": "Test_pool",
"templateVMName": "ubuntu-22.04-server-cloudimg-kubeone-amd64",
"vmNetName": "NETWORK"
},
"operatingSystem": "ubuntu",
"operatingSystemSpec": {
"distUpgradeOnBoot": false
},
"sshPublicKeys": [
"ecdsa-sha2-nistp521 <REDACTED> kubeone"
]
},
"replicas": 2
}
}
}
}When I run kubeone apply -m kubeone.yml -t tf.json -c credentials.yml I get the following error message at the step Creating worker machines...:
WARN[10:30:51 CET] Task failed, error was: kubernetes: creating *v1alpha1.MachineDeployment kube-system/kkp-test-pool1
admission webhook "machinedeployments.machine-controller.kubermatic.io" denied the request: validation failed: failed to get folder "/Customers/TEST/kubermatic/kubeone": folder '/Customers/TEST/kubermatic/kubeone' not foundExpected behavior
I expect that the worker nodes will be created in the specified vSphere folder.
How to reproduce the issue?
Setup the KubeOne VMs via Terraform and use the following value in the terraform.tfvars file:
folder_name = "/Customers/TEST/kubermatic/kubeone"
What KubeOne version are you using?
$ kubeone version
{
"kubeone": {
"major": "1",
"minor": "6",
"gitVersion": "1.6.0",
"gitCommit": "8b0973ca77856dca920798bbd5ff6b5c0f3f4856",
"gitTreeState": "",
"buildDate": "2023-02-23T19:25:26Z",
"goVersion": "go1.19.6",
"compiler": "gc",
"platform": "linux/amd64"
},
"machine_controller": {
"major": "1",
"minor": "56",
"gitVersion": "v1.56.0",
"gitCommit": "",
"gitTreeState": "",
"buildDate": "",
"goVersion": "",
"compiler": "",
"platform": "linux/amd64"
}
}Provide your KubeOneCluster manifest here (if applicable)
apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
kubernetes: '1.24.8'
cloudProvider:
vsphere: {}
cloudConfig: |
[Global]
secret-name = "vsphere-ccm-credentials"
secret-namespace = "kube-system"
port = "443"
insecure-flag = "1"
[VirtualCenter "VCENTER"]
[Workspace]
server = "SERVER"
datacenter = "DATACENTER"
default-datastore="DATASTORE"
resourcepool-path="Test_pool"
folder = "kubeone"
[Disk]
scsicontrollertype = pvscsi
[Network]
public-network = "NETWORK"What cloud provider are you running on?
VMware vSphere
What operating system are you running in your cluster?
Ubuntu 22.04
Additional information
If I update the value of the key kubeone_workers.value.kkp-test-pool1.cloudProviderSpec.folder in the file tf.json to /DATACENTER/vm/Customers/TEST/kubermatic/kubeone the creation of the worker nodes is working.
I tried to setup the full path for the folder as the value in the terraform.tfvars file (folder_name = "/DATACENTER/vm/Customers/TEST/kubermatic/kubeone").
But with this configuration it fails directly at the Terraform run with the following message:
vsphere_virtual_machine.control_plane[1]: Creating...
╷
│ Error: folder '/DATACENTER/vm/DATACENTER/vm/Customers/TEST/kubermatic/kubeone' not found
│
│ with vsphere_virtual_machine.control_plane[1],
│ on main.tf line 152, in resource "vsphere_virtual_machine" "control_plane":
│ 152: resource "vsphere_virtual_machine" "control_plane" {For me it looks like that the full folder path should be used as value for the key kubeone_workers.value.kkp-test-pool1.cloudProviderSpec.folder in the tf.json file.