Kubedriver compatability with Amazon EKS Cluster

[johnnally]

Hi,

We are investigating using kubedriver with EKS Cluster.
We load a deployment location for the EKS Cluster based on the kubectl config view --raw output.
In the kubectl config view output the user definition section looks similar to below ( i have replaced some info with <> below):

- name: arn:aws:eks:<cluster-location>:<cluster number>:cluster/<cluster-name>
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - <location>
      - eks
      - get-token
      - --cluster-name
      - <cluster-name>
      command: aws

afterwards I added the deployment location to CP4NA:

[ec2-user@ip-172-31-8-141 ~]$ lmctl deployment add --rm brent -i Kubernetes -d 'EKSCluster-New' -p deployment_location.yaml dev deploy --config ~/lmctl-config.yaml
| id     | name   | resourceManager   | infrastructureType   | description    |
|--------+--------+-------------------+----------------------+----------------|
| deploy | deploy | brent             | Kubernetes           | EKSCluster-New |

When I try a simple pod instance installation on EKS Cluster via a CP4NA assembly I get the below error in the kubedriver log:

{"@timestamp": "2021-07-09T09:59:46.426Z", "@version": "1", "message": "[Errno 2] No such file or directory: 'aws': 'aws'", "host": "ib-orch001-kubedriver-028-664b6db4bc-pwh8d", "path": "/usr/local/lib/python3.7/site-packages/kubernetes/config/kube_config.py", "tags": [], "type": "logstash", "thread_name": "MainThread", "level": "ERROR", "logger_name": "root", "traceCtx.processid": "a2d8c669-dc43-44d3-a2a5-9232caf0afc7", "traceCtx.transactionid": "3abd2533-ab67-442d-af12-172c2d2967eb", "traceCtx.taskid": "7"}

afterwards in the kubedriver logs you can see some more stack messages that mention openshift client (/usr/local/lib/python3.7/site-packages/openshift/dynamic/client.py) etc,...

{"@timestamp": "2021-07-09T09:59:46.426Z", "@version": "1", "message": "[Errno 2] No such file or directory: 'aws': 'aws'", "host": "ib-orch001-kubedriver-028-664b6db4bc-pwh8d", "path": "/usr/local/lib/python3.7/site-packages/kubernetes/config/kube_config.py", "tags": [], "type": "logstash", "thread_name": "MainThread", "level": "ERROR", "logger_name": "root", "traceCtx.processid": "a2d8c669-dc43-44d3-a2a5-9232caf0afc7", "traceCtx.transactionid": "3abd2533-ab67-442d-af12-172c2d2967eb", "traceCtx.taskid": "7"}
{"@timestamp": "2021-07-09T09:59:46.453Z", "@version": "1", "message": "API error occurred: 403\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '75f4a366-2504-41e9-889b-cac012c6b48f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e8d368dd-67e7-4875-8826-7975db4db675', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7f4e1ebe-d3e5-47f3-85c2-f760ba425c6a', 'Date': 'Fri, 09 Jul 2021 09:59:46 GMT', 'Content-Length': '189'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"forbidden: User \\\\\"system:anonymous\\\\\" cannot get path \\\\\"/apis\\\\\"\",\"reason\":\"Forbidden\",\"details\":{},\"code\":403}\\n'\nOriginal traceback: \n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/client.py\", line 42, in inner\n    resp = func(self, *args, **kwargs)\n\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/client.py\", line 247, in request\n    _return_http_data_only=params.get('_return_http_data_only', True)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py\", line 345, in call_api\n    _preload_content, _request_timeout)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py\", line 176, in __call_api\n    _request_timeout=_request_timeout)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py\", line 366, in request\n    headers=headers)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py\", line 241, in GET\n    query_params=query_params)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py\", line 231, in request\n    raise ApiException(http_resp=r)\n", "host": "ib-orch001-kubedriver-028-664b6db4bc-pwh8d", "path": "/usr/local/lib/python3.7/site-packages/ignition/api/exceptions.py", "tags": [], "type": "logstash", "thread_name": "MainThread", "level": "ERROR", "logger_name": "ignition.api.exceptions", "stack_trace": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/client.py\", line 42, in inner\n    resp = func(self, *args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/client.py\", line 247, in request\n    _return_http_data_only=params.get('_return_http_data_only', True)\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py\", line 345, in call_api\n    _preload_content, _request_timeout)\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py\", line 176, in __call_api\n    _request_timeout=_request_timeout)\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py\", line 366, in request\n    headers=headers)\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py\", line 241, in GET\n    query_params=query_params)\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py\", line 231, in request\n    raise ApiException(http_resp=r)\nkubernetes.client.rest.ApiException: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '75f4a366-2504-41e9-889b-cac012c6b48f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e8d368dd-67e7-4875-8826-7975db4db675', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7f4e1ebe-d3e5-47f3-85c2-f760ba425c6a', 'Date': 'Fri, 09 Jul 2021 09:59:46 GMT', 'Content-Length': '189'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"forbidden: User \\\\\"system:anonymous\\\\\" cannot get path \\\\\"/apis\\\\\"\",\"reason\":\"Forbidden\",\"details\":{},\"code\":403}\\n'\n\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/flask/app.py\", line 1499, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/usr/local/lib/python3.7/site-packages/flask/app.py\", line 1485, in dispatch_request\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)\n  File \"/usr/local/lib/python3.7/site-packages/connexion/decorators/decorator.py\", line 48, in wrapper\n    response = function(request)\n  File \"/usr/local/lib/python3.7/site-packages/connexion/decorators/uri_parsing.py\", line 144, in wrapper\n    response = function(request)\n  File \"/usr/local/lib/python3.7/site-packages/connexion/decorators/validation.py\", line 184, in wrapper\n    response = function(request)\n  File \"/usr/local/lib/python3.7/site-packages/connexion/decorators/parameter.py\", line 121, in wrapper\n    return function(**kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/ignition/service/resourcedriver.py\", line 209, in execute_lifecycle\n    execute_response = self.service.execute_lifecycle(lifecycle_name, driver_files, system_properties, resource_properties, request_properties, associated_topology, deployment_location)\n  File \"/usr/local/lib/python3.7/site-packages/ignition/service/resourcedriver.py\", line 276, in execute_lifecycle\n    execute_response = self.handler.execute_lifecycle(lifecycle_name, driver_files_tree, PropValueMap(system_properties), PropValueMap(resource_properties), PropValueMap(request_properties), associated_topology, deployment_location)\n  File \"/usr/local/lib/python3.7/site-packages/kubedriver/resourcedriver/api_handler.py\", line 47, in execute_lifecycle\n    request_id = self.kegd_strategy_manager.apply_kegd_strategy(kube_location, keg_name, kegd_strategy, lifecycle_name, kegd_files, render_context)\n  File \"/usr/local/lib/python3.7/site-packages/kubedriver/kegd/manager.py\", line 33, in apply_kegd_strategy\n    context = self.context_factory.build(kube_location)\n  File \"/usr/local/lib/python3.7/site-packages/kubedriver/locationcontext/factory.py\", line 13, in build\n    api_ctl = self.api_ctl_factory.build(kube_location)\n  File \"/usr/local/lib/python3.7/site-packages/kubedriver/kubeclient/os_api_ctl_factory.py\", line 7, in build\n    return OpenshiftApiController(kube_location.client, default_namespace=kube_location.default_object_namespace)\n  File \"/usr/local/lib/python3.7/site-packages/kubedriver/kubeclient/os_api_ctl.py\", line 10, in __init__\n    self.dynamic_client = DynamicClient(base_kube_client)\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/client.py\", line 71, in __init__\n    self.__discoverer = discoverer(self, cache_file)\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/discovery.py\", line 259, in __init__\n    Discoverer.__init__(self, client, cache_file)\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/discovery.py\", line 31, in __init__\n    self.__init_cache()\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/discovery.py\", line 79, in __init_cache\n    self.discover()\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/discovery.py\", line 263, in discover\n    self.__resources = self.parse_api_groups(request_resources=False)\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/discovery.py\", line 131, in parse_api_groups\n    groups_response = self.client.request('GET', '/{}'.format(DISCOVERY_PREFIX)).groups\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/client.py\", line 44, in inner\n    raise api_exception(e)\nopenshift.dynamic.exceptions.ForbiddenError: 403\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '75f4a366-2504-41e9-889b-cac012c6b48f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e8d368dd-67e7-4875-8826-7975db4db675', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7f4e1ebe-d3e5-47f3-85c2-f760ba425c6a', 'Date': 'Fri, 09 Jul 2021 09:59:46 GMT', 'Content-Length': '189'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"forbidden: User \\\\\"system:anonymous\\\\\" cannot get path \\\\\"/apis\\\\\"\",\"reason\":\"Forbidden\",\"details\":{},\"code\":403}\\n'\nOriginal traceback: \n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/client.py\", line 42, in inner\n    resp = func(self, *args, **kwargs)\n\n  File \"/usr/local/lib/python3.7/site-packages/openshift/dynamic/client.py\", line 247, in request\n    _return_http_data_only=params.get('_return_http_data_only', True)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py\", line 345, in call_api\n    _preload_content, _request_timeout)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py\", line 176, in __call_api\n    _request_timeout=_request_timeout)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py\", line 366, in request\n    headers=headers)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py\", line 241, in GET\n    query_params=query_params)\n\n  File \"/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py\", line 231, in request\n    raise ApiException(http_resp=r)\n\n", "lineno": 38, "process": 11}

should kubedriver be expected to work with an Amazon EKS Cluster ?
or would require a change request for the driver etc..,

[dvaccarosenna]

Should kubedriver be expected to work with an Amazon EKS Cluster ?

I'd say yes, we'd like it to, but it's never been tested. The assumption has been that the kubernetes client we use supports everything kubectl does, as it uses the same config file. However, based on No such file or directory: 'aws': 'aws'" it's possible that EKS authentication requires additional libraries to be installed? As this is not installed, I assume authentication cannot be completed but the client continue anyway, which causes the Forbidden error seen later in the logs.

Not sure if the "Create kubeconfig manually" section in this will help: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html

Otherwise, we may have to look into adding the aws command line tool into the docker image?

[johnnally]

Hi Daniel,

we actually built a kubedriver image with asw cli installed on it.
it got us past the aws cli error, and now we are restricted by user permissions it seems on amazon eks side of things.
we will be looking into more detail later, but for now this is fine and can be closed.

Thanks,
John