CIS Hardening Guide

.github/styles/Google/Headings.yml

@@ -8,6 +8,9 @@ indicators:
   - ":"
 exceptions:
   - Azure
+  - API Server
+  - Authorization
+  - CIS Hardening
   - CLI
   - Cosmos
   - Docker

vcluster/learn-how-to/hardening-guide/2-etcd.mdx

@@ -0,0 +1,170 @@
+---
+title: Self assessment guide - ETCD configuration
+sidebar_label: Etcd
+description: Self assessment guide to validate ETCD configuration
+---
+
+This section covers security areas related to etcd configuration, including:
+- Encryption of sensitive data at rest on server, applications and in transit.
+
+_Assessment focus for vCluster_: Key areas include verifying correct authentication mechanisms are used and safeguarding
+the data at rest and in transit via TLS encryption.
+
+## 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated)
+
+**Result:** PASS
+
+**Audit:**
+Run the following command against the vCluster pod:
+```bash
+kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd
+```
+Verify that the --cert-file and the --key-file arguments are set as appropriate.
+
+**Expected Result:**
+```
+'--cert-file' and '--key-file' arguments are appropriately set
+```
+
+**Returned Value:**
+```bash
+31 root      2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster
+```
+
+## 2.2 Ensure that the --client-cert-auth argument is set to true (Automated)
+
+**Result:** PASS
+
+**Audit:**
+Run the following command against the vCluster pod:
+```bash
+kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd
+```
+Verify that the --client-cert-auth argument is set to true.
+
+**Expected Result:**
+```
+'--client-cert-auth' is set to 'true'
+```
+
+**Returned Value:**
+```bash
+31 root      2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster
+```
+
+## 2.3 Ensure that the --auto-tls argument is not set to true (Automated)
+
+**Result:** PASS
+
+**Audit:**
+Run the following command against the vCluster pod:
+```bash
+kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd
+```
+Verify that if the --auto-tls argument exists, it is not set to true.
+
+**Expected Result:**
+```
+'--auto-tls' argument does not exist
+```
+
+**Returned Value:**
+```bash
+31 root      2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster
+```
+
+## 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated)
+
+**Result:** PASS
+
+**Audit:**
+Run the following command against the vCluster pod:
+```bash
+kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd
+```
+Verify that the --peer-cert-file and --peer-key-file arguments are set as appropriate.
+
+**Expected Result:**
+```
+'--peer-cert-file' and '--peer-key-file' arguments are appropriately set
+```
+
+**Returned Value:**
+```bash
+31 root      2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster
+```
+
+## 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated)
+
+**Result:** PASS
+
+**Audit:**
+Run the following command against the vCluster pod:
+```bash
+kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd
+```
+Verify that the --peer-client-cert-auth argument is set to true.
+
+**Expected Result:**
+```
+'--peer-client-cert-auth' is set to 'true'
+```
+
+**Returned Value:**
+```bash
+31 root      2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster
+```
+
+## 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated)
+
+**Result:** PASS
+
+**Audit:**
+Run the following command against the vCluster pod:
+```bash
+kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd
+```
+Verify that if the --peer-auto-tls argument exists, it is not set to true.
+
+**Expected Result:**
+```
+'--peer-auto-tls' argument does not exist
+```
+
+**Returned Value:**
+```bash
+31 root      2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster
+```
+
+<!-- vale Google.Headings = NO -->
+## 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual)
+<!-- vale Google.Headings = YES -->
+
+**Result:** PASS
+
+**Audit:**
+Run the following command against the vCluster pod:
+```bash
+kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd
+```
+Note the file referenced by the --trusted-ca-file argument.
+
+Run the following command and note the file referenced by '--client-ca-file'
+```bash
+kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver
+```
+Verify that the file referenced by the --client-ca-file for apiserver is different from the --trusted-ca-file used by etcd.
+
+**Expected Result:**
+```
+The file referenced by the --client-ca-file for api-server is different from the --trusted-ca-file
+```
+
+**Returned Value:**
+```bash
+31 root      2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster
+```
+
+```bash
+47 root      6:43 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/16 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer= https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file= /data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false
+```

vcluster/learn-how-to/hardening-guide/3-control-plane.mdx

@@ -0,0 +1,137 @@
+---
+title: Self assessment guide - Control Plane Configuration
+sidebar_label: Control Plane Configuration
+description: Self assessment guide to validate control plane configuration
+---
+
+This section covers cluster-wide security areas, including:
+- Authentication and authorization mechanisms
+- Audit logging configuration
+
+_Assessment focus for vCluster_: Key areas include verifying audit logging is enabled, and alternative mechanisms such as OIDC are used.
+
+
+## 3.1 Authentication and Authorization
+
+### 3.1.1 Client certificate authentication should not be used for users (Manual)
+**Result:** WARN
+
+**Remediation:** Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of client certificates.
+
+### 3.1.2 Service account token authentication should not be used for users (Manual)
+**Result:** WARN
+
+**Remediation:** Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of service account tokens.
+
+### 3.1.3 Bootstrap token authentication should not be used for users (Manual)
+**Result:** WARN
+
+**Remediation:** Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of bootstrap tokens.
+
+## 3.2 Logging
+
+### 3.2.1 Ensure that a minimal audit policy is created (Automated)
+**Result:** PASS
+
+**Remediation:** 
+Follow the [Kubernetes Documentation](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/) and create a config map with a minimal audit policy
+```yaml title="audit-config-configmap.yaml"
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: audit-config
+  namespace: vcluster-my-vcluster
+data:
+  audit-policy.yaml: |
+    apiVersion: audit.k8s.io/v1
+    kind: Policy
+    rules:
+    - level: Metadata
+```
+
+Pass the below configuration as arguments to the API Server while creating the vCluster as:
+```yaml title="vcluster.yaml"
+controlPlane:
+  distro:
+    k8s:
+      enabled: true
+      apiServer:
+        extraArgs: 
+        - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
+  statefulSet:
+    persistence:
+      addVolumes:
+      - name: audit-policy
+        configMap:
+          name: audit-config
+      addVolumeMounts:
+      - name: audit-policy
+        mountPath: /etc/kubernetes
+```
+
+**Audit:**
+Create the vCluster using the above values file.
+```bash
+vcluster create my-vcluster -f vcluster.yaml --connect=false
+```
+
+Run the following command against the vCluster pod:
+```bash
+kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver
+```
+Verify that the --audit-policy-file is set.
+
+**Expected Result:**
+```
+'--audit-policy-file' is present
+```
+
+**Returned Value:**
+```bash
+26 root      0:08 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-policy-file=/etc/kubernetes/audit-policy.yaml
+```
+
+### 3.2.2 Ensure that the audit policy covers key security concerns (Manual)
+**Result:** WARN
+
+**Remediation:** Review the audit policy provided for the cluster and ensure that it covers at least the following areas :-
+- Access to Secrets managed by the cluster. Care should be taken to only log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, to avoid the risk of logging sensitive data.
+- Modification of pod and deployment objects.
+- Use of pods/exec, pods/portforward, pods/proxy and services/proxy.
+
+For most requests, minimally logging at the Metadata level is recommended (the most basic level of logging).
+Consider modification of the audit policy in use on the cluster to include these items, at a minimum. A sample policy that satisfies this criteria is as below.
+
+```yaml title="audit-config-configmap.yaml"
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: audit-config
+  namespace: vcluster-my-vcluster
+data:
+  audit-policy.yaml: |
+    apiVersion: audit.k8s.io/v1
+    kind: Policy
+    rules:
+    - level: Metadata
+      resources:
+      - group: ""
+        resources: ["secrets", "configmaps"]
+      - group: "authentication.k8s.io"
+        resources: ["tokenreviews"]
+    - level: RequestResponse
+      resources:
+      - group: ""
+        resources: ["pods"]
+      - group: "apps"
+        resources: ["deployments", "replicasets"]
+      verbs: ["create", "update", "patch", "delete"]
+    - level: RequestResponse
+      resources:
+      - group: ""
+        resources: ["pods/exec", "pods/portforward", "pods/proxy", "services/proxy"]
+    - level: Metadata
+      omitStages:
+      - RequestReceived
+```
+