diff --git a/evals/azure-skills/azure-kubernetes/eval.yaml b/evals/azure-skills/azure-kubernetes/eval.yaml index 42603f222..961e7a1e8 100644 --- a/evals/azure-skills/azure-kubernetes/eval.yaml +++ b/evals/azure-skills/azure-kubernetes/eval.yaml @@ -142,6 +142,26 @@ stimuli: config: pattern: "(?i)fatal error|unhandled exception|stack trace" + # ── routing-aks-review-prompt ── + # Jest: "invokes azure-kubernetes skill for AKS cluster review" + # Assertions: softCheckSkill + isSkillInvoked (invocation rate ≥ 80%) + - name: "Routing: AKS cluster review" + prompt: "I want to review my AKS cluster against best practices" + tags: + type: integration + tier: full + cost: llm + area: routing + earlyTerminate: '[{"type":"skill-call","skill":"azure-kubernetes"},{"type":"tool-call-count","count":3}]' + graders: + - type: skill-invocation + config: + required: + - azure-kubernetes + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + # ── day0-guidance-output-prompt ── # Jest: "responds with Day-0 vs Day-1 guidance" # Expects response mentioning tier, networking, or API server diff --git a/plugins/azure-skills/skills/azure-kubernetes/SKILL.md b/plugins/azure-skills/skills/azure-kubernetes/SKILL.md index 4e18f16ac..023ebf3fc 100644 --- a/plugins/azure-skills/skills/azure-kubernetes/SKILL.md +++ b/plugins/azure-skills/skills/azure-kubernetes/SKILL.md @@ -4,7 +4,7 @@ license: MIT metadata: author: Microsoft version: "0.0.0-placeholder" -description: "Plan, create, and configure production-ready Azure Kubernetes Service (AKS) clusters. Covers Day-0 checklist, SKU selection (Automatic vs Standard), networking options (private API server, Azure CNI Overlay, egress configuration), security, and operations (autoscaling, upgrade strategy, cost analysis). WHEN: create AKS environment, provision AKS, enable AKS observability, design AKS networking, choose AKS SKU, secure AKS, optimize AKS, AKS spot nodes, AKS cluster-autoscaler, rightsize AKS pod, pod rightsizing, over-provisioned AKS pod, pod resource requests and limits, Vertical Pod Autoscaler, VPA recommendations." +description: "Plan, create, configure, and review production-ready Azure Kubernetes Service (AKS) clusters. Covers Day-0 checklist, SKU selection (Automatic vs Standard), networking options (private API server, Azure CNI Overlay, egress configuration), security, and operations (autoscaling, upgrade strategy, cost analysis) and Day-2 review and audit. WHEN: create AKS environment, provision AKS, enable AKS observability, design AKS networking, choose AKS SKU, secure AKS, optimize AKS, AKS spot nodes, AKS cluster-autoscaler, rightsize AKS pod, pod rightsizing, over-provisioned AKS pod, pod resource requests and limits, Vertical Pod Autoscaler, VPA recommendations, review AKS, audit AKS, AKS remediation." --- # Azure Kubernetes Service @@ -16,9 +16,9 @@ description: "Plan, create, and configure production-ready Azure Kubernetes Serv ## Quick Reference | Property | Value | |----------|-------| -| Best for | AKS cluster planning and Day-0 decisions | +| Best for | AKS cluster planning (Day-0) and posture review / compliance audit (Day-2) | | MCP Tools | `mcp_azure_mcp_aks` | -| CLI | `az aks create`, `az aks show`, `kubectl get`, `kubectl describe` | +| CLI | `az aks create`, `az aks show`, `az aks nodepool list`, `kubectl get`, `kubectl describe` | | Related skills | azure-kubernetes-app-deploy (deploy an app to an existing cluster), azure-diagnostics (troubleshooting AKS), azure-validate (readiness checks), azure-kubernetes-automatic-readiness (migrate existing cluster to AKS Automatic) | ## When to Use This Skill @@ -32,6 +32,7 @@ Activate this skill when user wants to: - Define AKS upgrade and patching strategy - Understand AKS Automatic vs Standard SKU differences - Get a Day-0 checklist for AKS cluster setup and configuration +- Review or audit an existing AKS cluster against Microsoft Learn practices, Well-Architected pillars, the AKS Checklist, live diagnostics detectors, and container best practices (Day-2). See [azure-aks-review.md](./references/azure-aks-review.md). > **Deploying an application to an existing cluster?** This skill provisions and > configures the *cluster*. To containerize an app and deploy it to a cluster @@ -135,9 +136,14 @@ If the user is unsure, use safe defaults. | VPA Setup | vertical pod autoscaler, VPA recommendations, VPA enable | [azure-aks-vpa.md](./references/azure-aks-vpa.md) | | Cluster Autoscaler | idle nodes, CAS off, enable autoscaler, scale-down profile, node utilization | [azure-aks-autoscaler.md](./references/azure-aks-autoscaler.md) | | Spot Node Pools | Spot VMs, Spot nodes, batch workloads, cheaper nodes | [azure-aks-spot.md](./references/azure-aks-spot.md) | +| Cluster Review / Audit | review AKS, audit AKS cluster, AKS practices review, validate AKS posture, AKS compliance checklist, AKS remediation report | [azure-aks-review.md](./references/azure-aks-review.md) | > **Disambiguation:** If a prompt matches multiple rows (e.g., "cheaper nodes" could suggest both Spot and autoscaler), prefer the most specific match. If ambiguous, ask the user to clarify their intent before loading a reference file. +## Cluster Review / Audit (Day-2) + +When the user asks to **review**, **audit**, or **assess the compliance posture** of an existing AKS cluster, load [azure-aks-review.md](./references/azure-aks-review.md) for the full workflow: required inputs, 9-step review workflow, decision logic, output format, guardrails, and links to the checklist matrix, diagnostics detectors, container best practices, and audit report template. + ## Guardrails / Safety - Do not request or output secrets (tokens, keys). - Do not ask the user to paste subscription IDs. Discover subscription and resource scope via MCP tools (e.g., list subscriptions, list resource groups) or `az account show` / `az account list` so the agent can resolve context without exposing identifiers. diff --git a/plugins/azure-skills/skills/azure-kubernetes/assets/aks-audit-report-template.md b/plugins/azure-skills/skills/azure-kubernetes/assets/aks-audit-report-template.md new file mode 100644 index 000000000..066f010bf --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/assets/aks-audit-report-template.md @@ -0,0 +1,256 @@ +# Azure Kubernetes Service Review - + +## Table of Contents + +1. [Scope](#scope) +2. [Overview of AKS Practices](#overview-of-aks-practices) +3. [Subpage Compliance Rollup](#subpage-compliance-rollup) +4. [Detailed Analysis](#detailed-analysis) +5. [AKS Diagnostics Findings](#aks-diagnostics-findings) +6. [Container Practices Assessment](#container-practices-assessment) +7. [Kubernetes Warning Events by Namespace](#kubernetes-warning-events-by-namespace) +8. [Summary](#summary) +9. [Next Steps](#next-steps) +10. [Appendices](#appendices) + +## Scope + +- Cluster under review: `//` +- Kubernetes context: `` +- Review date: `` +- Extraction timestamps: `` / `` +- Kubernetes version: `` +- Region: `` +- Node count: `` (``) +- Review scope: + - AKS cluster configuration and Kubernetes in-cluster controls only. +- Out of scope: + - Other Azure resources that are not directly required to validate AKS controls. +- Limitations and constraints: + - `` + +## Overview of AKS Practices + +This assessment evaluates the AKS cluster against the following sources of practices: + +1. AKS practices (operator and developer categories) on Microsoft Learn +2. Azure Well-Architected AKS service guide (Reliability, Security, Cost Optimization, Operational Excellence, Performance Efficiency) on Microsoft Learn +3. The AKS Checklist at +4. AKS Diagnostics Detectors ("Diagnose and solve problems") — runtime risk alerts and health assessments from the Azure platform +5. Container Practices - Per-container assessment of image hygiene, security context, resource management, health probes, pod-level security, configuration hygiene, and network controls. + +The report combines Azure platform evidence (`az`), Kubernetes evidence (`kubectl`), and AKS diagnostics detector outputs to determine adherence status per checklist item. + +## Executive Summary + +Provide an Executive Summary consisting of two paragraphs; the first paragraph will contain the key areas where the AKS cluster shows strength and best practices. The second paragraph will itemise the top 5 area requiring attention. + +| Metric | Value | +|---|---| +| **Overall Compliance Score** | | +| Total Checks Assessed | | +| Meets | | +| Partially Meets | | +| Does Not Meet | | +| Not Applicable | | +| Not Assessed | | + +## Critical Findings Summary + +| # | Finding | Severity | Impact | Checklist Reference | +|---|---------|----------|--------|-------------------| + +--- + +## Subpage Compliance Rollup + +| Parent Page | Child Page | Total Checks | Scored Checks | Meets | Partially meets | Does not meet | Not applicable | Not assessed | Compliance % | +|---|---|---|---|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | `` | `` | `` | `<0-100 or N/A>` | + +Scoring formula: +- `Compliance % = ((Meets + (0.5 * Partially meets)) / Scored Checks) * 100` +- If `Scored Checks = 0`, set `Compliance %` to `N/A`. + +## Detailed Analysis + +| Check ID | Parent Page | Child Page | Pillar/Category | Checklist Item | Summary | Microsoft Learn Reference | Commands Executed | Evidence Summary | Comments | Status | Remediation Options (up to 5) | +|---|---|---|---|---|---|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | `` | `
` | `` | `` | `` | `1) ` `
` `2) ` `
` `3) ` | + +## AKS Diagnostics Findings + +Findings from the AKS cluster's built-in diagnostics detectors (Azure "Diagnose and solve problems"), extracted at ``. + +Extraction method used: `` + +### Risk Alerts + +Findings with Critical or Warning severity across all detector categories. + +| Detector | Category | Severity | Status | Description | Affected Resources | Recommendations | +|---|---|---|---|---|---|---| +| `` | `` | `` | `` | `` | `` | `` | + +If no Critical or Warning findings exist, state: _No risk alerts detected._ + +### Detector Results by Category + +Full results from all eight detector categories. + +| Category | Detector | Status | Key Findings | Cross-Referenced Check IDs | +|---|---|---|---|---| +| `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | +| `` | `` | `` | `` | `` | + +Categories queried: Best Practices, Cluster and Control Plane Availability and Performance, Connectivity Issues, Create, Upgrade, Delete and Scale, Deprecations, Identity and Security, Node Health, Storage. + +Supplemental live categories returned by the detector inventory, if any, should be listed separately with their detector type and handling notes. + +### Diagnostics Impact on Checklist Assessment + +List any checklist items whose status was adjusted based on diagnostics findings. + +| Check ID | Original Status | Adjusted Status | Diagnostics Justification | +|---|---|---|---| +| `` | `` | `` | `` | + +If no adjustments were made, state: _No checklist items required status adjustment based on diagnostics findings._ + +## Container Practices Assessment + +Container-level best-practice findings assessed at ``. Checks are sourced from the [AKS Container Best Practices Reference](../references/aks-container-best-practices.md). + +### Container Summary Statistics + +Container counting methodology: All running container instances are counted (including scaled replicas and injected sidecar/proxy containers). Sidecar containers are reported separately where they materially affect compliance metrics. + +| Metric | Cluster-Wide Count | Percentage | Notes | +|---|---|---|---| +| Total containers assessed | `` | — | Across all namespaces in scope | +|  of which sidecar/proxy containers | `` | `%` | Linkerd, Istio, Envoy, etc. | +| Containers without CPU/memory requests | `` | `%` | Excludes system namespaces | +| Containers without CPU/memory limits | `` | `%` | Excludes system namespaces | +| Containers running as root (or runAsNonRoot not set) | `` | `%` (app-only: `%`) | Excludes system namespaces | +| Containers without readiness probes | `` | `%` | Excludes init containers and system namespaces | +| Containers using `:latest` tag or no tag | `` | `%` | All namespaces | +| BestEffort QoS pods | `` | `%` | All namespaces | +| Privileged containers | `` | `%` | Excludes system namespaces | +| Containers with sensitive inline env vars | `` | `%` | Excludes system namespaces | +| Pods using `default` service account | `` | `%` | Excludes system namespaces | + +### Container Checks by Category Rollup + +| Category | Total Checks | Scored Checks | Meets | Partially meets | Does not meet | Not applicable | Not assessed | Compliance % | +|---|---|---|---|---|---|---|---|---| +| Image Hygiene | `` | `` | `` | `` | `` | `` | `` | `<0-100 or N/A>` | +| Container Security Context | `` | `` | `` | `` | `` | `` | `` | `<0-100 or N/A>` | +| Resource Management | `` | `` | `` | `` | `` | `` | `` | `<0-100 or N/A>` | +| Health Probes and Lifecycle | `` | `` | `` | `` | `` | `` | `` | `<0-100 or N/A>` | +| Pod-Level Security Controls | `` | `` | `` | `` | `` | `` | `` | `<0-100 or N/A>` | +| Container Configuration Hygiene | `` | `` | `` | `` | `` | `` | `` | `<0-100 or N/A>` | +| Network and Service Mesh Controls | `` | `` | `` | `` | `` | `` | `` | `<0-100 or N/A>` | +| ACR and Image Supply Chain | `` | `` | `` | `` | `` | `` | `` | `<0-100 or N/A>` | + +Scoring thresholds: ≥80% container compliance = Meets, 50-79% = Partially meets, <50% = Does not meet. + +### Container Checks Detail + +| Check ID | Category | Checklist Item | Commands Executed | Cluster-Wide Compliance % | Failing Namespaces | Status | Remediation Options (up to 5) | +|---|---|---|---|---|---|---|---| +| `` | `` | `` | `
` | `<0-100>%` | `` | `` | `1) ` `
` `2) ` | + +### System Namespace Findings + +Findings for system namespaces (`kube-system`, `kube-node-lease`, `kube-public`, `gatekeeper-system`) are reported separately. Violations in system namespaces may be expected for infrastructure components (CNI plugins, CSI drivers, monitoring agents). + +| Namespace | Check ID | Finding Summary | Justification | +|---|---|---|---| +| `` | `` | `<2 containers run in privileged mode>` | `` | + +If no system namespace findings require attention, state: _All system namespace container configurations are consistent with expected infrastructure requirements._ + +If `kubectl` access is unavailable, state: _Container best practices assessment could not be performed — kubectl access unavailable. Platform-level checks (CTR-IMG-06, CTR-ACR-*) assessed via az CLI where applicable._ + +## Kubernetes Warning Events by Namespace + +Warning-type Kubernetes events collected across all namespaces at ``. Event correlation window: `<30 days via Azure Monitor Logs | etcd-only: to >`. Each event is correlated to the owning top-level workload (Deployment, StatefulSet, DaemonSet, Job, or CronJob). + +| Namespace | Event Reason | Message Summary | Occurrences | First Seen | Last Seen | Source Component | Affected Workload(s) | Remediation Steps | +|---|---|---|---|---|---|---|---|---| +| `` | `` | `` | `<12>` | `` | `` | `` | `` | `1) ` `
` `2) ` `
` `3) ` | + +**Column rules:** +- **Namespace**: The Kubernetes namespace where the event was emitted. +- **Event Reason**: The Kubernetes event `.reason` field (e.g., `BackOff`, `FailedScheduling`, `Unhealthy`, `FailedMount`, `FailedCreate`, `OOMKilling`). +- **Message Summary**: A concise version of the event `.message`. Truncate to 200 characters if needed; include the essential detail (image name, resource name, error text). +- **Occurrences**: The event `.count` (number of times observed). +- **First Seen** / **Last Seen**: From `.firstTimestamp` and `.lastTimestamp`. +- **Source Component**: From `.source.component` (e.g., `kubelet`, `default-scheduler`, `kube-controller-manager`). +- **Affected Workload(s)**: The resolved top-level owning workload in `/` format (e.g., `Deployment/my-app`, `StatefulSet/redis`). If a single event reason affects multiple workloads in the same namespace, list all separated by `
`. **Never show only the Pod name** — always resolve through the owner chain. +- **Remediation Steps**: A numbered list of up to 5 implementation-ready actions specific to the event reason and message. Include exact commands, configuration changes, or resource adjustments. + +If no Warning events exist across any namespace, state: _No Kubernetes Warning events detected._ + +If `kubectl` access is unavailable, state: _Warning events could not be collected — kubectl access unavailable._ + +### Deep-Dive on Probe Failures + +_Include this sub-section when any workload has recurring `Unhealthy` events (≥10 occurrences or ≥7 day span). Produce one sub-section per affected namespace. Omit entirely if no workloads meet the threshold._ + +| Workload | Live Evidence | Interpretation | Recommended Change | +| --- | --- | --- | --- | +| `` | `` | `` | `` | + +## Summary + + + +## Next Steps — Prioritized Remediation Roadmap + +### Immediate (0–2 weeks) + +1. `` +2. `` + +### Short-term (2–6 weeks) + +3. `` +4. `` + +### Medium-term (1–3 months) + +5. `` +6. `` + +### Long-term (3–6 months) + +7. `` +8. `` + +## Appendices + +### Commands Reference + +| Purpose | Command | +|---|---| +| Cluster config | `az aks show -g -n -o json` | +| Node pools | `az aks nodepool list -g --cluster-name -o table` | +| Node image versions | `az aks nodepool list -g --cluster-name --query "[].{name:name,nodeImageVersion:nodeImageVersion}" -o table` | +| Namespaces | `kubectl get namespaces --context ` | +| Network policies | `kubectl get networkpolicy -A --context ` | +| PSS labels | `kubectl get ns -L pod-security.kubernetes.io/enforce --context ` | +| HPAs | `kubectl get hpa -A --context ` | +| KEDA ScaledObjects | `kubectl get scaledobject -A --context ` | +| SecretProviderClasses | `kubectl get secretproviderclass -A --context ` | +| All pods (JSON) | `kubectl get pods -A -o json --context ` | +| Warning events | `kubectl get events -A --field-selector type=Warning -o json --context ` | +| Diagnostic settings | `az monitor diagnostic-settings list --resource -o json` | +| Diagnostics detectors | `az rest --method get --url "/detectors/?api-version=2024-09-01"` | + diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-checklist-matrix.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-checklist-matrix.md new file mode 100644 index 000000000..590928617 --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-checklist-matrix.md @@ -0,0 +1,204 @@ +# AKS Checklist Matrix + +Use this checklist matrix as the canonical source for the audit. Execute at least one validation command per item when technically possible. Output all columns of every table. + +Legend: + +- `az`: Azure platform command +- `kubectl`: Kubernetes API command + +## Well-Architected Pillars (AKS Service Guide) + +| Check ID | Pillar | Checklist Item | Validation Commands | Learn Reference | +| --- | --- | --- | --- | --- | +| REL-01 | Reliability | Cluster uses availability zones where region supports zones | `az aks show -g -n --query "agentPoolProfiles[].availabilityZones" -o tsv` | | +| REL-02 | Reliability | Multi-region strategy for critical workloads | `az resource list -t Microsoft.ContainerService/managedClusters --query "[].{name:name,region:location}" -o table` | | +| REL-03 | Reliability | Internet-facing failover routing design (Front Door or Traffic Manager) | `az network front-door list -o table`; `az network traffic-manager profile list -o table` | | +| REL-04 | Reliability | System and user node pools are isolated | `az aks nodepool list -g --cluster-name -o table`; `kubectl get nodes -L agentpool` | | +| REL-05 | Reliability | System node pool sizing and minimum count meet guidance | `az aks nodepool list -g --cluster-name --query "[?mode=='System'].{name:name,count:count,vmSize:vmSize}" -o table` | | +| REL-06 | Reliability | Workloads define resource requests/limits | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{' req='}{.resources.requests}{' lim='}{.resources.limits}{'; '}{end}{'\n'}{end}"` | | +| REL-07 | Reliability | NAT Gateway used when high outbound concurrency is required | `az aks show -g -n --query "networkProfile.natGatewayProfile" -o json` | | +| REL-08 | Reliability | Backup and restore posture exists for cluster/app data | `az backup vault list -o table`; `az backup item list --vault-name -g -o table` | | +| SEC-01 | Security | Microsoft Entra ID integration is enabled | `az aks show -g -n --query "aadProfile" -o json` | | +| SEC-02 | Security | Local accounts disabled and Entra RBAC enforced where required | `az aks show -g -n --query "disableLocalAccounts" -o tsv`; `az aks show -g -n --query "azurePortalFqdn" -o tsv` | | +| SEC-03 | Security | API server access is private or IP-restricted | `az aks show -g -n --query "apiServerAccessProfile" -o json` | | +| SEC-04 | Security | Network policies are enabled (Azure/Calico/Cilium) | `az aks show -g -n --query "networkProfile.networkPolicy" -o tsv`; `kubectl get networkpolicy -A` | | +| SEC-05 | Security | Azure Policy add-on enabled for cluster governance | `az aks show -g -n --query "addonProfiles.azurepolicy.enabled" -o tsv` | | +| SEC-06 | Security | Workload identity enabled for pod-to-Azure auth | `az aks show -g -n --query "securityProfile.workloadIdentity" -o json`; `kubectl get sa -A -o yaml | grep -i azure.workload.identity` (Windows: use `kubectl get sa -A -o yaml | findstr /i azure.workload.identity`) | | +| SEC-07 | Security | Secrets externalized with Key Vault CSI where applicable | `az aks show -g -n --query "addonProfiles.azureKeyvaultSecretsProvider.enabled" -o tsv`; `kubectl get secretproviderclass -A` | | +| SEC-08 | Security | Defender for Containers enabled | `az security pricing show -n KubernetesService -o json` | | +| SEC-09 | Security | Egress control enforced through firewall/proxy pattern | `az aks show -g -n --query "networkProfile.outboundType" -o tsv`; `az network firewall list -o table` | | +| SEC-10 | Security | Pod security baseline/restricted controls enforced | `kubectl get ns -L pod-security.kubernetes.io/enforce,pod-security.kubernetes.io/audit,pod-security.kubernetes.io/warn` | | +| COST-01 | Cost Optimization | Cluster SKU/pricing tier aligns with environment intent | `az aks show -g -n --query "sku" -o json` | | +| COST-02 | Cost Optimization | Node pool VM sizing aligns with workload profile | `az aks nodepool list -g --cluster-name --query "[].{name:name,mode:mode,vmSize:vmSize,count:count}" -o table` | | +| COST-03 | Cost Optimization | Cluster autoscaler enabled for applicable pools | `az aks nodepool list -g --cluster-name --query "[].{name:name,auto:autoScalerEnabled,min:minCount,max:maxCount}" -o table` | | +| COST-04 | Cost Optimization | HPA/VPA used for rightsizing workloads | `kubectl get hpa -A`; `kubectl get vpa -A` | | +| COST-05 | Cost Optimization | Cost analysis add-on enabled and used | `az aks show -g -n --query "addonProfiles.costAnalysis.enabled" -o tsv` | | +| COST-06 | Cost Optimization | Spot node pools used intentionally where disruption-tolerant | `az aks nodepool list -g --cluster-name --query "[?scaleSetPriority=='Spot'].{name:name,spotMaxPrice:spotMaxPrice}" -o table` | | +| OPS-01 | Operational Excellence | IaC-driven cluster lifecycle (Bicep/Terraform) | `az deployment sub list -o table`; `az deployment group list -g -o table` | | +| OPS-02 | Operational Excellence | GitOps/automated delivery used for cluster and workloads | `az k8s-configuration flux list -g -c -t managedClusters -o table` | | +| OPS-03 | Operational Excellence | Monitoring strategy includes logs, metrics, and diagnostics | `az aks show -g -n --query "addonProfiles.omsagent.enabled" -o tsv`; `az monitor diagnostic-settings list --resource ` | | +| OPS-04 | Operational Excellence | Testing in production/chaos strategy exists for critical services | `az chaos target list -g --location -o table` | | +| OPS-05 | Operational Excellence | Policy-based governance enforced for cluster/workload standards | `az policy assignment list --scope -o table` | | +| PERF-01 | Performance Efficiency | Capacity planning covers SKU, scaling, IP, failover | `az aks show -g -n --query "{kubernetesVersion:kubernetesVersion,networkProfile:networkProfile}" -o json`; `kubectl top nodes` | | +| PERF-02 | Performance Efficiency | Cluster autoscaler + HPA/KEDA strategy implemented | `az aks nodepool list -g --cluster-name -o table`; `kubectl get hpa -A`; `kubectl get scaledobject -A` | | +| PERF-03 | Performance Efficiency | Workloads segmented across node pools for independent scaling | `kubectl get pods -A -o wide`; `az aks nodepool list -g --cluster-name -o table` | | +| PERF-04 | Performance Efficiency | High scale Container Insights mode used for large clusters | `az aks show -g -n --query "addonProfiles.omsagent.config" -o json` | | +| PERF-05 | Performance Efficiency | LocalDNS considered/enabled for large cluster DNS performance | `kubectl -n kube-system get ds | grep -i dns` (Windows: use `kubectl -n kube-system get ds | findstr /i dns`); `kubectl -n kube-system get cm coredns -o yaml` | | + +## AKS Best-Practices Categories (Operator and Developer) + +| Check ID | Category | Checklist Item | Validation Commands | Learn Reference | +| --- | --- | --- | --- | --- | +| AKS-OP-01 | Multi-tenancy | Namespaces and quotas isolate tenants/workloads | `kubectl get ns`; `kubectl get resourcequota -A`; `kubectl get limitrange -A` | | +| AKS-OP-02 | Multi-tenancy | Scheduler controls: taints/tolerations, affinity, anti-affinity | `kubectl get nodes -o json | jq '.items[].spec.taints'`; `kubectl get deploy -A -o yaml | grep -i -E "affinity|tolerations|nodeSelector"` (Windows: use `kubectl get deploy -A -o yaml | findstr /i "affinity tolerations nodeSelector"`) | | +| AKS-OP-03 | Multi-tenancy | Kubernetes RBAC and Azure RBAC least privilege applied | `kubectl get clusterrolebinding`; `az role assignment list --scope -o table` | | +| AKS-OP-04 | Security | API server hardening and controlled admin paths | `az aks show -g -n --query "apiServerAccessProfile" -o json`; `kubectl get clusterrolebinding -o wide` | | +| AKS-OP-05 | Security | Container image scanning and trusted registries used | `az acr task list-runs -r -o table`; `kubectl get pods -A -o jsonpath="{.items[*].spec.containers[*].image}"` | | +| AKS-OP-06 | Security | Pod security controls reduce privilege escalation risk | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.securityContext}{' '}{end}{'\n'}{end}"` | | +| AKS-OP-07 | Network and storage | Network model and ingress/WAF posture align to risk profile | `az aks show -g -n --query "networkProfile" -o json`; `kubectl get ingress -A` | | +| AKS-OP-08 | Network and storage | Storage classes and backup strategy match workload RPO/RTO | `kubectl get sc`; `kubectl get pvc -A`; `az backup vault list -o table` | | +| AKS-OP-09 | Enterprise-ready workloads | Multi-region BCDR pattern in place for mission-critical workloads | `az resource list -t Microsoft.ContainerService/managedClusters -o table`; `az acr replication list -r -o table` | | +| AKS-DEV-01 | Developer best practices | Requests/limits and probes are consistently defined | `kubectl get deploy -A -o yaml | grep -i -E "requests|limits|livenessProbe|readinessProbe"` (Windows: use `kubectl get deploy -A -o yaml | findstr /i "requests limits livenessProbe readinessProbe"`) | | +| AKS-DEV-02 | Developer best practices | Deployment reliability practices used (PDB, rolling updates, health checks) | `kubectl get pdb -A`; `kubectl get deploy -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{.spec.strategy.type}{'\n'}{end}"` | | +| AKS-DEV-03 | Developer best practices | Secret handling avoids inline sensitive material in manifests | `kubectl get secret -A`; `kubectl get deploy -A -o yaml | grep -i -E "envFrom|secretKeyRef"` (Windows: use `kubectl get deploy -A -o yaml | findstr /i "envFrom secretKeyRef"`) | | + +## Deeper Child-Page Checks (AKS Best-Practices Subpages) + +### Hierarchy Mapping (Parent Page -> Child Page -> Check IDs) + +| Parent Page | Child Page | Check IDs | +| --- | --- | --- | +| Cluster operator best practices | Cluster isolation | `AKS-ISO-01`, `AKS-ISO-02` | +| Cluster operator best practices | Basic scheduler | `AKS-SCHB-01`, `AKS-SCHB-02` | +| Cluster operator best practices | Advanced scheduler | `AKS-SCHA-01`, `AKS-SCHA-02` | +| Cluster operator best practices | Authentication and authorization | `AKS-ID-01`, `AKS-ID-02`, `AKS-ID-03` | +| Cluster operator best practices | Cluster security and upgrades | `AKS-CSEC-01`, `AKS-CSEC-02` | +| Cluster operator best practices | Container image management and security | `AKS-CIMG-01`, `AKS-CIMG-02` | +| Cluster operator best practices | Network connectivity | `AKS-NET-01`, `AKS-NET-02`, `AKS-NET-03` | +| Cluster operator best practices | Storage and backups | `AKS-STO-01`, `AKS-STO-02` | +| Cluster operator best practices | Business continuity and disaster recovery | `AKS-MR-01`, `AKS-MR-02`, `AKS-MR-03` | +| Developer best practices | Application developers to manage resources | `AKS-DEVRM-01` | +| Developer best practices | Pod security | `AKS-DEVPS-01`, `AKS-DEVPS-02` | +| Developer best practices | Deployment and cluster reliability | `AKS-RELY-01`, `AKS-RELY-02` | + +### Normalized Child-Page Checks + +| Parent Page | Child Page | Check ID | Checklist Item | Validation Commands | Learn Reference | +| --- | --- | --- | --- | --- | --- | +| Cluster operator best practices | Cluster isolation | AKS-ISO-01 | Tenant workloads are isolated by namespace with namespace-level RBAC boundaries | `kubectl get ns`; `kubectl get rolebinding -A`; `kubectl get clusterrolebinding` | | +| Cluster operator best practices | Cluster isolation | AKS-ISO-02 | ResourceQuota and LimitRange controls are present for tenant namespaces | `kubectl get resourcequota -A`; `kubectl get limitrange -A` | | +| Cluster operator best practices | Basic scheduler | AKS-SCHB-01 | Pod disruption budgets are configured for critical workloads | `kubectl get pdb -A`; `kubectl get deploy -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': replicas='}{.spec.replicas}{'\n'}{end}"` | | +| Cluster operator best practices | Basic scheduler | AKS-SCHB-02 | Node pressure is controlled with requests/limits and quota policy | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{' req='}{.resources.requests}{' lim='}{.resources.limits}{'; '}{end}{'\n'}{end}"`; `kubectl get resourcequota -A` | | +| Cluster operator best practices | Advanced scheduler | AKS-SCHA-01 | Taints and tolerations are intentionally used for workload segregation | `kubectl get nodes -o jsonpath="{range .items[*]}{.metadata.name}{': '}{.spec.taints}{'\n'}{end}"`; `kubectl get deploy -A -o yaml | grep -i tolerations` (Windows: use `kubectl get deploy -A -o yaml | findstr /i tolerations`) | | +| Cluster operator best practices | Advanced scheduler | AKS-SCHA-02 | Node selectors, affinity, and anti-affinity are used for placement policy | `kubectl get deploy -A -o yaml | grep -i -E "nodeSelector|affinity|antiAffinity"` (Windows: use `kubectl get deploy -A -o yaml | findstr /i "nodeSelector affinity antiAffinity"`) | | +| Cluster operator best practices | Authentication and authorization | AKS-ID-01 | Microsoft Entra integration is enabled for cluster authn/authz | `az aks show -g -n --query "aadProfile" -o json` | | +| Cluster operator best practices | Authentication and authorization | AKS-ID-02 | Least-privilege RBAC model avoids broad cluster-admin grants | `kubectl get clusterrolebinding -o wide`; `kubectl get rolebinding -A` | | +| Cluster operator best practices | Authentication and authorization | AKS-ID-03 | Workload identity is used instead of pod-managed secrets where possible | `az aks show -g -n --query "securityProfile.workloadIdentity" -o json`; `kubectl get serviceaccount -A -o yaml | grep -i azure.workload.identity` (Windows: use `kubectl get serviceaccount -A -o yaml | findstr /i azure.workload.identity`) | | +| Cluster operator best practices | Cluster security and upgrades | AKS-CSEC-01 | AKS version and node image patch posture are current and supported | `az aks show -g -n --query "{kubernetesVersion:kubernetesVersion,nodeResourceGroup:nodeResourceGroup}" -o json`; `az aks nodepool list -g --cluster-name -o table` | | +| Cluster operator best practices | Cluster security and upgrades | AKS-CSEC-02 | API server access is restricted via private endpoint or authorized ranges | `az aks show -g -n --query "apiServerAccessProfile" -o json` | | +| Cluster operator best practices | Container image management and security | AKS-CIMG-01 | Images are pulled from approved registries only | `kubectl get pods -A -o jsonpath="{.items[*].spec.containers[*].image}"`; `az aks check-acr -g -n --acr ` | | +| Cluster operator best practices | Container image management and security | AKS-CIMG-02 | Image scanning and update automation are in place | `az acr task list-runs -r -o table`; `az security assessment list --query "[?contains(displayName,'Container')].{name:displayName,status:status.code}" -o table` | | +| Cluster operator best practices | Network connectivity | AKS-NET-01 | Network plugin choice aligns with requirements and scale design | `az aks show -g -n --query "networkProfile" -o json` | | +| Cluster operator best practices | Network connectivity | AKS-NET-02 | Ingress endpoints are protected with WAF where internet-facing | `kubectl get ingress -A`; `az network application-gateway list -o table`; `az network front-door waf-policy list -o table` | | +| Cluster operator best practices | Network connectivity | AKS-NET-03 | Node/admin access paths are restricted and audited | `az aks show -g -n --query "linuxProfile" -o json`; `az network nsg list -o table` | | +| Cluster operator best practices | Storage and backups | AKS-STO-01 | StorageClass defaults and CSI usage align with workload needs | `kubectl get storageclass`; `kubectl get pvc -A -o wide`; `kubectl get csidriver` | | +| Cluster operator best practices | Storage and backups | AKS-STO-02 | Backup approach covers PV data and cluster state where required | `az backup vault list -o table`; `az backup item list --vault-name -g -o table` | | +| Cluster operator best practices | Business continuity and disaster recovery | AKS-MR-01 | Critical workload has multi-cluster deployment pattern across regions | `az resource list -t Microsoft.ContainerService/managedClusters --query "[].{name:name,location:location}" -o table` | | +| Cluster operator best practices | Business continuity and disaster recovery | AKS-MR-02 | Global traffic routing and failover are implemented and testable | `az network front-door list -o table`; `az network traffic-manager profile list -o table` | | +| Cluster operator best practices | Business continuity and disaster recovery | AKS-MR-03 | Container image geo-replication supports regional failover | `az acr replication list -r -o table` | | +| Developer best practices | Application developers to manage resources | AKS-DEVRM-01 | Requests, limits, probes, and rollout strategy are consistently defined | `kubectl get deploy -A -o yaml | grep -i -E "requests|limits|livenessProbe|readinessProbe|strategy"` (Windows: use `kubectl get deploy -A -o yaml | findstr /i "requests limits livenessProbe readinessProbe strategy"`) | | +| Developer best practices | Pod security | AKS-DEVPS-01 | Pods run with non-root, read-only root FS, and no privilege escalation where feasible | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{' sec='}{.securityContext}{'; '}{end}{'\n'}{end}"` | | +| Developer best practices | Pod security | AKS-DEVPS-02 | Secrets are externalized and not hardcoded in pod specs | `kubectl get deploy -A -o yaml | grep -i -E "secretKeyRef|envFrom"` (Windows: use `kubectl get deploy -A -o yaml | findstr /i "secretKeyRef envFrom"`); `kubectl get secretproviderclass -A` | | +| Developer best practices | Deployment and cluster reliability | AKS-RELY-01 | Deployment reliability controls exist (PDB, replicas, readiness, rolling updates) | `kubectl get pdb -A`; `kubectl get deploy -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': strategy='}{.spec.strategy.type}{', replicas='}{.spec.replicas}{'\n'}{end}"` | | +| Developer best practices | Deployment and cluster reliability | AKS-RELY-02 | Workload spread and disruption tolerance are configured | `kubectl get deploy -A -o yaml | grep -i -E "topologySpreadConstraints|maxUnavailable"` (Windows: use `kubectl get deploy -A -o yaml | findstr /i "topologySpreadConstraints maxUnavailable"`); `kubectl get poddisruptionbudget -A` | | + +### Subpage Rollup Scoring Rules + +- Group deeper checks by `Parent Page` and `Child Page`. +- For each group, compute: + - `Total Checks`: count of all checks in the group. + - `Scored Checks`: checks with status in `Meets`, `Partially meets`, `Does not meet`. + - `Meets`, `Partially meets`, `Does not meet`, `Not applicable`, `Not assessed` counts. + - `Compliance %`: `((Meets + (0.5 * Partially meets)) / Scored Checks) * 100`. +- If `Scored Checks` is `0`, set `Compliance %` to `N/A`. + +## External Checklist Incorporation (the-aks-checklist.com) + +Source: + +- + +### Conflict Resolution Policy + +- If an AKS Checklist item clashes with an existing Microsoft Learn-backed check, follow Microsoft Learn. +- In clashes, keep the existing check ID and Learn reference; add AKS Checklist reference as context only. +- Add only non-conflicting AKS Checklist items as new `AKSC-*` checks. + +### Reconciled Overlaps (Mapped to Existing Checks) + +| AKS Checklist Theme | Existing Check IDs (Microsoft Learn authoritative) | +| --- | --- | +| Managed Entra auth, disable local accounts, RBAC | `SEC-01`, `SEC-02`, `AKS-ID-01`, `AKS-ID-02` | +| Workload identity and ACR integration | `SEC-06`, `AKS-ID-03`, `AKS-CIMG-01` | +| API server private/authorized ranges | `SEC-03`, `AKS-CSEC-02` | +| Network policy and egress control | `SEC-04`, `SEC-09`, `AKS-NET-02`, `AKS-NET-03` | +| System/user nodepool isolation | `REL-04`, `REL-05` | +| Cluster autoscaler/HPA/KEDA | `COST-03`, `COST-04`, `PERF-02` | +| Storage class and backup posture | `AKS-STO-01`, `AKS-STO-02`, `REL-08` | +| Multi-region and traffic failover | `REL-02`, `REL-03`, `AKS-MR-01`, `AKS-MR-02`, `AKS-MR-03` | +| Defender and policy governance | `SEC-08`, `SEC-05`, `OPS-05` | + +### Supplemental AKS Checklist Checks (Non-Conflicting Additions) + +| Parent Page | Child Page | Check ID | Checklist Item | Validation Commands | Learn Reference | AKS Checklist Reference | +| --- | --- | --- | --- | --- | --- | --- | +| Cluster operator best practices | Networking | AKSC-NET-01 | Azure CNI subnet capacity and max pods per node are validated for IP exhaustion risk | `az aks show -g -n --query "{networkPlugin:networkProfile.networkPlugin,maxPods:agentPoolProfiles[].maxPods,podCidr:networkProfile.podCidr,serviceCidr:networkProfile.serviceCidr}" -o json`; `az network vnet subnet show --ids --query "{prefix:addressPrefix,ipConfigurations:ipConfigurations}" -o json` | | | +| Cluster operator best practices | Networking | AKSC-NET-02 | CIDR overlap checks are verified for pod/service/subnet/peered VNets | `az aks show -g -n --query "{podCidr:networkProfile.podCidr,serviceCidr:networkProfile.serviceCidr,dnsServiceIP:networkProfile.dnsServiceIP}" -o json`; `az network vnet list -o json` | | | +| Cluster operator best practices | Networking | AKSC-NET-03 | Private Link or service endpoints are used for AKS-to-PaaS connectivity where required | `az network private-endpoint list -o table`; `az network vnet subnet list --resource-group --vnet-name -o table` | | | +| Cluster operator best practices | Cluster security and upgrades | AKSC-SEC-01 | KMS-based etcd encryption with customer-managed keys is enabled where compliance requires CMK | `az aks show -g -n --query "securityProfile.azureKeyVaultKms" -o json` | | | +| Cluster operator best practices | Cluster security and upgrades | AKSC-SEC-02 | Image Cleaner is enabled to remove stale and vulnerable cached images on nodes | `az aks show -g -n --query "securityProfile.imageCleaner" -o json` | | | +| Cluster operator best practices | Cluster operations | AKSC-OPS-01 | AKS auto-certificate rotation is enabled | `az aks show -g -n --query "autoScalerProfile" -o json`; `az aks show -g -n --query "securityProfile" -o json` | | | +| Cluster operator best practices | Cluster operations | AKSC-OPS-02 | Control plane logs are enabled and routed to Azure Monitor diagnostics | `az monitor diagnostic-settings list --resource -o json`; `az aks show -g -n --query "addonProfiles.omsagent.enabled" -o tsv` | | | +| Cluster operator best practices | Cluster operations | AKSC-OPS-03 | ContainerLogV2 schema is enabled for container logging | `az monitor data-collection-rule list -o json`; `az monitor diagnostic-settings list --resource -o json` | | | +| Cluster operator best practices | Cluster operations | AKSC-OPS-04 | Alerts are configured for critical AKS and workload metrics | `az monitor metrics alert list -g -o table`; `az monitor scheduled-query list -g -o table` | | | +| Cluster operator best practices | Cluster operations | AKSC-OPS-05 | Event Grid integration for AKS events is configured for automation workflows | `az eventgrid system-topic list -o table`; `az eventgrid event-subscription list --source-resource-id -o table` | | | +| Cluster operator best practices | Business continuity and disaster recovery | AKSC-BCDR-01 | Availability zone support is enabled for control-plane resiliency strategy | `az aks show -g -n --query "agentPoolProfiles[].availabilityZones" -o tsv` | | | +| Cluster operator best practices | Business continuity and disaster recovery | AKSC-BCDR-02 | ACR zone redundancy and soft delete policy are enabled where required | `az acr show -n --query "zoneRedundancy" -o tsv`; `az acr config soft-delete show -r -o json` | | | +| Cluster operator best practices | Business continuity and disaster recovery | AKSC-BCDR-03 | AKS backups are scheduled with retention policy aligned to RPO/RTO | `az backup policy list --resource-group --vault-name -o table`; `az backup item list --resource-group --vault-name -o table` | | | +| Developer best practices | Application deployment | AKSC-APP-01 | Startup probes are configured for slow-boot workloads | `kubectl get deploy -A -o yaml | grep -i startupProbe` (Windows: use `kubectl get deploy -A -o yaml | findstr /i startupProbe`) | | | +| Developer best practices | Application deployment | AKSC-APP-02 | PreStop lifecycle hooks are configured for graceful shutdown | `kubectl get deploy -A -o yaml | grep -i preStop` (Windows: use `kubectl get deploy -A -o yaml | findstr /i preStop`) | | | +| Developer best practices | Application deployment | AKSC-APP-03 | Deployments avoid naked pods and define replica targets | `kubectl get pod -A --show-labels`; `kubectl get deploy -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': replicas='}{.spec.replicas}{'\n'}{end}"` | | | +| Developer best practices | Image management | AKSC-IMG-01 | Only approved registries are allowed via policy/admission controls | `az policy assignment list --scope -o table`; `kubectl get validatingwebhookconfiguration` | | | +| Developer best practices | Image management | AKSC-IMG-02 | Container image scans are enforced in CI/CD before deployment | `az security assessment list --query "[?contains(displayName,'container')].{name:displayName,status:status.code}" -o table`; `az acr task list-runs -r -o table` | | | +| Platform-specific | Windows | AKSC-WIN-01 | Windows node pools use Azure CNI and OS/image patch alignment is maintained | `az aks nodepool list -g --cluster-name --query "[?osType=='Windows'].{name:name,orchestratorVersion:orchestratorVersion,osSKU:osSKU}" -o table`; `az aks show -g -n --query "networkProfile.networkPlugin" -o tsv` | | | + +### External Hierarchy Mapping (Parent Page -> Child Page -> Check IDs) + +| Parent Page | Child Page | Check IDs | +| --- | --- | --- | +| Cluster operator best practices | Networking | `AKSC-NET-01`, `AKSC-NET-02`, `AKSC-NET-03` | +| Cluster operator best practices | Cluster security and upgrades | `AKSC-SEC-01`, `AKSC-SEC-02` | +| Cluster operator best practices | Cluster operations | `AKSC-OPS-01`, `AKSC-OPS-02`, `AKSC-OPS-03`, `AKSC-OPS-04`, `AKSC-OPS-05` | +| Cluster operator best practices | Business continuity and disaster recovery | `AKSC-BCDR-01`, `AKSC-BCDR-02`, `AKSC-BCDR-03` | +| Developer best practices | Application deployment | `AKSC-APP-01`, `AKSC-APP-02`, `AKSC-APP-03` | +| Developer best practices | Image management | `AKSC-IMG-01`, `AKSC-IMG-02` | +| Platform-specific | Windows | `AKSC-WIN-01` | + +## AKS Diagnostics Detectors + +The audit also incorporates live findings from the AKS "Diagnose and solve problems" detectors. These are runtime assessments executed by Azure against the cluster. + +- Extraction methods, detector categories, and interpretation rules are documented in [AKS Diagnostics Detectors Reference](./aks-diagnostics-detectors.md). +- Detector findings that align to existing checklist items must be cross-referenced in the Detailed Analysis table (update `Evidence Summary` and `Comments`). +- Detector findings that have no matching checklist item are reported in the dedicated **AKS Diagnostics Findings** section of the report. +- Critical or Warning findings may override a checklist item's status (see cross-reference rules in the detectors reference). + +## Assessment Notes + +- If a command requires unsupported tooling (example: `jq` on Windows), use an equivalent `az --query` or `kubectl -o jsonpath` expression. +- If command execution is blocked by RBAC, capture the exact error in evidence and mark `Not assessed`. +- Use `Not applicable` when a control legitimately does not apply to the workload architecture, and explain why in the report `Comments` column. +- For each failed or partial check, provide up to five remediation actions in priority order. +- For checks incorporated from the AKS Checklist, if guidance conflicts with existing checks, apply Microsoft Learn guidance as the final recommendation. +- AKS Diagnostics detector findings are point-in-time. Record the execution timestamp and note that results may change as cluster state evolves. diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-container-best-practices.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-container-best-practices.md new file mode 100644 index 000000000..8f6e47c68 --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-container-best-practices.md @@ -0,0 +1,224 @@ +# AKS Container Best Practices Reference + +This reference provides a comprehensive set of container-level best-practice checks that can be validated via `kubectl` and `az` CLI commands against an AKS cluster. These checks complement the cluster-level and platform-level controls in the [AKS Checklist Matrix](./aks-checklist-matrix.md) by diving deeper into container runtime configuration, image hygiene, security context, resource management, health probes, lifecycle hooks, and pod-level security controls. + +## Purpose + +The existing checklist matrix validates many controls at a high level (e.g., "workloads define resource requests/limits"). This reference provides granular, field-level validation commands that systematically audit every container across all namespaces. The goal is to surface containers that violate specific best-practice rules so the audit report can quantify non-compliance and provide targeted remediation. + +## Extraction Strategy + +### General Approach + +1. **Namespace enumeration**: Determine the set of namespaces in scope. By default, assess all non-system namespaces. System namespaces (`kube-system`, `kube-node-lease`, `kube-public`, `gatekeeper-system`) are reported separately since they contain platform-managed workloads with different security constraints. +2. **Workload enumeration**: For each namespace, enumerate Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs. Avoid assessing naked Pods directly (they are flagged separately as a best-practice violation). +3. **Container-level extraction**: For each workload, extract container specs (including init containers and ephemeral containers) and validate against the checks below. +4. **Aggregation**: Produce per-namespace and cluster-wide summaries showing the count and percentage of containers that pass or fail each check. + +### Command Compatibility + +- All commands use `kubectl -o jsonpath` or `kubectl -o json` for structured output. +- On Windows, use `findstr` instead of `grep` where text filtering is needed. +- Where `jq` is referenced as an ideal approach, a `kubectl -o jsonpath` alternative is always provided. +- Replace `` with the target cluster context in all commands. + +## Container Best-Practice Checks + +### Image Hygiene + +| Check ID | Category | Checklist Item | Validation Commands | Reference | +| --- | --- | --- | --- | --- | +| CTR-IMG-01 | Image Hygiene | Container images do not use the `:latest` tag or omit a tag entirely | `kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{.image}{'\n'}{end}{end}" --context \| grep -i ":latest"` (Windows: use ``kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{.image}{'\n'}{end}{end}" --context \| findstr /i ":latest"``) | | +| CTR-IMG-02 | Image Hygiene | Images use immutable tags or digests for production workloads | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.image}{', '}{end}{'\n'}{end}" --context ` — check for digest references (`@sha256:`) vs mutable tags | | +| CTR-IMG-03 | Image Hygiene | ImagePullPolicy is set explicitly and appropriately (Always for mutable tags, IfNotPresent for immutable/digest) | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'='}{.imagePullPolicy}{', '}{end}{'\n'}{end}" --context ` | | +| CTR-IMG-04 | Image Hygiene | Image pull secrets are configured for private registry access | `kubectl get serviceaccount -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': imagePullSecrets='}{.imagePullSecrets}{'\n'}{end}" --context `; `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': pullSecrets='}{.spec.imagePullSecrets}{'\n'}{end}" --context ` | | +| CTR-IMG-05 | Image Hygiene | All container images are sourced from approved/private registries (no public Docker Hub direct pulls) | `kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{.image}{'\n'}{end}{end}" --context ` — identify images without a registry prefix (implicit `docker.io`) or from public registries | | +| CTR-IMG-06 | Image Hygiene | ACR integration is configured for pull authentication (attach or workload identity) | `az aks show -g -n --query "identityProfile.kubeletidentity.resourceId" -o tsv`; `az role assignment list --scope --query "[?roleDefinitionName=='AcrPull']" -o table` | | + +### Container Security Context + +| Check ID | Category | Checklist Item | Validation Commands | Reference | +| --- | --- | --- | --- | --- | +| CTR-SEC-01 | Security Context | Containers set `runAsNonRoot: true` in securityContext | `kubectl get pods -A -o json --context ` — for each container, check `.spec.containers[*].securityContext.runAsNonRoot` and `.spec.securityContext.runAsNonRoot` (pod-level fallback); report containers where neither is `true` | | +| CTR-SEC-02 | Security Context | Containers set `readOnlyRootFilesystem: true` | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=roRootFS:'}{.securityContext.readOnlyRootFilesystem}{', '}{end}{'\n'}{end}" --context ` | | +| CTR-SEC-03 | Security Context | Containers set `allowPrivilegeEscalation: false` | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=allowPrivEsc:'}{.securityContext.allowPrivilegeEscalation}{', '}{end}{'\n'}{end}" --context ` | | +| CTR-SEC-04 | Security Context | No containers run in `privileged: true` mode | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=privileged:'}{.securityContext.privileged}{', '}{end}{'\n'}{end}" --context ` — flag any container where `privileged=true` | | +| CTR-SEC-05 | Security Context | Linux capabilities are dropped (`drop: ["ALL"]`) and only required capabilities are added | `kubectl get pods -A -o json --context ` — for each container, check `.spec.containers[*].securityContext.capabilities.drop` includes `ALL` and `.add` is either absent or minimal | | +| CTR-SEC-06 | Security Context | Seccomp profile is set (RuntimeDefault or Localhost) | `kubectl get pods -A -o json --context ` — check `.spec.securityContext.seccompProfile.type` and `.spec.containers[*].securityContext.seccompProfile.type`; report containers without a seccomp profile | | +| CTR-SEC-07 | Security Context | Containers specify explicit `runAsUser` and `runAsGroup` (non-zero) | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=runAsUser:'}{.securityContext.runAsUser}{',runAsGroup:'}{.securityContext.runAsGroup}{'; '}{end}{'\n'}{end}" --context ` | | + +### Resource Management + +| Check ID | Category | Checklist Item | Validation Commands | Reference | +| --- | --- | --- | --- | --- | +| CTR-RES-01 | Resource Management | All containers define CPU and memory requests | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=cpuReq:'}{.resources.requests.cpu}{',memReq:'}{.resources.requests.memory}{'; '}{end}{'\n'}{end}" --context ` — flag containers with empty request fields | | +| CTR-RES-02 | Resource Management | All containers define CPU and memory limits | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=cpuLim:'}{.resources.limits.cpu}{',memLim:'}{.resources.limits.memory}{'; '}{end}{'\n'}{end}" --context ` | | +| CTR-RES-03 | Resource Management | QoS class distribution is understood and appropriate (Guaranteed preferred for critical workloads) | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': qosClass='}{.status.qosClass}{'\n'}{end}" --context ` — tally Guaranteed, Burstable, and BestEffort counts | | +| CTR-RES-04 | Resource Management | Resource limits are not excessively higher than requests (limit-to-request ratio within reasonable bounds) | `kubectl get pods -A -o json --context ` — compare `.resources.requests.cpu` vs `.resources.limits.cpu` and `.resources.requests.memory` vs `.resources.limits.memory` for each container; flag ratios exceeding 5:1 for CPU or 3:1 for memory | | +| CTR-RES-05 | Resource Management | Ephemeral storage requests and limits are set for containers that write temporary data | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=ephReq:'}{.resources.requests.ephemeral-storage}{',ephLim:'}{.resources.limits.ephemeral-storage}{'; '}{end}{'\n'}{end}" --context ` | | +| CTR-RES-06 | Resource Management | Init containers define appropriate resource requests and limits | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.initContainers[*]}{.name}{'=cpuReq:'}{.resources.requests.cpu}{',memReq:'}{.resources.requests.memory}{',cpuLim:'}{.resources.limits.cpu}{',memLim:'}{.resources.limits.memory}{'; '}{end}{'\n'}{end}" --context ` | | + +### Health Probes and Lifecycle + +| Check ID | Category | Checklist Item | Validation Commands | Reference | +| --- | --- | --- | --- | --- | +| CTR-PROBE-01 | Health Probes | All long-running containers define a readiness probe | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=readiness:'}{.readinessProbe}{'; '}{end}{'\n'}{end}" --context ` — flag containers with no readinessProbe | | +| CTR-PROBE-02 | Health Probes | All long-running containers define a liveness probe | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=liveness:'}{.livenessProbe}{'; '}{end}{'\n'}{end}" --context ` | | +| CTR-PROBE-03 | Health Probes | Startup probes are configured for containers with long initialization times | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=startup:'}{.startupProbe}{'; '}{end}{'\n'}{end}" --context ` | | +| CTR-PROBE-04 | Health Probes | Probe configuration uses appropriate timing values (initialDelaySeconds, periodSeconds, failureThreshold) | `kubectl get deploy -A -o json --context ` — for each container, extract probe timing fields and identify probes with defaults only (`initialDelaySeconds: 0`, `periodSeconds: 10`, `failureThreshold: 3`) that may not suit the workload | | +| CTR-PROBE-05 | Health Probes | Liveness and readiness probes use different endpoints or checks (liveness probe should not test downstream dependencies) | `kubectl get deploy -A -o json --context ` — compare liveness and readiness probe paths/commands for each container; flag identical configurations | | +| CTR-LIFE-01 | Lifecycle | `preStop` lifecycle hooks are defined for graceful shutdown of containers receiving traffic | `kubectl get deploy -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.template.spec.containers[*]}{.name}{'=preStop:'}{.lifecycle.preStop}{'; '}{end}{'\n'}{end}" --context ` | | +| CTR-LIFE-02 | Lifecycle | `terminationGracePeriodSeconds` is set appropriately (not left at default 30s if workload needs longer shutdown) | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': terminationGrace='}{.spec.terminationGracePeriodSeconds}{'\n'}{end}" --context ` | | +| CTR-LIFE-03 | Lifecycle | Rolling update strategy is configured with appropriate `maxSurge` and `maxUnavailable` values | `kubectl get deploy -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': strategy='}{.spec.strategy.type}{',maxSurge='}{.spec.strategy.rollingUpdate.maxSurge}{',maxUnavailable='}{.spec.strategy.rollingUpdate.maxUnavailable}{'\n'}{end}" --context ` | | + +### Pod-Level Security Controls + +| Check ID | Category | Checklist Item | Validation Commands | Reference | +| --- | --- | --- | --- | --- | +| CTR-POD-01 | Pod Security | `automountServiceAccountToken: false` is set on pods and/or service accounts that do not need API access | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': autoMount='}{.spec.automountServiceAccountToken}{', sa='}{.spec.serviceAccountName}{'\n'}{end}" --context `; `kubectl get sa -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': autoMount='}{.automountServiceAccountToken}{'\n'}{end}" --context ` | | +| CTR-POD-02 | Pod Security | Pods do not use `hostNetwork: true` unless explicitly required | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': hostNetwork='}{.spec.hostNetwork}{'\n'}{end}" --context ` — flag non-system pods with `hostNetwork=true` | | +| CTR-POD-03 | Pod Security | Pods do not use `hostPID: true` or `hostIPC: true` | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': hostPID='}{.spec.hostPID}{', hostIPC='}{.spec.hostIPC}{'\n'}{end}" --context ` | | +| CTR-POD-04 | Pod Security | No `hostPath` volumes are mounted in production workloads | `kubectl get pods -A -o json --context ` — check `.spec.volumes[*].hostPath` for non-null entries; exclude system namespaces from violation count | | +| CTR-POD-05 | Pod Security | Pod Security Standards labels are applied at the namespace level (`pod-security.kubernetes.io/enforce`) | `kubectl get ns --show-labels --context ` — check for `pod-security.kubernetes.io/enforce=baseline` or `restricted` labels | | +| CTR-POD-06 | Pod Security | Default service accounts are not used for workloads (dedicated service accounts per workload) | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': sa='}{.spec.serviceAccountName}{'\n'}{end}" --context ` — flag pods using `default` service account | | + +### Container Configuration Hygiene + +| Check ID | Category | Checklist Item | Validation Commands | Reference | +| --- | --- | --- | --- | --- | +| CTR-CFG-01 | Configuration Hygiene | No sensitive data is passed via environment variables directly (prefer secrets or Key Vault CSI) | `kubectl get deploy -A -o json --context ` — inspect `.spec.template.spec.containers[*].env[*]` for suspicious keys (e.g., containing `PASSWORD`, `SECRET`, `KEY`, `TOKEN`, `CONNECTION_STRING`) that use `.value` instead of `.valueFrom` | | +| CTR-CFG-02 | Configuration Hygiene | ConfigMaps and Secrets are used instead of baked-in configuration | `kubectl get deploy -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': envFrom='}{.spec.template.spec.containers[*].envFrom}{'\n'}{end}" --context `; `kubectl get configmap -A --context ` — verify workloads reference ConfigMaps/Secrets | | +| CTR-CFG-03 | Configuration Hygiene | No naked Pods exist (all pods are managed by a controller: Deployment, StatefulSet, DaemonSet, Job) | `kubectl get pods -A -o json --context ` — flag pods where `.metadata.ownerReferences` is empty or absent | | +| CTR-CFG-04 | Configuration Hygiene | Container ports are explicitly declared in pod specs | `kubectl get pods -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.containers[*]}{.name}{'=ports:'}{.ports}{'; '}{end}{'\n'}{end}" --context ` — flag containers with no ports declared that serve network traffic | | +| CTR-CFG-05 | Configuration Hygiene | Containers do not run multiple processes per container (single-concern principle) | Manual review — examine `command` and `args` fields: `kubectl get deploy -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': '}{range .spec.template.spec.containers[*]}{.name}{'=cmd:'}{.command}{',args:'}{.args}{'; '}{end}{'\n'}{end}" --context ` | | + +### Network and Service Mesh Controls + +| Check ID | Category | Checklist Item | Validation Commands | Reference | +| --- | --- | --- | --- | --- | +| CTR-NET-01 | Network | Network policies exist for application namespaces to restrict pod-to-pod traffic | `kubectl get networkpolicy -A -o wide --context ` — flag non-system namespaces with no network policies | | +| CTR-NET-02 | Network | Default-deny ingress network policy is applied per application namespace | `kubectl get networkpolicy -A -o json --context ` — check for policies with `spec.podSelector: {}` and empty `ingress` array (deny-all) | | +| CTR-NET-03 | Network | Services use `ClusterIP` type by default (LoadBalancer and NodePort are justified and documented) | `kubectl get svc -A -o jsonpath="{range .items[*]}{.metadata.namespace}/{.metadata.name}{': type='}{.spec.type}{'\n'}{end}" --context ` — flag `LoadBalancer` or `NodePort` services in application namespaces | | + +### ACR and Image Supply Chain (Platform-Level) + +| Check ID | Category | Checklist Item | Validation Commands | Reference | +| --- | --- | --- | --- | --- | +| CTR-ACR-01 | Image Supply Chain | ACR vulnerability scanning (Defender for Containers) is enabled | `az security pricing show -n ContainerRegistry -o json`; `az acr show -n --query "id" -o tsv` | | +| CTR-ACR-02 | Image Supply Chain | ACR image quarantine or approval gates are in place | `az acr config content-trust show -r -o json`; `az policy assignment list --scope -o table` | | +| CTR-ACR-03 | Image Supply Chain | ACR retention policy is configured to limit stale image accumulation | `az acr config retention show -r -o json` | | +| CTR-ACR-04 | Image Supply Chain | ACR webhook or Event Grid notifications are configured for image push events | `az acr webhook list -r -o table` | | + +## Summary Statistics Extraction + +To produce the cluster-wide summary for the report, run these aggregation commands: + +### Total Container Count + +```bash +kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{'1\n'}{end}{end}" --context | wc -l +(Windows: use `kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{'1\n'}{end}{end}" --context | find /c "1"`) +``` + +### Containers Without Resource Requests + +```bash +kubectl get pods -A -o json --context +``` + +Parse JSON output: count containers where `.resources.requests.cpu` or `.resources.requests.memory` is null or empty. + +### Containers Running as Root + +```bash +kubectl get pods -A -o json --context +``` + +Parse JSON output: count containers where neither pod-level nor container-level `runAsNonRoot` is `true` and `runAsUser` is `0` or unset. + +### Containers Without Readiness Probes + +```bash +kubectl get pods -A -o json --context +``` + +Parse JSON output: count containers (excluding init containers) where `.readinessProbe` is null. + +### Latest Tag Usage + +```bash +kubectl get pods -A -o jsonpath="{range .items[*]}{range .spec.containers[*]}{.image}{'\n'}{end}{end}" --context +``` + +Parse output: count images ending in `:latest` or with no tag specified. + +### BestEffort QoS Pods + +```bash +kubectl get pods -A -o jsonpath="{range .items[*]}{.status.qosClass}{'\n'}{end}" --context +``` + +Count occurrences of `BestEffort`. + +## Mapping to Existing Checklist Items + +Several container best-practice checks overlap with or deepen existing items in the [AKS Checklist Matrix](./aks-checklist-matrix.md). When both a matrix check and a container check apply, the container check provides deeper per-container evidence while the matrix check provides the pass/fail status. + +| Container Check ID | Overlapping Matrix Check ID | Relationship | +| --- | --- | --- | +| CTR-IMG-05 | AKS-CIMG-01 | Deepens — provides per-container image source evidence | +| CTR-IMG-06 | AKS-CIMG-01 | Deepens — validates ACR pull authentication at platform level | +| CTR-SEC-01, CTR-SEC-02, CTR-SEC-03, CTR-SEC-04 | AKS-DEVPS-01, SEC-10 | Deepens — per-field validation vs aggregate security context check | +| CTR-RES-01, CTR-RES-02 | REL-06, AKS-DEV-01, AKS-DEVRM-01 | Deepens — per-container resource field validation | +| CTR-PROBE-01, CTR-PROBE-02 | AKS-DEV-01, AKS-DEVRM-01 | Deepens — per-container probe presence validation | +| CTR-PROBE-03 | AKSC-APP-01 | Equivalent — startup probe validation | +| CTR-LIFE-01 | AKSC-APP-02 | Equivalent — preStop hook validation | +| CTR-CFG-03 | AKSC-APP-03 | Equivalent — naked pod detection | +| CTR-NET-01 | SEC-04 | Deepens — per-namespace network policy coverage | +| CTR-POD-05 | SEC-10 | Deepens — explicit PSS label validation | +| CTR-ACR-01 | SEC-08, AKS-CIMG-02 | Deepens — ACR-specific scanning validation | + +### Overlap Rules + +1. Container checks that **deepen** an existing matrix check: run the container check commands and include per-container findings in the `Evidence Summary` of the parent matrix check row. Also include a summary in the Container Best Practices Assessment section. +2. Container checks that are **equivalent** to a matrix check: the container check commands replace or supplement the matrix check validation commands. Report results in both sections. +3. Container checks with **no overlap**: report exclusively in the Container Best Practices Assessment section of the report. + +## Scoring Rules + +Container best-practice checks are scored per namespace and per cluster: + +### Per-Namespace Scoring + +- For each check, compute the percentage of containers (or pods, as appropriate) that comply. +- Compliance threshold: **80%** of containers must pass for a namespace to receive `Meets` status. +- Between **50-79%**: `Partially meets`. +- Below **50%**: `Does not meet`. +- If the check is not applicable to the namespace workload type, mark `Not applicable`. + +### Cluster-Wide Scoring + +- Aggregate across all assessed namespaces. +- Compute the overall container compliance percentage per check. +- Use the same thresholds as per-namespace scoring. + +### Rollup Summary + +Produce a rollup table grouped by Container Check Category: + +- `Total Checks` +- `Scored Checks` +- `Meets` +- `Partially meets` +- `Does not meet` +- `Not applicable` +- `Not assessed` +- `Compliance %` + +## Interpretation Guidelines + +- **System namespaces** (`kube-system`, `kube-node-lease`, `kube-public`, `gatekeeper-system`): Platform-managed workloads may legitimately violate some container checks (e.g., `hostNetwork`, privileged containers for CNI/CSI drivers). Report these separately and note that violations in system namespaces are expected for certain infrastructure components. +- **DaemonSets**: Infrastructure DaemonSets (e.g., monitoring agents, log collectors) may require elevated privileges. Flag but do not automatically mark as `Does not meet` — instead note whether the elevation is justified. +- **Jobs and CronJobs**: Batch workloads may not require readiness probes or preStop hooks. Adjust expectations accordingly and note in comments. +- **Init containers**: Resource management checks apply. Health probe checks do not apply to init containers. +- **Per-container vs per-pod checks**: Some checks (e.g., `hostNetwork`, `terminationGracePeriodSeconds`) are pod-level. Others (e.g., `securityContext`, `resources`) are per-container. The commands above target the correct level. diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-diagnostics-detectors.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-diagnostics-detectors.md new file mode 100644 index 000000000..5e1994f40 --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-diagnostics-detectors.md @@ -0,0 +1,147 @@ +# AKS Diagnostics Detectors Reference + +This reference documents the AKS built-in diagnostics detectors available through Azure's "Diagnose and solve problems" capability. These detectors provide runtime risk alerts and health assessments that supplement the static checklist matrix. + +## Extraction Methods (Priority Order) + +### 1. AKS MCP Server (Primary) + +Use the AKS MCP tools to extract diagnostics data programmatically: + +1. **Discover available AKS actions first**: + - Tool: `mcp_azure_mcp_aks` + - Pattern: call discovery on the AKS MCP entry point (for example, `discover`/`learn`) and read the returned action names and schemas. + - Rule: do not hardcode detector action names; use whatever the live MCP surface returns. + +2. **List all available detectors**: + - Tool: `mcp_azure_mcp_aks` + - Action: use the discovered list-detectors action from step 1. + - Parameters: use the discovered schema. This usually includes cluster targeting fields (for example, a full cluster ARM resource ID, or subscription/resource group/cluster name). + - Returns: List of detector names, descriptions, and categories. + +3. **Run detectors by category**: + - Tool: `mcp_azure_mcp_aks` + - Action: use the discovered run-by-category action. + - Parameters: use the discovered schema for category execution and cluster targeting. + - Run once per category listed below. + +4. **Run a specific detector** (for deeper investigation): + - Tool: `mcp_azure_mcp_aks` + - Action: use the discovered run-single-detector action. + - Parameters: use the discovered schema for detector name and cluster targeting. + +> 💡 **Tip:** A live detector-list call via `mcp_azure_mcp_aks` against a representative AKS cluster returned ~119 detectors across 14 categories (8 core + 6 supplemental). Treat the cluster's live inventory as authoritative; the tables below are reference baselines. + +### 2. Azure REST API (Fallback) + +If MCP tools are unavailable, use the AKS Diagnostics REST API via Azure CLI: + +```bash +# List all detectors +az rest --method get \ + --url "/subscriptions//resourceGroups//providers/Microsoft.ContainerService/managedClusters//detectors?api-version=2025-10-01" + +# Run a specific detector +az rest --method get \ + --url "/subscriptions//resourceGroups//providers/Microsoft.ContainerService/managedClusters//detectors/?api-version=2025-10-01" +``` + +### 3. Azure AppLens / Diagnostics API (Alternative) + +```bash +# List available detectors via the diagnostics provider +az rest --method get \ + --url "/subscriptions//resourceGroups//providers/Microsoft.ContainerService/managedClusters//providers/Microsoft.ResourceHealth/diagnostics?api-version=2025-10-01" +``` + +## Detector Categories + +The detector inventory has two layers of category handling: + +1. **Core audit categories**: the eight categories that must always be queried during an audit. +2. **Supplemental live categories**: additional categories that may appear in the live detector inventory for a specific cluster and should be incorporated when present. + +### Core Audit Categories + +| Category | Audit Relevance | Pillar Alignment | +| --- | --- | --- | +| Best Practices | Direct alignment with checklist controls | All pillars | +| Cluster and Control Plane Availability and Performance | API server health, etcd latency, control plane SLA | Reliability, Performance Efficiency | +| Connectivity Issues | DNS, networking, load balancer, ingress, egress | Security, Reliability | +| Create, Upgrade, Delete and Scale | Upgrade readiness, scaling failures, provisioning issues | Reliability, Operational Excellence, Performance Efficiency | +| Deprecations | Deprecated APIs, features, or configurations requiring action | Operational Excellence | +| Identity and Security | RBAC, Microsoft Entra ID, secrets, network policies, Defender findings | Security | +| Node Health | Node readiness, resource pressure, OS/kernel issues | Reliability, Performance Efficiency | +| Storage | PV/PVC issues, CSI driver health, disk attach failures | Reliability | + +### Supplemental Live Categories + +The live detector catalog can also return additional categories. These are not a replacement for the core eight. They should be enumerated using the discovered detector-list action via `mcp_azure_mcp_aks` and incorporated into the report when present. + +| Category | Typical Content | Handling Guidance | +| --- | --- | --- | +| Risk Alerts | Cross-category risk summaries such as availability, reliability, and support-eligibility alerts (e.g. `riskalerts-availability`, `riskalerts-reliability`, `riskalerts-support-eligibility`) | Always capture. Treat as high-value supplemental evidence and include all Critical and Warning results in the Risk Alerts table. | +| Control Plane | Focused control plane detectors such as scheduler-specific analysis (e.g. `KubeScheduler`) | Capture and cross-reference to reliability and performance checks where applicable. | +| Keystone | Service-specific platform diagnostics surfaced by the AKS diagnostics backend (e.g. `keystone`) | Capture as supplemental evidence. If no checklist item maps cleanly, keep in the diagnostics section only. | +| Workflow Nodes | Guided troubleshooting nodes rather than direct compliance detectors (e.g. `operation-troubleshooter`, Kusto spoke providers) | Record availability and use them to drive deeper investigation, but do not score them as standalone checklist controls unless they emit concrete findings. | +| zWorkflowDetectors | Reconciliation workflow detectors (e.g. `ReconcileManagedCluster`, `ReconcileNodePool`) used for orchestration rather than direct compliance scoring | Record availability. Run only when investigating a specific reconcile/operation failure; do not score as standalone checklist controls. | +| Analysis - L2 Support Topic | Topic-level analysis aggregators that fan out to multiple underlying detectors (e.g. `st-l2-analysis-connectivity`, `st-l2-analysis-crud`, `st-l2-analysis-storage`) | Run when category-level execution is needed for the matching topic. Treat outputs as cross-references to the relevant core category rather than as new checklist controls. | +| Analysis - L3 Support Topic | Scenario-specific analysis nodes for targeted investigations (e.g. `st-l3-analysis-cluster-upgrade`, `st-l3-node-high-cpu-mem`, `st-l3-analysis-creating-nodepool`) | Use on demand to drill into a specific failure scenario surfaced by core detectors or risk alerts. Capture findings as supplemental evidence linked to the impacted checklist item. | + +### Category Execution Rules + +1. Query all eight core audit categories on every audit. +2. Enumerate supplemental live categories from the detector inventory on every audit. +3. If a supplemental category supports category-level execution, run it. +4. If category-level execution is not supported, run the underlying detectors individually where possible. +5. If a supplemental category is inventory-only (for example a workflow or overview node), record it as supplemental metadata with handling notes. +6. Do not let supplemental-category coverage replace or weaken the requirement to query the core eight. + +## Severity Levels + +Detector findings use the following severity levels: + +| Severity | Meaning | Audit Impact | +| --- | --- | --- | +| Critical | Active issue causing service impact or significant risk | Must be addressed; may override checklist status to `Does not meet` | +| Warning | Potential risk or degraded posture detected | Should be addressed; may affect checklist status | +| Info | Informational finding with no immediate risk | Record as evidence; no status change required | +| Healthy | Detector ran and no issues detected | Positive evidence for checklist items | +| None | Detector returned no severity (informational or not applicable) | Record as supplemental context; verify the detector executed correctly | + +## Mapping Detectors to Checklist Items + +When a detector finding relates to an existing checklist item (from the checklist matrix), cross-reference it: + +| Detector Category | Likely Checklist Alignment | +| --- | --- | +| Best Practices | `AKS-OP-*`, `AKS-DEV-*`, `AKSC-*`, `COST-*`, `CTR-*`, `OPS-04` | +| Cluster and Control Plane Availability and Performance | `REL-01` through `REL-08`, `PERF-01` through `PERF-05`, `OPS-03` | +| Connectivity Issues | `SEC-03`, `SEC-04`, `SEC-09`, `AKS-NET-*`, `AKSC-NET-*` | +| Create, Upgrade, Delete and Scale | `AKS-CSEC-01`, `AKS-CSEC-02`, `OPS-01`, `OPS-02` | +| Deprecations | `AKS-CSEC-01`, `OPS-01` | +| Identity and Security | `SEC-01` through `SEC-10`, `AKS-ID-*`, `OPS-05` | +| Node Health | `REL-04`, `REL-05`, `REL-06`, `PERF-01`, `PERF-03` | +| Storage | `AKS-STO-01`, `AKS-STO-02`, `AKSC-BCDR-03` | + +### Cross-Reference Rules + +1. If a detector finding directly validates or contradicts a checklist item, update that item's `Evidence Summary` and `Comments` with the detector output. +2. If a detector finding has no matching checklist item, include it only in the AKS Diagnostics Findings section (not in the main Detailed Analysis table). +3. If a Critical detector finding contradicts a checklist item's `Meets` status, re-evaluate and downgrade the status with explanation. +4. Detector findings supplement but do not replace validation command evidence for checklist items. +5. Supplemental categories such as `Risk Alerts`, `Control Plane`, and `Keystone` can influence checklist status when they contain concrete detector findings. +6. `Workflow Nodes`, `zWorkflowDetectors`, and category-overview detectors (e.g. `aks-category-*`) are supporting evidence and orchestration aids; they should not be scored as direct control checks unless a concrete detector result is also captured. +7. `Analysis - L2/L3 Support Topic` detectors are scenario aggregators; map their findings back to the matching core category rather than treating them as new controls. + +## Interpretation Guidelines + +- **Risk Alerts**: The AKS diagnostics API exposes a concrete `Risk Alerts` category that aggregates cross-category risk summaries (see Supplemental Live Categories above). This is distinct from the general concept of risk-level findings, which are any `Critical` or `Warning` results returned by detectors in any category. Capture the `Risk Alerts` category results in the Risk Alerts table; cross-reference individual Critical/Warning findings from other categories to their respective checklist items. +- **Troubleshooting Results**: The full set of detector outputs organized first by the eight core audit categories, then by any supplemental live categories. Even `Healthy` results provide positive evidence for the audit. +- **Detector Descriptions**: Each detector includes a description of what it checks. Record this in the evidence to explain what was assessed. +- **Time Sensitivity**: Detector results are point-in-time snapshots. Record the timestamp of execution in the report. +- **Supplemental Category Semantics**: A live category can represent a concrete detector bucket, a summary view, or a troubleshooting workflow. Preserve the returned detector type in the report so readers understand whether the item is scored evidence or supporting context. + +## Guardrails / Safety + +- Do not skip execution of any detector. ALL detectors MUST be run. diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-container-assessment.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-container-assessment.md new file mode 100644 index 000000000..60b8d302e --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-container-assessment.md @@ -0,0 +1,42 @@ +# Container Assessment Workflow + +## Step 5: Run Container Best Practices Assessment + +See [AKS Container Best Practices](./aks-container-best-practices.md) for complete checks, commands, and scoring. + +### Namespace Setup + +Separate system namespaces (`kube-system`, `kube-node-lease`, `kube-public`, `gatekeeper-system`) from application namespaces for distinct reporting. + +### Check Categories + +Execute all validation commands for: + +| Category | Check IDs | Scope | +|----------|-----------|-------| +| Image Hygiene | `CTR-IMG-*` | Tag pinning, pull policies, private registries, ACR | +| Security Context | `CTR-SEC-*` | runAsNonRoot, readOnlyRootFilesystem, privileges, capabilities, seccomp | +| Resource Management | `CTR-RES-*` | CPU/memory requests/limits, QoS distribution, limit-to-request ratios | +| Health Probes & Lifecycle | `CTR-PROBE-*`, `CTR-LIFE-*` | Readiness/liveness/startup probes, preStop hooks, rolling updates | +| Pod Security | `CTR-POD-*` | Service account tokens, hostNetwork/PID/IPC, hostPath, PSS labels | +| Config Hygiene | `CTR-CFG-*` | Inline secrets, ConfigMap/Secret usage, naked pods, ports | +| Network | `CTR-NET-*` | NetworkPolicy coverage, default-deny, service types | +| Supply Chain | `CTR-ACR-*` | Vulnerability scanning, content trust, retention, webhooks | + +### Counting Methodology + +Count every **running container instance** (not unique specs) — includes scaled replicas and injected sidecars. Separately report sidecar/proxy container counts. State methodology explicitly in report. + +### Sidecar-Aware Compliance + +Where sidecars materially affect metrics (e.g., `runAsNonRoot`, readiness probes), report both sidecar-inclusive and application-only percentages. + +### Scoring + +- Compute per-namespace and cluster-wide statistics +- Thresholds: ≥80% = Meets, 50-79% = Partially meets, <50% = Does not meet +- Cross-reference with checklist matrix overlap mapping + +### Fallback + +If `kubectl` unavailable, use `az aks command invoke`. If also unavailable, mark `CTR-*` checks `Not assessed` except platform-level checks (`CTR-IMG-06`, `CTR-ACR-*`) assessable via `az` CLI. diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-diagnostics.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-diagnostics.md new file mode 100644 index 000000000..f246f4749 --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-diagnostics.md @@ -0,0 +1,25 @@ +# Diagnostics Workflow + +## Step 3: Run Diagnostics Detectors + +See [AKS Diagnostics Detectors Reference](./aks-diagnostics-detectors.md) for extraction methods and interpretation rules. + +### Primary Method: AKS MCP Server + +1. Call `mcp_azure_mcp_aks` first to discover available AKS actions and parameter schemas (do not hardcode action names). +2. **Core audit categories** (always query all 8): `Best Practices`, `Cluster and Control Plane Availability and Performance`, `Connectivity Issues`, `Create, Upgrade, Delete and Scale`, `Deprecations`, `Identity and Security`, `Node Health`, `Storage`. +3. Use `mcp_azure_mcp_aks` with the discovered run-by-category action for each core category. +4. Enumerate **supplemental categories** from live catalog (e.g., `Risk Alerts`, `Control Plane`, `Keystone`, `Workflow Nodes`). +5. For supplemental categories: prefer category-level execution using the discovered action; if unsupported, run individual detectors using the discovered run-single-detector action and record why. +6. For Critical/Warning findings needing deeper investigation, call `mcp_azure_mcp_aks` with the discovered single-detector action and the target detector name. + +### Fallback Method + +If MCP unavailable, use `az rest` calls to AKS Diagnostics REST API (see detectors reference). + +### Post-Processing + +- Record extraction timestamp (UTC and local time). +- Classify findings by severity: `Critical`, `Warning`, `Info`, `Healthy/None`. +- Cross-reference findings to checklist items using detectors reference mapping. +- If a Critical finding contradicts a checklist item's `Meets` status, downgrade and record justification. diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-quality-gates.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-quality-gates.md new file mode 100644 index 000000000..3550f36dc --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-quality-gates.md @@ -0,0 +1,77 @@ +# Quality Gates & Decision Logic + +## Pre-Finalization Verification + +Before finalizing, verify every item: + +### Checklist Coverage + +- Every matrix item represented in the report +- Every row has ≥1 executed command or explicit `Not assessed` reason +- Every row has a concise `Comments` value and ≤5 remediation actions +- All references point to Microsoft Learn +- Scope and out-of-scope statements are explicit +- Subpage rollup includes every `Parent Page/Child Page` group +- AKS Checklist clashes resolved in favor of Microsoft Learn + +### Diagnostics + +- All 8 core detector categories queried and present +- Supplemental categories enumerated and assessed (or marked informational-only with rationale) +- Critical findings contradicting checklist status reconciled with justification +- Extraction timestamp recorded (UTC + local) + +### Warning Events + +- Section present and populated (or marked `Not assessed` with reason) +- Every Warning row includes resolved top-level workload (not Pod name) and 1-5 remediation steps +- Extraction timestamp recorded (UTC + local) +- Time window ≥30 days with Azure Monitor Logs; if etcd-only, limitation documented +- Probe deep-dive present for workloads meeting threshold (≥10 occurrences or ≥7 day span) +- Deep-dive references specific log patterns, probe config, dependency interactions + +### Container Assessment + +- Section present and populated (or marked `Not assessed` with reason) +- Summary statistics, category rollup, and detail table all present +- All `CTR-*` checks from container best practices reference covered +- System namespace findings reported separately; counting methodology stated +- Sidecar-inclusive and app-only percentages reported where material + +### Validation Integrity + +- Node image staleness checked; Critical if >90 days old +- Inline secrets (`CTR-CFG-01`) reports exact container count +- Next Steps uses time-boxed roadmap with CLI commands for Immediate items +- Commands Reference table present for audit reproducibility +- No check marked `Meets` when command failed/timed out/returned insufficient evidence + +## Decision Logic + +### Access Fallback Chain + +| Scenario | Action | +|----------|--------| +| `az` unavailable | `kubectl` only; mark Azure checks `Not assessed` | +| `kubectl` unavailable | `az aks command invoke`; if also unavailable, mark Kubernetes checks `Not assessed` | +| MCP tools unavailable | `az rest` API calls; if both unavailable, mark Diagnostics `Not assessed` | +| `kubectl` unavailable for `CTR-*` | `az aks command invoke`; if unavailable, mark `CTR-*` `Not assessed` except `CTR-IMG-06`/`CTR-ACR-*` (via `az` CLI) | +| Insufficient permissions | Mark `Not assessed`; record exact missing permission | +| Not applicable to cluster | Set `Not applicable`; explain in Comments | + +### Conservative Assessment + +- Failed/timed-out/ambiguous → `Not assessed` (never infer `Meets` from absent evidence) +- Risk-acceptance judgments: state assumption in Comments; `Meets` only if evidence positively supports + +### Service Mesh Interaction + +When Linkerd/Istio detected, evaluate impact on probe failures, security context metrics, sidecar counts. Factor mesh-specific behaviors into probe deep-dive and container compliance scoring. + +## Guardrails + +- Do not request or output secrets (tokens, keys) +- Do not use other scripts; follow this skill's instructions and template only +- Do not use existing reports as sources +- Place temporary files in `/ - /` +- Save temporary files in JSON format diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-report.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-report.md new file mode 100644 index 000000000..5142d99e0 --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-report.md @@ -0,0 +1,36 @@ +# Report Generation Workflow + +## Step 8: Generate Markdown Report + +Use [AKS Audit Report Template](../assets/aks-audit-report-template.md) and fill all sections. + +### Report Sections + +1. Table of contents +2. Scope +3. Overview of AKS best practices +4. Detailed analysis table per pillar/checklist item +5. AKS Diagnostics Findings +6. Container Best Practices Assessment +7. Kubernetes Warning Events by Namespace +8. Summary +9. Next steps +10. Appendices + +### Required Characteristics + +- **Table of contents** with working anchors +- **Scope**: cluster identifier, extraction timestamps (UTC + local), Kubernetes version, region, node count with pool breakdown, out-of-scope, limitations +- **Detailed analysis**: all checklist items from both source pages +- **Summary**: 1-2 paragraphs +- **Next Steps** — time-boxed remediation roadmap: + - **Immediate (0-2 weeks)**: Critical security/reliability with CLI commands + - **Short-term (2-6 weeks)**: High-severity hardening (network policies, secrets migration, PSS) + - **Medium-term (1-3 months)**: Operational maturity (probes, privileged containers, GitOps) + - **Long-term (3-6 months)**: Strategic architecture (image digests, multi-region DR, chaos engineering) +- **Commands Reference**: Table of every major command category with exact syntax for reproducibility +- **Appendices**: + - Full checklist table (Check ID, Parent/Child Page, Pillar, Item, Commands, Output) + - Full diagnostics findings with raw evidence and exact commands/output + - Full container best practices results with per-namespace breakdowns + - Full Kubernetes Warning events with remediation diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-scope-and-checklist.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-scope-and-checklist.md new file mode 100644 index 000000000..736b8f1ab --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-scope-and-checklist.md @@ -0,0 +1,20 @@ +# Scope & Checklist Workflow + +## Step 1: Establish Review Scope + +1. Confirm exact AKS cluster: `subscriptionId`, `resourceGroup`, `clusterName`, `kubeContext`. +2. State scope: AKS cluster platform and in-cluster Kubernetes configuration only. +3. State out-of-scope: non-AKS Azure resources unless directly required for AKS controls. +4. Record limitations: permissions, missing add-ons, private API server access, disconnected networks, preview features. + +## Step 2: Load Canonical Checklist + +1. Use [AKS Checklist Matrix](./aks-checklist-matrix.md) as base list. +2. Keep both sources in scope: + - AKS best-practices categories (operator/developer) + - Well-Architected AKS pillars (Reliability, Security, Cost Optimization, Operational Excellence, Performance Efficiency) +3. Include deeper child-page checks: cluster isolation, scheduler (basic/advanced), identity, cluster security, container image management, network, storage, multi-region, resource management, pod security, app/cluster reliability. +4. Incorporate AKS Checklist sections: `Identity`, `Cluster security`, `Networking`, `Storage`, `Resource management`, `Cluster operations`, `BCDR`, `Windows`, `Application deployment`, `Image management`. +5. **Source precedence**: Microsoft Learn wins when guidance clashes. AKS Checklist is supplementary. +6. Never silently drop items. If a check cannot execute, mark `Not assessed` with explanation. +7. Use strict hierarchy (`Parent Page -> Child Page -> Check ID`) for rollup. diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-validation-and-findings.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-validation-and-findings.md new file mode 100644 index 000000000..b1214b473 --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-validation-and-findings.md @@ -0,0 +1,47 @@ +# Validation & Findings Workflow + +## Step 6: Run Validation Commands + +1. Execute Azure-platform checks with `az`. +2. **Primary**: Kubernetes checks with `kubectl`. **Fallback**: `az aks command invoke`. If both unavailable, mark `Not assessed`. +3. Prefer read-only commands. +4. Capture evidence: short outputs, key fields, timestamps (UTC and local). +5. **Node image staleness**: `az aks nodepool list -g --cluster-name --query "[].{name:name,nodeImageVersion:nodeImageVersion}"`. Flag Critical if any pool image >90 days old. Cross-reference `nodeOsUpgradeChannel`. +6. **Inline secrets** (`CTR-CFG-01`): Scan all container env vars for sensitive values. Report exact count — don't dismiss partial findings. +7. Determine status per check: `Meets`, `Partially meets`, `Does not meet`, `Not applicable`, `Not assessed`. +8. **Conservative rule**: Failed/timed-out/insufficient commands → `Not assessed`, never `Meets`. + +## Step 7: Produce Detailed Findings + +### Per-Check Analysis Table + +Each checklist item gets one row with ALL columns: Check ID, Parent Page, Child Page, Pillar/category, Checklist item, Control intent summary, Microsoft Learn URL, Commands run, Evidence summary, Comments, Status, Remediation actions (up to 5). + +### Rollup Table + +Group by `Parent Page`/`Child Page`: Total Checks, Scored Checks, Meets, Partially meets, Does not meet, Not applicable, Not assessed, Compliance %. + +### AKS Diagnostics Findings Section + +- **Risk Alerts table**: All Critical/Warning findings from any detector category +- **Detector Results by Category**: Full results from 8 core categories with cross-referenced check IDs +- **Supplemental Categories**: Additional live categories with detector type (`Detector`, `CategoryOverview`, `WorkflowNode`) and handling notes +- **Diagnostics Impact table**: Checklist items whose status was adjusted by detector findings + +### Container Best Practices Section + +- **Summary Statistics**: Cluster-wide counts/percentages (total containers, without requests, running as root, without probes, latest tags, BestEffort QoS, privileged, inline secrets) +- **Category Rollup**: One row per check category with compliance % +- **Detail Table**: One row per `CTR-*` check with ID, Category, Item, Commands, Compliance %, Per-Namespace Breakdown, Status, Remediation + +### Warning Events Section + +- **Summary Table**: Namespace, Event Reason, Message Summary, Occurrences, First/Last Seen, Source, Affected Workload(s) (resolved top-level), Remediation Steps (1-5) +- **Probe Failure Deep-Dive**: Sub-section per affected namespace with Workload, Live Evidence, Interpretation, Recommended Change columns +- **Time Window note**: State effective correlation window in preamble + +### Remediation Rules + +- Implementation-ready with exact Azure/Kubernetes feature names +- Prefer least-privilege and policy-driven controls +- Sequence: short-term mitigation → medium-term hardening → long-term architecture diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-warning-events.md b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-warning-events.md new file mode 100644 index 000000000..709f61466 --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/aks-review-warning-events.md @@ -0,0 +1,54 @@ +# Warning Events Workflow + +## Step 4: Collect Kubernetes Warning Events by Namespace + +### Collection + +1. **Primary**: `kubectl`. **Fallback**: `az aks command invoke`. If both unavailable, mark `Not assessed`. +2. Enumerate namespaces: `kubectl get namespaces -o jsonpath='{.items[*].metadata.name}' --context ` +3. Per namespace, retrieve Warning events: + ```bash + kubectl get events --namespace --field-selector type=Warning -o json --context + ``` +4. **30-day coverage**: Supplement etcd events with Azure Monitor Logs (Container Insights): + ```kql + KubeEvents + | where TimeGenerated > ago(30d) + | where Reason in ("Unhealthy","BackOff","FailedScheduling","FailedMount","OOMKilling","FailedCreate") + | summarize Count=sum(Count), FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated) + by Namespace, Name, Reason, Message, SourceComponent + | order by Count desc + ``` + If unavailable, note limitation and use etcd events only. +5. Merge and deduplicate by `(namespace, reason, message-pattern, workload)`, using widest time window. + +### Event Processing + +Extract per Warning: `reason`, `message`, `involvedObject.kind/.name`, `count`, `firstTimestamp`/`lastTimestamp`, `source.component`. + +### Workload Correlation + +Resolve each Pod to top-level workload owner chain: Pod → ReplicaSet → Deployment (or StatefulSet/DaemonSet/Job → CronJob). + +```bash +kubectl get pod -n -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}' --context +``` + +Record as `/` (e.g., `Deployment/my-app`). For ReplicaSet owners, resolve one level further to Deployment. + +### Remediation + +Per deduplicated Warning, produce up to 5 remediation steps: specific, implementation-ready, sequenced from quick fix to architectural. + +### Probe Failure Deep-Dive + +For workloads with `Unhealthy` events meeting threshold (**≥10 occurrences** or **≥7 day span**): + +1. Collect deployment spec (probe config), app logs (48h, tail 250), sidecar/proxy logs (48h, tail 150). +2. Build root-cause analysis per workload: + - **Live Evidence**: Synthesize spec, probes, events, logs. Reference specific errors: timeouts, circuit-breaker, HTTP 502/503, dependency throttling, protocol-detection timeouts. + - **Interpretation**: True crash vs. dependency latency vs. startup timing vs. rollout churn vs. probe misconfiguration. Steady-state vs. transient. + - **Recommended Change**: Implementation-ready — health endpoint redesign, circuit-breaker tuning, startup probe introduction, rollout smoothing. +3. **Service mesh awareness**: When Linkerd/Istio detected, evaluate proxy initialization timing impact. Look for HTTP 502 during startup, proxy readiness failures, `connection refused`. Recommend mesh-specific mitigations (e.g., `config.linkerd.io/proxy-await=enabled`, startup probe budgets, proxy resource tuning). + +Record extraction timestamp (UTC and local time). diff --git a/plugins/azure-skills/skills/azure-kubernetes/references/azure-aks-review.md b/plugins/azure-skills/skills/azure-kubernetes/references/azure-aks-review.md new file mode 100644 index 000000000..171ed51ca --- /dev/null +++ b/plugins/azure-skills/skills/azure-kubernetes/references/azure-aks-review.md @@ -0,0 +1,65 @@ +# AKS Cluster Review / Audit (Day-2) + +Use this workflow when the user asks to **review**, **audit**, or **assess the compliance posture** of an existing AKS cluster (e.g. "review AKS", "audit AKS cluster", "AKS practices review", "validate AKS posture", "AKS compliance checklist", "AKS remediation report"). + +It produces an evidence-driven AKS assessment against [AKS best practices](https://learn.microsoft.com/en-us/azure/aks/best-practices), the [Well-Architected AKS service guide](https://learn.microsoft.com/en-us/azure/well-architected/service-guides/azure-kubernetes-service), the [AKS Checklist](https://www.the-aks-checklist.com/), live diagnostics detectors, and [container best practices](./aks-container-best-practices.md), and emits a comprehensive Markdown audit report. + +## Required Inputs + +| Input | Required | Default | +|-------|----------|---------| +| `subscriptionId` | Yes | — | +| `resourceGroup` | Yes | — | +| `clusterName` | Yes | — | +| `kubeContext` | Yes | — | +| `reportOutputPath` | No | `./reports/azure kubernetes review - - .md` | +| `namespaceFilter` | No | all | +| `nodePoolFilter` | No | all | +| `excludeChecks` | No | none | + +## Review Workflow + +1. **Establish scope** — Confirm cluster identity, state in/out-of-scope, record limitations. See [Scope & Checklist](./aks-review-scope-and-checklist.md). +2. **Load canonical checklist** — Use the [AKS Checklist Matrix](./aks-checklist-matrix.md) with Microsoft Learn precedence. See [Scope & Checklist](./aks-review-scope-and-checklist.md). +3. **Run diagnostics detectors** — Query the 8 core + supplemental categories via AKS MCP or `az rest`. See [Diagnostics Workflow](./aks-review-diagnostics.md). +4. **Collect warning events** — Gather Warning-type Kubernetes events per namespace, correlate to workloads, deep-dive probe failures. See [Warning Events Workflow](./aks-review-warning-events.md). +5. **Run container assessment** — Execute all `CTR-*` checks. See [Container Assessment Workflow](./aks-review-container-assessment.md). +6. **Run validation commands** — Execute `az` / `kubectl` checks, determine pass/fail per checklist item. See [Validation & Findings](./aks-review-validation-and-findings.md). +7. **Produce detailed findings** — Build analysis tables, rollups, diagnostics impact, container results, warning events. See [Validation & Findings](./aks-review-validation-and-findings.md). +8. **Generate report** — Fill the [AKS Audit Report Template](../assets/aks-audit-report-template.md). See [Report Generation](./aks-review-report.md). +9. **Quality gates** — Verify completeness. See [Quality Gates & Decision Logic](./aks-review-quality-gates.md). + +## Review Decision Logic + +See [Quality Gates & Decision Logic](./aks-review-quality-gates.md) for full rules. Key principles: + +- If `az` unavailable → `kubectl` only, mark Azure checks `Not assessed` +- If `kubectl` unavailable → `az aks command invoke`, then `Not assessed` +- If MCP unavailable → `az rest`, then `Not assessed` +- Never infer `Meets` from absent evidence + +## Review Output + +One Markdown file at `reportOutputPath`: `Azure Kubernetes Service Review - - .md`. `reportOutputPath` is the full file path for the final Markdown report, not a directory. + +## Review Guardrails + +- Do not skip execution of any checklist, matrix, detector or workflow step. +- ALL diagnostics detectors MUST be run to surface evidence and findings. Do not skip or cherry-pick based on expected outcomes, even if they are expected to return no findings, even if you think it would be inefficient. +- Do not request or output secrets (tokens, keys). +- Follow this skill's instructions and [template](../assets/aks-audit-report-template.md) only — no other scripts or existing reports. +- Place temporary files in `/ - /`. + +## Review References + +- [AKS Checklist Matrix](./aks-checklist-matrix.md) +- [AKS Diagnostics Detectors](./aks-diagnostics-detectors.md) +- [AKS Container Best Practices](./aks-container-best-practices.md) +- [AKS Audit Report Template](../assets/aks-audit-report-template.md) +- [Scope & Checklist Workflow](./aks-review-scope-and-checklist.md) +- [Diagnostics Workflow](./aks-review-diagnostics.md) +- [Warning Events Workflow](./aks-review-warning-events.md) +- [Container Assessment Workflow](./aks-review-container-assessment.md) +- [Validation & Findings](./aks-review-validation-and-findings.md) +- [Report Generation](./aks-review-report.md) +- [Quality Gates & Decision Logic](./aks-review-quality-gates.md)