feat: add azure-iac-security skill for pre-deployment IaC security scanning

evals/azure-iac-security/eval.yaml

@@ -0,0 +1,148 @@
+# Vally eval config for azure-iac-security
+#
+# Integration evaluation — verifies the skill is invoked for pre-deployment IaC security
+# scanning prompts (ARM/Bicep/Terraform), including AI-workload templates and attack-path
+# requests, AND that it does NOT route for runtime compliance audits (azure-compliance) or
+# deploy-readiness checks (azure-validate). The "When to Use" table in SKILL.md documents
+# the boundary.
+
+name: azure-iac-security-integration-eval
+description: |
+  Integration evaluation for the azure-iac-security skill. Tests skill routing for
+  static, pre-deployment security scanning of ARM, Bicep, and Terraform templates against
+  MCSB v3.0, MITRE ATT&CK, and AI-workload risk, using invocation-rate measurement
+  (5 runs, 80% threshold).
+
+tags:
+  type: integration
+  skill: azure-iac-security
+
+config:
+  runs: 5
+  timeout: "10m"
+  executor: integration-test-agent-runner
+  model: claude-sonnet-4.6
+
+scoring:
+  threshold: 0.8
+
+stimuli:
+  # ── bicep-scan-prompt ──
+  - name: "Scan Bicep for security issues"
+    prompt: "Scan this Bicep template for security issues before I deploy it: storage account with supportsHttpsTrafficOnly set to false."
+    tags:
+      type: integration
+      tier: full
+      cost: llm
+      area: routing
+      earlyTerminate: '[{"type":"skill-call","skill":"azure-iac-security"},{"type":"tool-call-count","count":3}]'
+    graders:
+      - type: skill-invocation
+        config:
+          required:
+            - azure-iac-security
+      - type: output-not-matches
+        config:
+          pattern: "(?i)fatal error|unhandled exception|stack trace"
+
+  # ── arm-mcsb-prompt ──
+  - name: "Check ARM template against MCSB"
+    prompt: "Review this ARM template for MCSB compliance and list the security misconfigurations with fixes."
+    tags:
+      type: integration
+      tier: full
+      cost: llm
+      area: routing
+      earlyTerminate: '[{"type":"skill-call","skill":"azure-iac-security"},{"type":"tool-call-count","count":3}]'
+    graders:
+      - type: skill-invocation
+        config:
+          required:
+            - azure-iac-security
+      - type: output-not-matches
+        config:
+          pattern: "(?i)fatal error|unhandled exception|stack trace"
+
+  # ── terraform-scan-prompt ──
+  - name: "Terraform IaC security scan"
+    prompt: "Check my Terraform azurerm configuration for security vulnerabilities and misconfigurations before deployment."
+    tags:
+      type: integration
+      tier: full
+      cost: llm
+      area: routing
+      earlyTerminate: '[{"type":"skill-call","skill":"azure-iac-security"},{"type":"tool-call-count","count":3}]'
+    graders:
+      - type: skill-invocation
+        config:
+          required:
+            - azure-iac-security
+      - type: output-not-matches
+        config:
+          pattern: "(?i)fatal error|unhandled exception|stack trace"
+
+  # ── attack-path-prompt ──
+  - name: "MITRE attack path from IaC findings"
+    prompt: "Generate MITRE ATT&CK attack paths from the security findings in my Bicep infrastructure template."
+    tags:
+      type: integration
+      tier: full
+      cost: llm
+      area: routing
+      earlyTerminate: '[{"type":"skill-call","skill":"azure-iac-security"},{"type":"tool-call-count","count":3}]'
+    graders:
+      - type: skill-invocation
+        config:
+          required:
+            - azure-iac-security
+      - type: output-not-matches
+        config:
+          pattern: "(?i)fatal error|unhandled exception|stack trace"
+
+  # ── ai-workload-scan-prompt ──
+  - name: "Scan Azure OpenAI IaC for security"
+    prompt: "Check this Bicep that deploys an Azure OpenAI account for security issues before deployment — publicNetworkAccess is Enabled and disableLocalAuth is false."
+    tags:
+      type: integration
+      tier: full
+      cost: llm
+      area: routing
+      earlyTerminate: '[{"type":"skill-call","skill":"azure-iac-security"},{"type":"tool-call-count","count":3}]'
+    graders:
+      - type: skill-invocation
+        config:
+          required:
+            - azure-iac-security
+      - type: output-not-matches
+        config:
+          pattern: "(?i)fatal error|unhandled exception|stack trace"
+
+  # ── negative: runtime compliance (should NOT route here — belongs to azure-compliance) ──
+  - name: "Runtime compliance audit does not route to IaC scanner"
+    prompt: "Audit my already-deployed storage account for compliance against Azure Security Benchmark."
+    tags:
+      type: integration
+      tier: full
+      cost: llm
+      area: routing
+      earlyTerminate: '[{"type":"tool-call-count","count":4}]'
+    graders:
+      - type: skill-invocation
+        config:
+          disallowed:
+            - azure-iac-security
+
+  # ── negative: deploy-readiness (should NOT route here — belongs to azure-validate) ──
+  - name: "Deploy-readiness check does not route to IaC scanner"
+    prompt: "Will my ARM template deploy successfully? Validate it for deployment errors."
+    tags:
+      type: integration
+      tier: full
+      cost: llm
+      area: routing
+      earlyTerminate: '[{"type":"tool-call-count","count":4}]'
+    graders:
+      - type: skill-invocation
+        config:
+          disallowed:
+            - azure-iac-security

plugin/skills/azure-iac-security/SKILL.md

@@ -0,0 +1,141 @@
+---
+name: azure-iac-security
+description: "Static pre-deployment security scan of Azure IaC (ARM, Bicep, Terraform) against MCSB v3.0 and MITRE ATT&CK. WHEN: \"scan my IaC for security before deploy\", \"check my template against MCSB\"."
+license: MIT
+metadata:
+  author: Microsoft
+  version: "0.0.0-placeholder"
+---
+
+# Azure IaC Security
+
+Static, pre-deployment security analysis of Azure Infrastructure as Code (ARM JSON, Bicep,
+Terraform). Catches misconfigurations **in the template before deployment**, mapped to
+Microsoft Cloud Security Benchmark (MCSB) v3.0 and MITRE ATT&CK.
+
+## Quick Reference
+
+| Property | Details |
+|---|---|
+| Best for | Scanning IaC template *files* for security issues before deploy |
+| Inputs | ARM JSON, Bicep, or Terraform template files |
+| Frameworks | MCSB v3.0 (NS, IM, PA, DP, LT pillars), MITRE ATT&CK; MITRE ATLAS + OWASP LLM Top 10 for AI workloads |
+| Output | JSON findings: control, severity, vulnerable code, code fix, Learn URL |
+| Grounding | **Live Microsoft Learn MCP is authoritative**; static tables are seed hints only |
+| Not for | Auditing *deployed* resources (use `azure-compliance`) or deploy readiness (use `azure-validate`) |
+
+## When to Use This Skill
+
+Activate when the user wants to:
+- Scan an ARM/Bicep/Terraform file for security vulnerabilities or misconfigurations
+- Check IaC against MCSB v3.0 (network, identity, data protection, logging)
+- Harden a template before deployment (shift-left security)
+- Scan **AI/ML workloads** (Azure OpenAI, AI Services, ML workspaces, AI Search) for insecure
+  configuration and map to MITRE ATLAS / OWASP LLM Top 10
+- Generate MITRE ATT&CK attack paths from IaC findings
+- Get code-level remediation for insecure resource properties
+
+**Do NOT use for:** runtime/deployed-resource audits (→ `azure-compliance`) or
+"will it deploy" preflight checks (→ `azure-validate`).
+
+## Scope Guard (refuse out-of-scope requests)
+
+This skill does **one** thing: static, pre-deployment **security analysis of Azure IaC**. When
+invoked — even if invoked explicitly — it must first confirm the request is in scope. If it is
+not, **refuse in one line stating this skill's purpose and do not perform the unrelated task**:
+
+- **No Azure IaC to scan** (no template file/content provided, or only a general question) →
+  ask for an ARM/Bicep/Terraform template; do not fabricate a scan.
+- **Non-Azure IaC** (AWS CloudFormation, GCP, or **raw K8s/Helm manifests** — *not* an AKS
+  cluster authored in ARM/Bicep/Terraform, which IS in scope) → out of scope, zero findings,
+  redirect to the native scanner (see [template parsing](references/template-parsing.md)).
+- **Unrelated task** (recipes, prose, general coding, chit-chat, anything not IaC security) →
+  refuse and restate purpose. Do **not** answer the unrelated request. Example reply:
+  `The azure-iac-security skill only performs static security scanning of Azure IaC (ARM, Bicep, Terraform). It can't help with that request.`
+
+## MCP Tools
+
+| Tool | Purpose |
+|---|---|
+| `microsoft_docs_search` | Ground control guidance in official MCSB / Azure security docs |
+| `microsoft_docs_fetch` | Fetch a full Learn page for a control or resource property |
+| `bicepschema_get` | Look up valid property names / API versions and secure defaults for a Bicep resource type |
+
+**Grounding-first (authoritative).** Static reference tables in this skill are *seed hints* that
+go stale fast — **Azure ships new features and new security properties for services daily**, MCSB
+is revised, and new resource types (AI and otherwise) appear constantly. No hardcoded table can
+keep up. The **live Microsoft Learn** docs, exposed through the already-registered `azure` MCP
+server (`@azure/mcp`), are the source of truth. On **every scan**, for each resource type, use
+`microsoft_docs_search` / `microsoft_docs_fetch` (and `bicepschema_get`) to:
+1. **Reconcile** each seed check's control ID, name, Docs URL, and the resource's current secure
+   default before emitting it; and
+2. **Discover newly-shipped security-relevant properties/checks** for that resource type/API
+   version that are *not* in the seed tables, and evaluate those too.
+This keeps coverage current for every service without waiting for a table update. Every finding
+cites a `learn.microsoft.com` URL. If MCP is unavailable, fall back to seed-table URLs and mark
+the finding `grounding: offline` (note that new-feature discovery is skipped).
+
+## Workflow
+
+1. **Detect template type** — read the file(s). ARM JSON has `"$schema"` + `resources[]`;
+   Bicep has `resource` declarations with `@` API versions; Terraform has `azurerm_` blocks.
+   **Scope gate:** if the file is non-Azure IaC (AWS CloudFormation `AWSTemplateFormatVersion` /
+   `AWS::*`, `provider "aws"`/`aws_*`, GCP, or raw K8s/Helm manifests — an **AKS cluster** in
+   ARM/Bicep/Terraform is Azure IaC and IS in scope), **stop immediately with zero
+   findings** — state it is out of scope and redirect to the relevant native scanner (e.g.
+   `cfn-lint`, Checkov). Do not emit MCSB findings or a best-effort scan. See
+   [template parsing](references/template-parsing.md).
+2. **Extract resources** — enumerate each resource with its `type`, name, properties, and
+   source line number. See [template parsing](references/template-parsing.md).
+3. **Preprocess** — resolve `[parameters()]`/`[variables()]`/ARM functions, associate any
+   `.parameters.json` / `.bicepparam` files, and scan once per environment. Detect secrets in
+   parameter values. See [template preprocessing](references/template-preprocessing.md).
+4. **Ground controls (MCP) — authoritative, every run.** For every detected resource type, call
+   `microsoft_docs_search` to (a) confirm the current MCSB control ID/name/URL and the resource's
+   secure defaults, and (b) **discover newly-shipped security-relevant properties/checks** for
+   that resource type/API version that aren't in the seed tables. Seed tables are a starting
+   point; live Learn overrides them and extends coverage. For resource types **not in the seed
+   tables** (new/AI services), grounding is the *primary* source — do not skip.
+5. **Evaluate each pillar** against the resolved properties (reconcile IDs/URLs via step 4):
+   - [Network Security (NS)](references/mcsb-ns.md)
+   - [Identity Management (IM)](references/mcsb-im.md)
+   - [Privileged Access (PA)](references/mcsb-pa.md) — least-privilege RBAC
+   - [Data Protection (DP)](references/mcsb-dp.md)
+   - [Logging & Threat Detection (LT)](references/mcsb-lt.md)
+   - [AI Workload Security](references/ai-workload-security.md) — Azure OpenAI / AI Services /
+     ML workspaces / AI Search (MITRE ATLAS + OWASP LLM Top 10)
+6. **Assemble findings** in the standard schema — required: `control_id`, `severity`,
+   `description`, `line_number`, `vulnerable_code`, `code_fix`, `priority`, `azure_guidance`.
+   See [output schema](references/output-schema.md).
+7. **(Optional) Attack paths** — when asked, map findings to MITRE ATT&CK techniques (and
+   MITRE ATLAS / OWASP LLM for AI workloads) and chain them into attack scenarios.
+   See [attack paths](references/mitre-attack.md).
+8. **Report** — group findings by severity (Critical → Low), provide the code fix inline for
+   each, and cite the Learn URL. See [remediation patterns](references/remediation-patterns.md).
+
+## Severity & Priority
+
+| Severity | Priority | Guidance |
+|---|---|---|
+| Critical | P0 | Public exposure / no auth — fix before deploy |
+| High | P1 | Weak crypto, missing encryption, broad RBAC |
+| Medium | P2 | Missing logging, short retention |
+| Low | P3 | Hardening / defense-in-depth gaps |
+
+## Error Handling
+
+| Error | Message | Remediation |
+|---|---|---|
+| No template found | "No IaC file provided" | Ask the user for a file path or template content |
+| Non-Azure IaC | "Out of scope: this skill only scans Azure ARM/Bicep/Terraform" | Stop with zero findings; redirect to the cloud's native scanner (e.g. `cfn-lint`/Checkov for AWS) |
+| Unknown template type | "Cannot detect ARM/Bicep/Terraform" | Confirm file type; check for a valid schema/provider |
+| MCP unavailable | "Docs grounding offline" | Proceed using reference-table URLs; note grounding was offline |
+| Unsupported resource | "No seed mapping for `<type>`" | Ground the type live via `microsoft_docs_search`; if still unmapped, report as unscanned and suggest manual review |
+
+## Supported Resource Types
+
+Storage accounts, SQL servers/databases, Key Vault, NSGs, virtual networks, App Service /
+Function Apps, VMs and VMSS, Container Registry, AKS, Service Bus, Cosmos DB, Redis, and
+**AI/ML workloads** (Azure OpenAI, Azure AI Services, Machine Learning workspaces, AI Search).
+For any type without a seed mapping, the skill grounds the current guidance live via Microsoft
+Learn rather than skipping it.

plugin/skills/azure-iac-security/references/ai-workload-security.md

@@ -0,0 +1,56 @@
+# AI Workload Security — Azure AI/ML IaC
+
+Services are increasingly built *with* AI on rapid release cycles, and static rules go stale
+fast. For AI resource types, treat this file as SEED HINTS and **ground live** via
+`microsoft_docs_search` / `bicepschema_get` before emitting a finding — the authoritative
+control/property/default comes from current Microsoft Learn, not this table.
+
+## In-scope AI/ML resource types
+| Resource type | Notes |
+|---|---|
+| `Microsoft.CognitiveServices/accounts` | Azure OpenAI, Azure AI Services / multi-service (`kind` = `OpenAI`, `AIServices`, etc.) |
+| `Microsoft.CognitiveServices/accounts/deployments` | Model deployments; check RAI/content-filter policy |
+| `Microsoft.MachineLearningServices/workspaces` | Azure ML / AI Foundry hub & project |
+| `Microsoft.MachineLearningServices/workspaces/onlineEndpoints` | Inference endpoints |
+| `Microsoft.Search/searchServices` | AI Search (RAG/grounding index over sensitive data) |
+
+## Seed control checks (reconcile against live Learn before emitting)
+| Property / condition | Insecure value | MCSB seed | Why it matters for AI |
+|---|---|---|---|
+| `properties.publicNetworkAccess` | `Enabled` | NS-2 | Model/inference endpoint reachable from internet; use private endpoint |
+| `properties.networkAcls.defaultAction` | `Allow` | NS-2 | No IP/VNet allowlist on the AI data plane |
+| `properties.disableLocalAuth` | `false` | IM-1 | API-key auth instead of Entra ID → key theft = full model access |
+| `properties.encryption` (missing/`Microsoft.CognitiveServices`) | not CMK | DP-5 | Prompts, fine-tune data, embeddings not CMK-encrypted |
+| deployment `raiPolicyName` / content filter | absent | AI-SAFETY | No abuse/jailbreak/harmful-content filtering on the model deployment |
+| ML `hbiWorkspace` | `false` (sensitive data) | AI-SAFETY (confirm live) | High-business-impact flag off for sensitive training data |
+| ML associated storage/keyvault `publicNetworkAccess` | `Enabled` | NS-2 | Training data / secrets exposed via linked resources |
+| AI Search `disableLocalAuth` / `authOptions` apiKey | key auth | IM-1 | Grounding index (often RAG over sensitive corp data) key-accessible |
+| managed identity `identity.type` | absent | IM-3 | Service can't pull models/data via workload identity; falls back to keys |
+
+## AI threat-framework mapping (attack-path / AI-risk mode)
+
+### MITRE ATLAS (adversarial ML — parallels ATT&CK for AI systems)
+| IaC misconfig | ATLAS technique | Tactic |
+|---|---|---|
+| Public/keyed model endpoint (NS-2, IM-1) | AML.T0040 ML Model Inference API Access | ML Model Access |
+| No content/RAI filter on deployment | AML.T0051 LLM Prompt Injection | Initial Access / Execution |
+| Unrestricted inference endpoint | AML.T0024 Exfiltration via ML Inference API | Exfiltration |
+| Public training data / linked storage (NS-2, DP-4) | AML.T0020 Poison Training Data | Resource Development / Persistence |
+| Model artifact readable/writable | AML.T0010 ML Supply Chain Compromise | Initial Access |
+| Keyed/over-permissive endpoint | AML.T0044 Full ML Model Access | ML Model Access |
+
+### OWASP Top 10 for LLM Applications (2025)
+| IaC misconfig | OWASP LLM risk |
+|---|---|
+| No content/RAI filter, no input validation surface | LLM01 Prompt Injection |
+| Overly broad model/data output exposure | LLM02 Sensitive Information Disclosure |
+| Public training data / linked storage writable | LLM04 Data and Model Poisoning |
+| Managed identity absent → keys / broad grants | LLM06 Excessive Agency |
+| CMK/encryption off, model artifact exposed | LLM03 Supply Chain / LLM10 Unbounded Consumption |
+| Public unauthenticated inference endpoint | LLM10 Unbounded Consumption (cost/DoS) |
+
+## Reporting
+Emit AI findings in the same JSON schema. Set `control_id` to the reconciled MCSB control (or
+`AI-SAFETY` for RAI/content-filter gaps). In attack-path/AI-risk mode add `atlas` and
+`owasp_llm` fields alongside `mitre_attack`. Always ground the resource's current secure
+defaults via `microsoft_docs_search` before flagging a missing property.