diff --git a/authors.yaml b/authors.yaml index d101801..bc6dfac 100644 --- a/authors.yaml +++ b/authors.yaml @@ -33,3 +33,9 @@ farzad528: title: "Principal Product Manager" avatar: "https://github.com/farzad528.png?size=200" githubUrl: "https://github.com/farzad528" + +meerakurup: + name: "Meera Kurup" + title: "Product Manager II" + avatar: "https://github.com/meerakurup.png?size=200" + githubUrl: "https://github.com/meerakurup" diff --git a/notebooks/migrate-gpt-4o-mini-to-gpt-5-1.ipynb b/notebooks/migrate-gpt-4o-mini-to-gpt-5-1.ipynb new file mode 100644 index 0000000..e874cab --- /dev/null +++ b/notebooks/migrate-gpt-4o-mini-to-gpt-5-1.ipynb @@ -0,0 +1,592 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "d34c68ce", + "metadata": {}, + "source": [ + "## Overview\n", + "\n", + "Moving a working app from `gpt-4o-mini` to `gpt-5.1` is mostly a *code* change, not a prompt change. `gpt-5.1` is a reasoning model, so the request shape is different: a few Chat Completions parameters were renamed, some sampling knobs are no longer accepted, and there are new controls (`reasoning_effort`, `verbosity`) plus a new token category (`reasoning_tokens`) to account for.\n", + "\n", + "One nuance makes `gpt-5.1` a friendly target for `gpt-4o-mini` users: its `reasoning_effort` defaults to `none`. Left at the default it behaves like a fast, low-latency chat model — close to what you had — and you dial reasoning up only where a task needs it.\n", + "\n", + "This recipe walks the exact code diff. It keeps your existing messages and system prompt untouched and only changes how you *call* the model. Prompt rewriting is deliberately out of scope here — use your prompt-optimization tool for that.\n", + "\n", + "**Who this is for:** Developers with a `gpt-4o-mini` Chat Completions integration who want the smallest reliable code change to run on `gpt-5.1`.\n", + "\n", + "**By the end, you can:**\n", + "- Map every `gpt-4o-mini` request parameter to its `gpt-5.1` equivalent.\n", + "- Replace `max_tokens` with `max_completion_tokens` and drop the parameters `gpt-5.1` rejects.\n", + "- Use `reasoning_effort` and `verbosity` to trade latency for depth, and read `reasoning_tokens` to keep cost accounting correct.\n", + "- Wrap it all in one compatibility shim so you can migrate incrementally.\n", + "\n", + "**Prerequisites:**\n", + "- Azure subscription with access to Azure OpenAI in Microsoft Foundry.\n", + "- A `gpt-5.1` deployment on your resource.\n", + "- Optionally, your existing `gpt-4o-mini` deployment to compare side by side.\n", + "- Your identity has `Foundry User` (or equivalent data-plane access) on the resource.\n", + "- Azure CLI signed in locally (`az login`) if you use `DefaultAzureCredential`.\n", + "- Local environment variables:\n", + " - `AZURE_OPENAI_ENDPOINT`, for example `https://.openai.azure.com`\n", + " - `TARGET_DEPLOYMENT`, for example `gpt-5.1`\n", + " - `SOURCE_DEPLOYMENT` (optional), for example `gpt-4o-mini`\n", + "\n", + "**Time estimate:** ~15 minutes once the `gpt-5.1` deployment exists.\n", + "\n", + "---\n", + "\n", + "## Outline\n", + "\n", + "1. What actually changes in the code.\n", + "2. Setup and a keyless client.\n", + "3. The \"before\": your existing `gpt-4o-mini` call.\n", + "4. Why a copy-paste port fails.\n", + "5. The \"after\": the minimal migrated call.\n", + "6. A compatibility shim for incremental migration.\n", + "7. Account for reasoning tokens in usage.\n", + "8. Optional: adopt the Responses API." + ] + }, + { + "cell_type": "markdown", + "id": "f90814a9", + "metadata": {}, + "source": [ + "## 1. What actually changes in the code\n", + "\n", + "Only the call site changes. Your `messages` list — system prompt included — stays exactly as it is. Here is the full parameter map for a Chat Completions migration:\n", + "\n", + "| `gpt-4o-mini` (Chat Completions) | `gpt-5.1` | What to do |\n", + "|---|---|---|\n", + "| `max_tokens` | `max_completion_tokens` | Rename the key; the value semantics are the same output-token budget. |\n", + "| `temperature` (custom value) | not supported | Remove it. `gpt-5.1` runs at the default temperature. |\n", + "| `top_p` | not supported | Remove it. |\n", + "| `presence_penalty`, `frequency_penalty` | not supported | Remove them. |\n", + "| `logit_bias`, `logprobs`, `top_logprobs` | not supported | Remove them. |\n", + "| *(none)* | `reasoning_effort` | New. `none`, `minimal`, `low`, `medium`, `high`. On `gpt-5.1` the default is `none`, which stays closest to `gpt-4o-mini` (no reasoning tokens); raise it to turn reasoning on. |\n", + "| *(none)* | `verbosity` | New. `low`, `medium`, `high` — controls answer length without editing the prompt. |\n", + "| `usage.completion_tokens` | same, plus `completion_tokens_details.reasoning_tokens` | Hidden reasoning tokens are billed as output tokens — include them in cost math. |\n", + "| `\"role\": \"system\"` | still accepted | System messages work, so you do not have to switch to `developer` messages to migrate. |\n", + "\n", + "The two client-facing shapes, side by side:\n", + "\n", + "```python\n", + "# BEFORE — gpt-4o-mini\n", + "resp = client.chat.completions.create(\n", + " model=\"gpt-4o-mini\",\n", + " messages=messages,\n", + " max_tokens=800,\n", + " temperature=0.2,\n", + " top_p=0.9,\n", + ")\n", + "\n", + "# AFTER — gpt-5.1\n", + "resp = client.chat.completions.create(\n", + " model=\"gpt-5.1\",\n", + " messages=messages, # unchanged\n", + " max_completion_tokens=800, # renamed from max_tokens\n", + " reasoning_effort=\"none\", # default; keeps behavior closest to gpt-4o-mini\n", + " verbosity=\"low\", # new knob (optional)\n", + ")\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "fea93878", + "metadata": {}, + "source": [ + "## 2. Setup and a keyless client\n", + "\n", + "Both models are reached through the same Azure OpenAI v1 endpoint, so the client itself does not change during migration — only the deployment name you pass per call. Install the SDK and configure keyless auth with `DefaultAzureCredential`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e5071c6c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-07T18:38:07.521739Z", + "iopub.status.busy": "2026-07-07T18:38:07.521099Z", + "iopub.status.idle": "2026-07-07T18:38:30.213196Z", + "shell.execute_reply": "2026-07-07T18:38:30.211972Z" + } + }, + "outputs": [], + "source": [ + "%pip install openai azure-identity python-dotenv --quiet" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "87b1f653", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-07T18:38:30.218840Z", + "iopub.status.busy": "2026-07-07T18:38:30.218210Z", + "iopub.status.idle": "2026-07-07T18:39:02.537557Z", + "shell.execute_reply": "2026-07-07T18:39:02.535697Z" + } + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n", + "from dotenv import load_dotenv\n", + "from openai import OpenAI\n", + "\n", + "load_dotenv()\n", + "\n", + "endpoint = os.environ.get(\"AZURE_OPENAI_ENDPOINT\", \"\").rstrip(\"/\")\n", + "source_deployment = os.environ.get(\"SOURCE_DEPLOYMENT\", \"\")\n", + "target_deployment = os.environ.get(\"TARGET_DEPLOYMENT\", \"gpt-5.1\")\n", + "\n", + "if not endpoint:\n", + " raise ValueError(\n", + " \"Set AZURE_OPENAI_ENDPOINT to your resource endpoint, for example https://.openai.azure.com\"\n", + " )\n", + "\n", + "# Azure OpenAI accepts Microsoft Entra bearer tokens for this audience.\n", + "token_provider = get_bearer_token_provider(\n", + " DefaultAzureCredential(), \"https://cognitiveservices.azure.com/.default\"\n", + ")\n", + "\n", + "client = OpenAI(\n", + " base_url=f\"{endpoint}/openai/v1/\",\n", + " api_key=token_provider,\n", + ")\n", + "\n", + "print(f\"Endpoint: {endpoint}\")\n", + "print(f\"Source deployment: {source_deployment}\")\n", + "print(f\"Target deployment: {target_deployment}\")" + ] + }, + { + "cell_type": "markdown", + "id": "98d2301e", + "metadata": {}, + "source": [ + "## 3. The \"before\": your existing `gpt-4o-mini` call\n", + "\n", + "This is a typical `gpt-4o-mini` Chat Completions call. It uses `max_tokens` and custom sampling parameters (`temperature`, `top_p`) — all valid for `gpt-4o-mini`. Notice the `messages` list; it does not change anywhere in this recipe.\n", + "\n", + "The deployment name comes from a constant, not a string literal, so switching models later is a one-line change." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2cd5ebc4", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-07T18:39:02.547713Z", + "iopub.status.busy": "2026-07-07T18:39:02.547185Z", + "iopub.status.idle": "2026-07-07T18:39:29.838229Z", + "shell.execute_reply": "2026-07-07T18:39:29.836603Z" + } + }, + "outputs": [], + "source": [ + "messages = [\n", + " {\"role\": \"system\", \"content\": \"You are a concise assistant for release notes.\"},\n", + " {\"role\": \"user\", \"content\": \"Summarize: we shipped SSO, fixed a billing race condition, and added dark mode.\"},\n", + "]\n", + "\n", + "\n", + "def call_gpt_4o_mini(client, deployment, messages):\n", + " \"\"\"The original call shape written for gpt-4o-mini.\"\"\"\n", + " return client.chat.completions.create(\n", + " model=deployment,\n", + " messages=messages,\n", + " max_tokens=800,\n", + " temperature=0.2,\n", + " top_p=0.9,\n", + " )\n", + "\n", + "\n", + "# Run this only if the source deployment still exists.\n", + "if source_deployment:\n", + " before = call_gpt_4o_mini(client, source_deployment, messages)\n", + " print(before.choices[0].message.content)" + ] + }, + { + "cell_type": "markdown", + "id": "e0ef12a0", + "metadata": {}, + "source": [ + "## 4. Why a copy-paste port fails\n", + "\n", + "**When you hit this:** you change only the `model` argument to `gpt-5.1` and rerun.\n", + "\n", + "Sending the same keyword arguments to `gpt-5.1` returns a `400` because the request carries parameters the model does not accept. The two you will see first:\n", + "\n", + "```text\n", + "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.\n", + "Unsupported value: 'temperature' does not support 0.2 with this model. Only the default (1) value is supported.\n", + "```\n", + "\n", + "So the migration is mechanical: rename `max_tokens`, and remove the sampling parameters `gpt-5.1` rejects. Run the next cell to see that `400` firsthand; then section 5 shows the finished shape." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6c69c53f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-07T18:39:29.844163Z", + "iopub.status.busy": "2026-07-07T18:39:29.843777Z", + "iopub.status.idle": "2026-07-07T18:39:30.120792Z", + "shell.execute_reply": "2026-07-07T18:39:30.118947Z" + }, + "language": "python" + }, + "outputs": [], + "source": [ + "# Demonstrate the failure: send the gpt-4o-mini call shape unchanged to gpt-5.1.\n", + "# call_gpt_4o_mini() is the exact function from section 3 -- only the deployment changes.\n", + "from openai import BadRequestError\n", + "\n", + "try:\n", + " naive = call_gpt_4o_mini(client, target_deployment, messages)\n", + " print(naive.choices[0].message.content)\n", + "except BadRequestError as err:\n", + " print(\"gpt-5.1 rejected the gpt-4o-mini call shape:\")\n", + " print(err)\n" + ] + }, + { + "cell_type": "markdown", + "id": "2a4f1965", + "metadata": {}, + "source": [ + "## 5. The \"after\": the minimal migrated call\n", + "\n", + "**What changed:** `max_tokens` became `max_completion_tokens`; `temperature` and `top_p` are gone; `reasoning_effort` and `verbosity` are added as optional controls. The `messages` list is byte-for-byte identical to the `gpt-4o-mini` version.\n", + "\n", + "**How to adapt:** `gpt-5.1` defaults `reasoning_effort` to `none`. For the closest latency match to `gpt-4o-mini`, leave it at `none` (or omit it) and raise it (`low` → `medium` → `high`) only on tasks that need deeper reasoning. Lower `verbosity` to keep answers short without touching the prompt." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4057a5aa", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-07T18:39:30.125886Z", + "iopub.status.busy": "2026-07-07T18:39:30.125512Z", + "iopub.status.idle": "2026-07-07T18:39:31.246966Z", + "shell.execute_reply": "2026-07-07T18:39:31.245436Z" + } + }, + "outputs": [], + "source": [ + "def call_gpt_5_1(client, deployment, messages):\n", + " \"\"\"The same request, migrated to gpt-5.1.\"\"\"\n", + " return client.chat.completions.create(\n", + " model=deployment,\n", + " messages=messages, # unchanged from gpt-4o-mini\n", + " max_completion_tokens=800, # renamed from max_tokens\n", + " reasoning_effort=\"none\", # none | minimal | low | medium | high (default: none)\n", + " verbosity=\"low\", # low | medium | high\n", + " )\n", + "\n", + "\n", + "after = call_gpt_5_1(client, target_deployment, messages)\n", + "print(after.choices[0].message.content)" + ] + }, + { + "cell_type": "markdown", + "id": "63ce3bfd", + "metadata": {}, + "source": [ + "## 6. A compatibility shim for incremental migration\n", + "\n", + "**When to use:** you cannot flip every call site at once, or you route the same request to `gpt-4o-mini` and `gpt-5.1` behind a flag.\n", + "\n", + "**What it does:** the caller always passes a plain output-token budget plus optional `reasoning_effort` / `verbosity`. The shim translates those into the right keyword arguments for whichever model family it targets, dropping parameters the target rejects.\n", + "\n", + "**How to adapt:** this helper deliberately targets `gpt-5.1`, whose `none` effort and `verbosity` controls it uses. Add an explicit capability entry before routing another model family through it. Keep `reasoning_effort=\"none\"` for a low-latency default and raise it per route." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c5bb676b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-07T18:39:31.252683Z", + "iopub.status.busy": "2026-07-07T18:39:31.251914Z", + "iopub.status.idle": "2026-07-07T18:39:32.627345Z", + "shell.execute_reply": "2026-07-07T18:39:32.625779Z" + } + }, + "outputs": [], + "source": [ + "GPT_5_1_PREFIXES = (\"gpt-5.1\",)\n", + "\n", + "# Sampling parameters that gpt-5.1 rejects.\n", + "UNSUPPORTED_ON_REASONING = {\n", + " \"temperature\",\n", + " \"top_p\",\n", + " \"presence_penalty\",\n", + " \"frequency_penalty\",\n", + " \"logit_bias\",\n", + " \"logprobs\",\n", + " \"top_logprobs\",\n", + "}\n", + "\n", + "\n", + "def is_gpt_5_1(deployment_family: str) -> bool:\n", + " \"\"\"Recognize the only model family this helper supports.\"\"\"\n", + " return deployment_family.lower().startswith(GPT_5_1_PREFIXES)\n", + "\n", + "\n", + "def build_request(\n", + " deployment: str,\n", + " family: str,\n", + " messages: list,\n", + " max_output_tokens: int,\n", + " reasoning_effort: str = \"none\",\n", + " verbosity: str = \"medium\",\n", + " **legacy_params,\n", + ") -> dict:\n", + " \"\"\"Return kwargs for client.chat.completions.create for either model family.\"\"\"\n", + " request = {\"model\": deployment, \"messages\": messages}\n", + "\n", + " if is_gpt_5_1(family):\n", + " request[\"max_completion_tokens\"] = max_output_tokens\n", + " request[\"reasoning_effort\"] = reasoning_effort\n", + " request[\"verbosity\"] = verbosity\n", + " # Silently drop anything gpt-5.1 would reject.\n", + " for key, value in legacy_params.items():\n", + " if key not in UNSUPPORTED_ON_REASONING:\n", + " request[key] = value\n", + " else:\n", + " request[\"max_tokens\"] = max_output_tokens\n", + " request.update(legacy_params)\n", + "\n", + " return request\n", + "\n", + "\n", + "def call_model(client, deployment, family, messages, **kwargs):\n", + " request = build_request(deployment, family, messages, **kwargs)\n", + " return client.chat.completions.create(**request)\n", + "\n", + "\n", + "# Same caller code works for both models — only `family` differs.\n", + "response = call_model(\n", + " client,\n", + " target_deployment,\n", + " family=\"gpt-5.1\",\n", + " messages=messages,\n", + " max_output_tokens=800,\n", + " reasoning_effort=\"none\",\n", + " verbosity=\"low\",\n", + " temperature=0.2, # accepted for gpt-4o-mini, dropped for gpt-5.1\n", + ")\n", + "print(response.choices[0].message.content)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9e27b9a4", + "metadata": {}, + "outputs": [], + "source": [ + "# Credential-free checks for the reusable request translator.\n", + "gpt_5_1_request = build_request(\n", + " deployment=\"target-deployment\",\n", + " family=\"gpt-5.1\",\n", + " messages=messages,\n", + " max_output_tokens=800,\n", + " reasoning_effort=\"minimal\",\n", + " verbosity=\"low\",\n", + " temperature=0.2,\n", + " top_p=0.9,\n", + " )\n", + "assert gpt_5_1_request[\"max_completion_tokens\"] == 800\n", + "assert gpt_5_1_request[\"reasoning_effort\"] == \"minimal\"\n", + "assert gpt_5_1_request[\"verbosity\"] == \"low\"\n", + "assert \"max_tokens\" not in gpt_5_1_request\n", + "assert \"temperature\" not in gpt_5_1_request and \"top_p\" not in gpt_5_1_request\n", + "\n", + "legacy_request = build_request(\n", + " deployment=\"source-deployment\",\n", + " family=\"gpt-4o-mini\",\n", + " messages=messages,\n", + " max_output_tokens=800,\n", + " temperature=0.2,\n", + " )\n", + "assert legacy_request[\"max_tokens\"] == 800\n", + "assert legacy_request[\"temperature\"] == 0.2\n", + "assert \"reasoning_effort\" not in legacy_request\n", + "print(\"Request translation checks passed.\")" + ] + }, + { + "cell_type": "markdown", + "id": "ed211d14", + "metadata": {}, + "source": [ + "## 7. Account for reasoning tokens in usage\n", + "\n", + "When `reasoning_effort` is above `none`, `gpt-5.1` spends hidden **reasoning tokens** before it writes the visible answer. They are not returned in the message content, but they are billed as output tokens and they add latency. If your dashboards or budgets only track `completion_tokens`, they were already counting reasoning tokens — but the new `completion_tokens_details.reasoning_tokens` field lets you separate thinking from output.\n", + "\n", + "Because the migrated calls in this recipe use the default `none`, the breakdown below shows `0` reasoning tokens — the same as `gpt-4o-mini`. Raise `reasoning_effort` and that number climbs; turning it back down (toward `none`) is the lever if reasoning tokens dominate cost." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "983b55f5", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-07T18:39:32.631467Z", + "iopub.status.busy": "2026-07-07T18:39:32.630606Z", + "iopub.status.idle": "2026-07-07T18:39:32.645893Z", + "shell.execute_reply": "2026-07-07T18:39:32.644388Z" + } + }, + "outputs": [], + "source": [ + "usage = after.usage\n", + "details = usage.completion_tokens_details\n", + "reasoning_tokens = getattr(details, \"reasoning_tokens\", 0) or 0\n", + "visible_output_tokens = usage.completion_tokens - reasoning_tokens\n", + "\n", + "print(f\"Prompt tokens: {usage.prompt_tokens}\")\n", + "print(f\"Completion tokens: {usage.completion_tokens}\")\n", + "print(f\" - reasoning tokens: {reasoning_tokens}\")\n", + "print(f\" - visible output: {visible_output_tokens}\")\n", + "print(f\"Total tokens: {usage.total_tokens}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b6592351", + "metadata": {}, + "source": [ + "## 8. Optional: adopt the Responses API\n", + "\n", + "Chat Completions is enough to migrate and keeps the diff tiny. If you also want the forward-looking surface for reasoning models, the Responses API exposes the same controls under slightly different names:\n", + "\n", + "| Chat Completions | Responses API |\n", + "|---|---|\n", + "| `max_completion_tokens` | `max_output_tokens` |\n", + "| `reasoning_effort=\"none\"` | `reasoning={\"effort\": \"none\"}` |\n", + "| `verbosity=\"low\"` | `text={\"verbosity\": \"low\"}` |\n", + "| `messages=[...]` | `input=[...]` |\n", + "\n", + "This is an optional second step, not part of the minimal migration." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a24f119d", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-07T18:39:32.651050Z", + "iopub.status.busy": "2026-07-07T18:39:32.650410Z", + "iopub.status.idle": "2026-07-07T18:39:34.634326Z", + "shell.execute_reply": "2026-07-07T18:39:34.632560Z" + } + }, + "outputs": [], + "source": [ + "responses_result = client.responses.create(\n", + " model=target_deployment,\n", + " input=messages,\n", + " max_output_tokens=800,\n", + " reasoning={\"effort\": \"none\", \"summary\": \"auto\"},\n", + " text={\"verbosity\": \"low\"},\n", + ")\n", + "print(responses_result.output_text)" + ] + }, + { + "cell_type": "markdown", + "id": "9a52eb6e", + "metadata": {}, + "source": [ + "## Takeaways and next steps\n", + "\n", + "**What you changed in code:**\n", + "- `max_tokens` → `max_completion_tokens` (Chat Completions) or `max_output_tokens` (Responses).\n", + "- Removed `temperature`, `top_p`, and the penalty / logprob / logit_bias parameters.\n", + "- Added `reasoning_effort` (default `none` on `gpt-5.1`) and `verbosity` as the new control surface.\n", + "- Started reading `completion_tokens_details.reasoning_tokens` for accurate accounting.\n", + "- Left every prompt and `messages` list untouched.\n", + "\n", + "**Reusable shim to drop into your project:**\n", + "\n", + "```python\n", + "GPT_5_1_PREFIXES = (\"gpt-5.1\",)\n", + "UNSUPPORTED_ON_GPT_5_1 = {\n", + " \"temperature\", \"top_p\", \"presence_penalty\", \"frequency_penalty\",\n", + " \"logit_bias\", \"logprobs\", \"top_logprobs\",\n", + "}\n", + "\n", + "def build_request(deployment, family, messages, max_output_tokens,\n", + " reasoning_effort=\"none\", verbosity=\"medium\", **legacy):\n", + " req = {\"model\": deployment, \"messages\": messages}\n", + " if family.lower().startswith(GPT_5_1_PREFIXES):\n", + " req[\"max_completion_tokens\"] = max_output_tokens\n", + " req[\"reasoning_effort\"] = reasoning_effort\n", + " req[\"verbosity\"] = verbosity\n", + " req.update({key: value for key, value in legacy.items()\n", + " if key not in UNSUPPORTED_ON_GPT_5_1})\n", + " else:\n", + " req[\"max_tokens\"] = max_output_tokens\n", + " req.update(legacy)\n", + " return req\n", + "```\n", + "\n", + "**Common failure modes:**\n", + "\n", + "| Symptom | Likely cause | Fix |\n", + "|---|---|---|\n", + "| `400 Unsupported parameter: 'max_tokens'` | Left the old key in the request | Rename to `max_completion_tokens`. |\n", + "| `400 ... 'temperature' does not support 0.2` | Sent a custom sampling value | Remove `temperature` / `top_p` / penalties. |\n", + "| `400 ... 'reasoning_effort' ...` | Used an effort value unsupported by the deployed model | On `gpt-5.1`, use `none`, `minimal`, `low`, `medium`, or `high`. |\n", + "| Answers feel shallower than expected | `reasoning_effort` left at the `none` default | Raise it to `low`/`medium`/`high` for that route. |\n", + "| Cost per call jumped | Reasoning tokens counted as output | Inspect `reasoning_tokens`; lower `reasoning_effort`. |\n", + "| `401` from the client | Wrong token audience or no sign-in | Use `https://cognitiveservices.azure.com/.default` and run `az login`. |\n", + "\n", + "**Next steps:**\n", + "- Sweep `reasoning_effort` and `verbosity` on your own evals to find the cost/quality point for each route.\n", + "- Once behavior is verified, run your prompt-optimization pass to tune the (still unchanged) prompts for the new model.\n", + "- Consider migrating the highest-traffic path to the Responses API for reasoning summaries and streaming." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv (3.13.14.final.0)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.14" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/notebooks/migrate-gpt-5-5-to-claude-opus-4-8.ipynb b/notebooks/migrate-gpt-5-5-to-claude-opus-4-8.ipynb new file mode 100644 index 0000000..b35f213 --- /dev/null +++ b/notebooks/migrate-gpt-5-5-to-claude-opus-4-8.ipynb @@ -0,0 +1,1358 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cc464ac2", + "metadata": {}, + "source": [ + "> **Migrate a Chat Completions workload from `gpt-5.5` to `claude-opus-4-8` on Microsoft Foundry, behind one adapter, with eval gates.**\n", + "\n", + "Anthropic's Claude models are now hosted on Azure as first-class Foundry deployments, billed through Azure Marketplace in Claude Consumption Units. For teams running gpt-5.x today, the engineering move is small (one adapter, two SDKs, one extra Entra audience), but the request shape, the response shape, and the gating contract are all different from a same-family upgrade.\n", + "\n", + "This recipe walks the move end-to-end on a **single Foundry resource** that hosts both `gpt-5.5` and `claude-opus-4-8`. Both are first-class Foundry deployments: one served via the OpenAI Chat Completions path, the other via the Anthropic Messages path. You build the adapter, prove parity on a small eval set, and decide whether to flip.\n", + "\n", + "**The rollout pattern this notebook teaches: test side by side first, then switch users over.** Instead of swapping the model in one risky step, you do it in two phases:\n", + "\n", + "1. **Test side by side (users don't see Claude yet).** Your app still serves every user request from `gpt-5.5`. For some fraction of those same requests, you also send the prompt to `claude-opus-4-8` in the background and log the result. The user never sees the Claude response; it's compared against the gpt-5.5 response and scored offline against your eval gates. Production behavior is unchanged; you're just collecting evidence on real traffic.\n", + "2. **Switch users over.** Once the side-by-side data shows Claude is meeting your gates, you flip a single environment variable (`PROVIDER=anthropic`) so user-facing responses now come from `claude-opus-4-8`. The gpt-5.5 deployment stays alive so you can roll back instantly if anything regresses.\n", + "\n", + "The adapter you build below makes both phases the same code path. The only difference between them is which provider's output is shown to the user.\n", + "\n", + "**What this recipe is not.** It doesn't change your business prompts (defer per-provider prompt optimization to a separate pass), and it doesn't pick a router. The aim is to give you the code-level changes and eval gates needed to adopt `claude-opus-4-8` safely, not to argue for one model over another.\n", + "\n", + "**How to run.** Run cells top to bottom on a fresh kernel; later cells depend on classes defined earlier. This notebook itself calls both adapters explicitly so you can compare them side by side. The `PROVIDER` env var described in the rollout pattern is what *your application* reads in production to pick which adapter handles real user traffic." + ] + }, + { + "cell_type": "markdown", + "id": "2486f0b3", + "metadata": {}, + "source": [ + "## Configure\n", + "\n", + "**Prerequisites.** One Foundry resource in the same Azure subscription with both deployments below created and the Anthropic Marketplace agreement accepted for your tenant. If you don't have that yet, the [Microsoft Learn quickstart](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-claude?tabs=python) and the [`Azure-Samples/claude`](https://github.com/Azure-Samples/claude) `azd up` starter (linked again at the end) walk you through it.\n", + "\n", + "You'll need one Foundry resource with two deployments connected to its project:\n", + "\n", + "- `gpt-5.5`: OpenAI Chat Completions on Foundry (the baseline)\n", + "- `claude-opus-4-8`: Anthropic on Foundry (the candidate)\n", + "\n", + "Both deployments must be in a region where the model is available and have enough quota for a small eval set. If your gpt-5.5 deployment is PTU-only and you have no PAYG opus quota yet, the eval cell will rate-limit on the Anthropic side, so request quota from your Foundry portal before continuing.\n", + "\n", + "A single Foundry role can get you through this notebook, `Foundry Owner`. Or you can use two Foundry roles with separated control and data plane permissions: `Foundry User` on the Foundry resource for inference (data plane), and the corresponding `Foundry Account Owner` role for deployment changes (control plane).\n", + "\n", + "Put these in a `.env` next to the notebook (or in your shell). Substitute your actual resource and deployment names:\n", + "\n", + "```bash\n", + "# Baseline: gpt-5.5 (OpenAI on Foundry, Chat Completions path)\n", + "OPENAI_ENDPOINT=https://.services.ai.azure.com\n", + "OPENAI_DEPLOYMENT=gpt-5.5\n", + "OPENAI_API_KEY= # or omit and use DefaultAzureCredential\n", + "OPENAI_API_VERSION=2024-10-21\n", + "\n", + "# Candidate: claude-opus-4-8 (Anthropic on Foundry, Messages path)\n", + "ANTH_BASE_URL=https://.services.ai.azure.com/anthropic\n", + "ANTH_DEPLOYMENT=claude-opus-4-8\n", + "ANTH_API_KEY= # or omit and use DefaultAzureCredential\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "52f394ec", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:43:18.682050Z", + "iopub.status.busy": "2026-06-30T21:43:18.679790Z", + "iopub.status.idle": "2026-06-30T21:43:38.327766Z", + "shell.execute_reply": "2026-06-30T21:43:38.325810Z" + } + }, + "outputs": [], + "source": [ + "%%capture\n", + "%pip install --quiet \\\n", + " \"openai>=1.50.0\" \\\n", + " \"anthropic>=0.55.0\" \\\n", + " \"azure-identity>=1.19.0\" \\\n", + " \"httpx>=0.27.0\" \\\n", + " \"jsonschema>=4.22.0\" \\\n", + " \"python-dotenv>=1.0.1\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6c010c50", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:43:38.337598Z", + "iopub.status.busy": "2026-06-30T21:43:38.336797Z", + "iopub.status.idle": "2026-06-30T21:43:38.469336Z", + "shell.execute_reply": "2026-06-30T21:43:38.467813Z" + } + }, + "outputs": [], + "source": [ + "import os\n", + "from urllib.parse import urlparse\n", + "from dotenv import load_dotenv\n", + "\n", + "load_dotenv(override=True)\n", + "\n", + "OPENAI_ENDPOINT = os.environ[\"OPENAI_ENDPOINT\"].rstrip(\"/\")\n", + "OPENAI_DEPLOYMENT = os.environ.get(\"OPENAI_DEPLOYMENT\", \"gpt-5.5\")\n", + "OPENAI_API_KEY = os.environ.get(\"OPENAI_API_KEY\") # None → use Entra\n", + "OPENAI_API_VERSION = os.environ.get(\"OPENAI_API_VERSION\", \"2024-10-21\")\n", + "\n", + "ANTH_BASE_URL = os.environ[\"ANTH_BASE_URL\"].rstrip(\"/\")\n", + "ANTH_DEPLOYMENT = os.environ.get(\"ANTH_DEPLOYMENT\", \"claude-opus-4-8\")\n", + "ANTH_API_KEY = os.environ.get(\"ANTH_API_KEY\") # None → use Entra\n", + "\n", + "def _redact_ai_url(url: str) -> str:\n", + " parsed = urlparse(url)\n", + " host = parsed.netloc.lower()\n", + " if host.endswith(\".services.ai.azure.com\"):\n", + " path = \"/anthropic\" if parsed.path.rstrip(\"/\") == \"/anthropic\" else \"\"\n", + " return f\"{parsed.scheme}://.services.ai.azure.com{path}\"\n", + " return \"\"\n", + "\n", + "print(f\"OpenAI endpoint : {_redact_ai_url(OPENAI_ENDPOINT)}\")\n", + "print(f\"OpenAI deployment : {OPENAI_DEPLOYMENT} (api-version {OPENAI_API_VERSION})\")\n", + "print(f\"Anthropic base : {_redact_ai_url(ANTH_BASE_URL)}\")\n", + "print(f\"Anthropic model : {ANTH_DEPLOYMENT}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "baea26d3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:43:38.473611Z", + "iopub.status.busy": "2026-06-30T21:43:38.473049Z", + "iopub.status.idle": "2026-06-30T21:44:34.979722Z", + "shell.execute_reply": "2026-06-30T21:44:34.977949Z" + } + }, + "outputs": [], + "source": [ + "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n", + "from openai import AzureOpenAI\n", + "from anthropic import AnthropicFoundry\n", + "\n", + "# Two audiences, one credential. Build separate token providers so the right scope\n", + "# travels with the right client.\n", + "_cred = DefaultAzureCredential()\n", + "openai_token_provider = get_bearer_token_provider(\n", + " _cred, \"https://cognitiveservices.azure.com/.default\"\n", + ")\n", + "anth_token_provider = get_bearer_token_provider(\n", + " _cred, \"https://ai.azure.com/.default\"\n", + ")\n", + "\n", + "# Baseline client: OpenAI on Foundry (Chat Completions path).\n", + "openai_client = AzureOpenAI(\n", + " azure_endpoint=OPENAI_ENDPOINT,\n", + " api_version=OPENAI_API_VERSION,\n", + " **({\"api_key\": OPENAI_API_KEY} if OPENAI_API_KEY is not None else {\"azure_ad_token_provider\": openai_token_provider}),\n", + ")\n", + "\n", + "# Candidate client: Anthropic on Foundry. base_url ends in /anthropic;\n", + "# the SDK appends /v1/messages itself.\n", + "anth_client = AnthropicFoundry(\n", + " base_url=ANTH_BASE_URL,\n", + " **({\"api_key\": ANTH_API_KEY} if ANTH_API_KEY\n", + " else {\"azure_ad_token_provider\": anth_token_provider}),\n", + ")\n", + "\n", + "print(\"clients built. openai:\", type(openai_client).__name__,\n", + " \"| anthropic:\", type(anth_client).__name__)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a65ff371", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:44:34.984863Z", + "iopub.status.busy": "2026-06-30T21:44:34.984030Z", + "iopub.status.idle": "2026-06-30T21:44:38.211886Z", + "shell.execute_reply": "2026-06-30T21:44:38.209841Z" + } + }, + "outputs": [], + "source": [ + "import httpx\n", + "from urllib.parse import urlparse\n", + "\n", + "# Reachability probe only. The Foundry host root doesn't serve a meaningful endpoint\n", + "# for HEAD, so a 404 or 405 here is normal; it confirms DNS, TLS, and routing work.\n", + "# We're not exercising the OpenAI or Anthropic API path; that happens in later cells.\n", + "# Any 2xx/3xx/4xx response = host is reachable. Only ERR (timeout / DNS) is a real failure.\n", + "def _host(url: str) -> str:\n", + " p = urlparse(url)\n", + " return f\"{p.scheme}://{p.netloc}\"\n", + "\n", + "for label, url in [(\"OpenAI\", OPENAI_ENDPOINT),\n", + " (\"Anthropic-on-Foundry\", _host(ANTH_BASE_URL))]:\n", + " display_url = _redact_ai_url(url)\n", + " try:\n", + " r = httpx.head(url, timeout=5.0)\n", + " print(f\" {label:<22} {display_url:<60} {r.status_code}\")\n", + " except Exception as exc:\n", + " print(f\" {label:<22} {display_url:<60} ERR {type(exc).__name__}: {exc}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d61eae1b", + "metadata": {}, + "source": [ + "## The call you make today\n", + "\n", + "This is the gpt-5.5 call you almost certainly have in production. The shape is the same regardless of which framework wraps it: a `messages[]` array (with `system` as the first message), `tools=[{type:\"function\",...}]`, `response_format={\"type\":\"json_schema\",\"strict\":true,...}`, and `reasoning_effort` + `max_completion_tokens` on the 5.x line.\n", + "\n", + "`model=` is the *deployment* name on Azure, not the model id.\n", + "\n", + "> **On the Foundry Responses API?** If your gpt-5.5 calls already go through `client.responses.create(...)` against a project-scoped endpoint, only this cell's request shape swaps; the Anthropic side of the notebook and the adapter contract are unchanged. The four differences worth knowing:\n", + ">\n", + "> 1. **`input=`** (a string or a list of typed input items) replaces `messages=`. `system` becomes `instructions=` at the top level.\n", + "> 2. **`max_output_tokens`** replaces `max_completion_tokens`.\n", + "> 3. **Tools are flat**: `{type:\"function\", name, description, parameters}` instead of `{type:\"function\", function:{name, ...}}`. The JSON Schema itself is unchanged.\n", + "> 4. **Reading the response**: `resp.output_text` is the convenience accessor; structured outputs land in `resp.output[*]` items rather than `choices[0].message`.\n", + ">\n", + "> In the adapter built later, this maps to a second `OpenAIResponsesAdapter` that swaps these four call-shape details and returns the same `Response` dataclass. Every other section (tools, structured outputs, streaming, eval, gates) continues to work as written." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "34e04ef3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:44:38.218348Z", + "iopub.status.busy": "2026-06-30T21:44:38.217578Z", + "iopub.status.idle": "2026-06-30T21:44:56.848780Z", + "shell.execute_reply": "2026-06-30T21:44:56.844608Z" + } + }, + "outputs": [], + "source": [ + "baseline_resp = openai_client.chat.completions.create(\n", + " model=OPENAI_DEPLOYMENT,\n", + " messages=[\n", + " {\"role\": \"system\", \"content\": \"You are a careful research assistant.\"},\n", + " {\"role\": \"user\", \"content\": \"In one sentence: what does Microsoft Foundry do?\"},\n", + " ],\n", + " max_completion_tokens=200,\n", + ")\n", + "\n", + "print(\"=== gpt-5.5 (OpenAI on Foundry) ===\")\n", + "print(baseline_resp.choices[0].message.content)\n", + "print()\n", + "print(\"usage:\", baseline_resp.usage)" + ] + }, + { + "cell_type": "markdown", + "id": "ed41261d", + "metadata": {}, + "source": [ + "## The call you make on Anthropic\n", + "\n", + "Same prompt, same Foundry resource, different path. The host is `.services.ai.azure.com`; the SDK takes a `base_url` ending in `/anthropic` and appends `/v1/messages` itself. Three shape changes worth noting up front:\n", + "\n", + "1. **`system` is a top-level argument**, not a message inside the `messages[]` array.\n", + "2. **Content is blocks**: a user turn's `content` may be a plain string *or* a list of typed blocks (`{type:\"text\",...}`, `{type:\"image\",...}`, `{type:\"tool_result\",...}`).\n", + "3. **`max_tokens` (not `max_completion_tokens`).** Anthropic also uses `stop_sequences` (not `stop`).\n", + "\n", + "This first call is intentionally bare on both sides (no reasoning knobs, just provider defaults) so you can see the shape difference without other variables moving. We turn on reasoning where the eval (later in the notebook) shows it actually helps.\n", + "\n", + "### Reasoning effort: where the comparison isn't apples-to-apples\n", + "\n", + "This is the most common source of misleading A/B numbers when migrating, and it's worth being precise about:\n", + "\n", + "| | **gpt-5.5** | **claude-opus-4-8** |\n", + "|---|---|---|\n", + "| Knob | `reasoning_effort` | `thinking={\"type\":\"adaptive\"}` + `output_config.effort` |\n", + "| Levels | `minimal`, `low`, `medium`, `high` | off (default) and `low`, `medium`, `high` (confirm any higher tiers against your model version's [Foundry docs](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/claude-models?tabs=pay-go)) |\n", + "| Default | `medium`; **always reasons** internally | **off**; no extended thinking unless you opt in |\n", + "| Output | Reasoning is internal; you never see the tokens | Surfaces as separate `thinking` content blocks (don't show users) |\n", + "\n", + "The honest read: **provider defaults are not equivalent.** gpt-5.5 at its default does silent internal reasoning; claude-opus-4-8 at its default does not. So \"default vs default\" measures *what production code typically does*, not equal reasoning budget.\n", + "\n", + "Three sensible comparison modes, pick the one that matches what you're evaluating:\n", + "\n", + "- **Production parity (recommended for migration eval).** Mirror whatever your live gpt-5.5 call sets, and on the Claude side opt into thinking only where the eval shows a quality lift. This is the version users actually experience.\n", + "- **Reasoning-budget parity.** gpt-5.5 default (`medium`) vs Claude with `thinking={\"type\":\"adaptive\"}` and `output_config={\"effort\":\"medium\"}`. Useful for capability comparison, but not 1:1; the providers spend the budget differently.\n", + "- **Latency / cost floor.** gpt-5.5 with `reasoning_effort=\"minimal\"` vs Claude with no `thinking`. Useful when you need to know the fastest, cheapest each can serve at acceptable quality.\n", + "\n", + "For the hello-world baseline below, defaults are the right call (that's what most readers' production code looks like), but don't read parity into the numbers. The eval cells that follow are where you'd pin reasoning effort to a fixed setting." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dfd3da50", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:44:56.854866Z", + "iopub.status.busy": "2026-06-30T21:44:56.853777Z", + "iopub.status.idle": "2026-06-30T21:45:01.830002Z", + "shell.execute_reply": "2026-06-30T21:45:01.827536Z" + } + }, + "outputs": [], + "source": [ + "candidate_resp = anth_client.messages.create(\n", + " model=ANTH_DEPLOYMENT,\n", + " system=\"You are a careful research assistant.\",\n", + " messages=[\n", + " {\"role\": \"user\", \"content\": \"In one sentence: what does Microsoft Foundry do?\"},\n", + " ],\n", + " max_tokens=200,\n", + ")\n", + "\n", + "# Anthropic returns content as a list of typed blocks. The assistant's text lives in\n", + "# `text` blocks; any `thinking` blocks (when extended thinking is enabled) should not\n", + "# be rendered to end users.\n", + "text_blocks = [b.text for b in candidate_resp.content if b.type == \"text\"]\n", + "print(\"=== claude-opus-4-8 (Anthropic on Foundry) ===\")\n", + "print(\"\\n\".join(text_blocks))\n", + "print()\n", + "print(\"usage:\", candidate_resp.usage)" + ] + }, + { + "cell_type": "markdown", + "id": "890051c3", + "metadata": {}, + "source": [ + "## What needs to change in your code\n", + "\n", + "When you adopt `claude-opus-4-8` alongside `gpt-5.5` on the same Foundry resource, these are the places your code touches the wire. Review each one and decide where the change lands (typically inside the adapter you build in the next section). Anything not listed here is unchanged.\n", + "\n", + "**Endpoint & auth**\n", + "- Add a second base URL: `https://.services.ai.azure.com/anthropic`. The SDK appends `/v1/messages` itself.\n", + "- For API-key calls, let `AnthropicFoundry` set its supported authentication header rather than constructing one yourself.\n", + "- Entra-ID calls use the audience `https://ai.azure.com/.default`, which is a different audience from the OpenAI path (`https://cognitiveservices.azure.com/.default`). Build a separate token provider per audience.\n", + "- No `?api-version=` query string; version is set by the `anthropic-version: 2023-06-01` header (the SDK handles this).\n", + "\n", + "**SDK & client class**\n", + "- Install `anthropic>=0.55` and import `from anthropic import AnthropicFoundry` (not the stock `Anthropic` class).\n", + "- Continue using `from openai import AzureOpenAI` for the gpt-5.5 path.\n", + "\n", + "**Request shape**\n", + "- `model=` still takes the **deployment** name (same convention as today).\n", + "- Move the system prompt out of `messages[]` into the top-level `system=` argument.\n", + "- Rename the token cap: `max_completion_tokens` becomes `max_tokens`.\n", + "- Rename stop sequences: `stop` becomes `stop_sequences`.\n", + "- User content can be a plain string *or* a list of typed blocks (`{type:\"text\"}`, `{type:\"image\"}`, `{type:\"tool_result\"}`). Image blocks differ from OpenAI's `image_url` shape.\n", + "\n", + "**Reasoning controls**\n", + "- Opt in per call with `thinking={\"type\":\"adaptive\"}` and steer with `output_config={\"effort\":\"low|medium|high\"}` (confirm whether higher tiers are available for your model version against the [Foundry Claude docs](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/claude-models?tabs=pay-go)). There's no equivalent of `reasoning_effort` as a single argument.\n", + "\n", + "**Tools**\n", + "- Tool declaration changes wrapper: `{type:\"function\",\"function\":{name, description, parameters}}` becomes `{name, description, input_schema}`. The JSON Schema itself passes through.\n", + "- Tool **results** are not `role:\"tool\"` messages; they're user messages whose content is `[{type:\"tool_result\", tool_use_id, content}]`.\n", + "\n", + "**Structured outputs**\n", + "- There is no `strict: true` JSON-schema mode. Replace it with **validate post-hoc and retry once** (full pattern in the structured outputs section).\n", + "\n", + "**Streaming**\n", + "- Event shape is different: typed events with `delta.type` set to `text_delta`, `input_json_delta`, or `thinking_delta`. Wrap both providers behind a single normalized event stream so downstream UI code doesn't need to know.\n", + "\n", + "**Billing**\n", + "- Claude usage rolls up under **Azure Marketplace** as Claude Consumption Units, a separate report from your existing OpenAI-on-Foundry consumption. Plan to reconcile both during migration.\n", + "\n", + "The next section turns this list into ~80 lines of adapter code." + ] + }, + { + "cell_type": "markdown", + "id": "8081e509", + "metadata": {}, + "source": [ + "## The adapter\n", + "\n", + "**When to use:** Use this when the app already has OpenAI-style chat call sites and you want to test Claude without rewriting business logic.\n", + "\n", + "**What it does:** It creates one provider-neutral contract for prompts, tools, structured outputs, usage, and streaming so only the adapter knows which SDK is underneath.\n", + "\n", + "**How to adapt:** Keep `Conversation` and `Response` stable, then add or swap provider-specific adapters behind the same `run()` method.\n", + "\n", + "Don't sprinkle `if PROVIDER == \"anthropic\"` through your app. Build one boundary, put both SDKs behind it, and let every call site speak the same shape.\n", + "\n", + "The contract:\n", + "\n", + "- A `Conversation` dataclass holds `system`, `messages[]`, `tools[]`, and the optional thinking/effort knobs. Tools use the OpenAI strict shape as the canonical form (most apps already have it that way).\n", + "- `OpenAIAdapter.run(conv, deployment)` and `AnthropicAdapter.run(conv, deployment)` both return a provider-neutral `Response(text, tool_calls, usage, raw)`.\n", + "- A tiny `openai_tools_to_anthropic()` helper translates schemas one way; tool results get wrapped by the adapter on the way back.\n", + "\n", + "Everything below the `Conversation` boundary is the adapter's problem. Above it, your app does not change." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5c02929e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:45:01.838043Z", + "iopub.status.busy": "2026-06-30T21:45:01.837388Z", + "iopub.status.idle": "2026-06-30T21:45:04.628958Z", + "shell.execute_reply": "2026-06-30T21:45:04.627573Z" + } + }, + "outputs": [], + "source": [ + "import json\n", + "from dataclasses import dataclass, field\n", + "from typing import Any, Optional\n", + "from jsonschema import Draft202012Validator, ValidationError\n", + "\n", + "# ---------- Canonical types ----------\n", + "\n", + "@dataclass\n", + "class Message:\n", + " role: str # \"system\" | \"user\" | \"assistant\" | \"tool\"\n", + " content: Any # str or list of typed blocks\n", + " tool_call_id: Optional[str] = None # only when role == \"tool\"\n", + "\n", + "@dataclass\n", + "class ToolSpec:\n", + " name: str\n", + " description: str\n", + " parameters: dict # JSON Schema (OpenAI strict shape)\n", + "\n", + "@dataclass\n", + "class ToolCall:\n", + " id: str\n", + " name: str\n", + " arguments: dict # parsed JSON\n", + "\n", + "@dataclass\n", + "class Conversation:\n", + " system: str\n", + " messages: list[Message] = field(default_factory=list)\n", + " tools: list[ToolSpec] = field(default_factory=list)\n", + " thinking: bool = False # turn on adaptive thinking (Anthropic only)\n", + " effort: str = \"medium\" # low | medium | high (confirm any higher tiers in the Foundry Claude docs for your model version)\n", + " max_tokens: int = 1024\n", + " json_schema: Optional[dict] = None # when set, output is validated against this schema\n", + "\n", + "@dataclass\n", + "class Response:\n", + " text: str\n", + " tool_calls: list[ToolCall]\n", + " usage: dict\n", + " raw: Any\n", + " retries: int = 0 # JSON-schema retry count (0 for OpenAI strict; 0 or 1 for Anthropic)\n", + "\n", + "# ---------- Helpers ----------\n", + "\n", + "def openai_tools_to_anthropic(tools: list[ToolSpec]) -> list[dict]:\n", + " \"\"\"OpenAI function tools -> Anthropic tool blocks. The JSON Schema passes through;\n", + " only the wrapper shape changes.\"\"\"\n", + " return [\n", + " {\"name\": t.name, \"description\": t.description, \"input_schema\": t.parameters}\n", + " for t in tools\n", + " ]\n", + "\n", + "def _strip_json_fences(raw: str) -> str:\n", + " \"\"\"Remove a ```json ... ``` wrapper if the model insisted on one.\"\"\"\n", + " raw = raw.strip()\n", + " if raw.startswith(\"```\"):\n", + " raw = raw.removeprefix(\"```json\").removeprefix(\"```\").removesuffix(\"```\").strip()\n", + " return raw\n", + "\n", + "def _anthropic_content(content: Any) -> Any:\n", + " \"\"\"Convert SDK content blocks to plain dictionaries when a prior response is reused.\"\"\"\n", + " if not isinstance(content, list):\n", + " return content\n", + " return [block.model_dump() if hasattr(block, \"model_dump\") else block for block in content]\n", + "\n", + "def anthropic_messages(messages: list[Message]) -> list[dict]:\n", + " \"\"\"Translate canonical history, including tool results, to Anthropic message blocks.\"\"\"\n", + " translated = []\n", + " for message in messages:\n", + " if message.role in (\"user\", \"assistant\"): \n", + " translated.append({\"role\": message.role,\n", + " \"content\": _anthropic_content(message.content)})\n", + " elif message.role == \"tool\":\n", + " if not message.tool_call_id:\n", + " raise ValueError(\"tool messages require tool_call_id\")\n", + " translated.append({\n", + " \"role\": \"user\",\n", + " \"content\": [{\n", + " \"type\": \"tool_result\",\n", + " \"tool_use_id\": message.tool_call_id,\n", + " \"content\": _anthropic_content(message.content),\n", + " }],\n", + " })\n", + " else:\n", + " raise ValueError(f\"Unsupported canonical role: {message.role}\")\n", + " return translated\n", + "\n", + "def tool_calls_meet_contract(tools: list[ToolSpec], expected_name: str, calls: list,\n", + " error: Optional[str] = None) -> bool:\n", + " \"\"\"Fail closed unless every tool call has the expected name and valid arguments.\"\"\"\n", + " tools_by_name = {tool.name: tool for tool in tools}\n", + " if error or not calls:\n", + " return False\n", + " for name, arguments in calls:\n", + " if name != expected_name or name not in tools_by_name:\n", + " return False\n", + " try:\n", + " Draft202012Validator(tools_by_name[name].parameters).validate(arguments)\n", + " except (ValidationError, TypeError):\n", + " return False\n", + " return True\n", + "\n", + "# ---------- Adapters ----------\n", + "\n", + "class OpenAIAdapter:\n", + " def __init__(self, client): self.client = client\n", + "\n", + " def run(self, conv: Conversation, deployment: str) -> Response:\n", + " msgs = [{\"role\": \"system\", \"content\": conv.system}]\n", + " for m in conv.messages:\n", + " entry = {\"role\": m.role, \"content\": m.content}\n", + " if m.tool_call_id is not None:\n", + " entry[\"tool_call_id\"] = m.tool_call_id\n", + " msgs.append(entry)\n", + " kwargs: dict = {\"model\": deployment, \"messages\": msgs,\n", + " \"max_completion_tokens\": conv.max_tokens}\n", + " if conv.tools:\n", + " kwargs[\"tools\"] = [\n", + " {\"type\": \"function\",\n", + " \"function\": {\"name\": t.name, \"description\": t.description,\n", + " \"parameters\": t.parameters}}\n", + " for t in conv.tools\n", + " ]\n", + " if conv.json_schema is not None:\n", + " # strict:true is the hard guarantee; retries are always 0 on this path.\n", + " kwargs[\"response_format\"] = {\n", + " \"type\": \"json_schema\",\n", + " \"json_schema\": {\"name\": \"output\", \"strict\": True, \"schema\": conv.json_schema},\n", + " }\n", + " resp = self.client.chat.completions.create(**kwargs)\n", + " msg = resp.choices[0].message\n", + " calls = [\n", + " ToolCall(id=tc.id, name=tc.function.name,\n", + " arguments=json.loads(tc.function.arguments or \"{}\"))\n", + " for tc in (msg.tool_calls or [])\n", + " ]\n", + " return Response(\n", + " text=msg.content or \"\",\n", + " tool_calls=calls,\n", + " usage={\"input\": resp.usage.prompt_tokens,\n", + " \"output\": resp.usage.completion_tokens},\n", + " raw=resp,\n", + " retries=0,\n", + " )\n", + "\n", + "\n", + "class AnthropicAdapter:\n", + " def __init__(self, client): self.client = client\n", + "\n", + " def run(self, conv: Conversation, deployment: str) -> Response:\n", + " if conv.json_schema is not None:\n", + " return self._run_with_json_schema(conv, deployment)\n", + "\n", + " msgs = anthropic_messages(conv.messages)\n", + " kwargs: dict = {\"model\": deployment, \"system\": conv.system,\n", + " \"messages\": msgs, \"max_tokens\": conv.max_tokens}\n", + " if conv.tools:\n", + " kwargs[\"tools\"] = openai_tools_to_anthropic(conv.tools)\n", + " if conv.thinking:\n", + " kwargs[\"thinking\"] = {\"type\": \"adaptive\"}\n", + " kwargs[\"output_config\"] = {\"effort\": conv.effort}\n", + " resp = self.client.messages.create(**kwargs)\n", + " return self._build_response(resp, retries=0)\n", + "\n", + " def _run_with_json_schema(self, conv: Conversation, deployment: str) -> Response:\n", + " \"\"\"Validate-and-retry once. Anthropic has no strict JSON-schema mode, so we\n", + " instruct, parse, validate, and retry exactly once on failure.\"\"\"\n", + " schema_text = json.dumps(conv.json_schema, indent=2)\n", + " msgs = anthropic_messages(conv.messages)\n", + " # Augment the final user turn with the schema. Preserves the caller's prompt.\n", + " if msgs and msgs[-1][\"role\"] == \"user\" and isinstance(msgs[-1][\"content\"], str):\n", + " msgs[-1] = {\"role\": \"user\",\n", + " \"content\": f\"{msgs[-1]['content']}\\n\\n\"\n", + " f\"Return JSON only matching this schema:\\n{schema_text}\"}\n", + " system = conv.system + \"\\n\\nReturn JSON only. No prose. No markdown fences.\"\n", + "\n", + " last_exc: Exception = RuntimeError(\"no attempts ran\")\n", + " for attempt in range(2):\n", + " resp = self.client.messages.create(\n", + " model=deployment, system=system, messages=msgs,\n", + " max_tokens=conv.max_tokens,\n", + " )\n", + " raw = \"\".join(b.text for b in resp.content if b.type == \"text\")\n", + " raw = _strip_json_fences(raw)\n", + " try:\n", + " parsed = json.loads(raw)\n", + " Draft202012Validator(conv.json_schema).validate(parsed)\n", + " return Response(\n", + " text=json.dumps(parsed),\n", + " tool_calls=[],\n", + " usage=self._usage(resp),\n", + " raw=resp,\n", + " retries=attempt,\n", + " )\n", + " except (json.JSONDecodeError, ValidationError) as exc:\n", + " last_exc = exc\n", + " msgs.append({\"role\": \"assistant\", \"content\": raw})\n", + " msgs.append({\"role\": \"user\",\n", + " \"content\": f\"That did not match the schema ({exc}). \"\n", + " \"Re-emit valid JSON only.\"})\n", + " raise RuntimeError(f\"Anthropic structured output failed twice: {last_exc}\")\n", + "\n", + " def _build_response(self, resp, retries: int) -> Response:\n", + " text = \"\".join(b.text for b in resp.content if b.type == \"text\")\n", + " calls = [\n", + " ToolCall(id=b.id, name=b.name, arguments=b.input or {})\n", + " for b in resp.content if b.type == \"tool_use\"\n", + " ]\n", + " return Response(text=text, tool_calls=calls,\n", + " usage=self._usage(resp), raw=resp, retries=retries)\n", + "\n", + " @staticmethod\n", + " def _usage(resp) -> dict:\n", + " u = resp.usage\n", + " return {\n", + " \"input\": u.input_tokens,\n", + " \"output\": u.output_tokens,\n", + " \"cache_read\": getattr(u, \"cache_read_input_tokens\", 0),\n", + " \"cache_create\": getattr(u, \"cache_creation_input_tokens\", 0),\n", + " }\n", + "\n", + "openai_adapter = OpenAIAdapter(openai_client)\n", + "anth_adapter = AnthropicAdapter(anth_client)\n", + "print(\"adapters ready\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "94e58478", + "metadata": {}, + "outputs": [], + "source": [ + "# Credential-free assertion: retain the assistant tool_use block and translate the\n", + " # canonical tool result into the Anthropic user-message shape.\n", + "history = [\n", + " Message(role=\"user\", content=\"What is the weather in Seattle?\"),\n", + " Message(\n", + " role=\"assistant\",\n", + " content=[{\n", + " \"type\": \"tool_use\",\n", + " \"id\": \"call_weather\",\n", + " \"name\": \"get_weather\",\n", + " \"input\": {\"city\": \"Seattle\", \"units\": \"f\"},\n", + " }],\n", + " ),\n", + " Message(\n", + " role=\"tool\",\n", + " tool_call_id=\"call_weather\",\n", + " content=\"{\\\"temperature\\\": 62, \\\"units\\\": \\\"f\\\"}\",\n", + " ),\n", + "]\n", + "translated = anthropic_messages(history)\n", + "assert translated[1][\"role\"] == \"assistant\"\n", + "assert translated[1][\"content\"][0][\"type\"] == \"tool_use\"\n", + "assert translated[2] == {\n", + " \"role\": \"user\",\n", + " \"content\": [{\n", + " \"type\": \"tool_result\",\n", + " \"tool_use_id\": \"call_weather\",\n", + " \"content\": \"{\\\"temperature\\\": 62, \\\"units\\\": \\\"f\\\"}\",\n", + " }],\n", + "}\n", + "weather_schema = ToolSpec(\n", + " name=\"get_weather\",\n", + " description=\"Get weather.\",\n", + " parameters={\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\"}},\n", + " \"required\": [\"city\"], \"additionalProperties\": False},\n", + ")\n", + "assert tool_calls_meet_contract([weather_schema], \"get_weather\",\n", + " [(\"get_weather\", {\"city\": \"Seattle\"})])\n", + "assert not tool_calls_meet_contract([weather_schema], \"get_weather\",\n", + " [(\"get_weather\", {\"city\": 42})])\n", + "assert not tool_calls_meet_contract([weather_schema], \"get_weather\", [],\n", + " error=\"JSONDecodeError: malformed arguments\")\n", + "print(\"Anthropic translation and tool-contract checks passed.\")" + ] + }, + { + "cell_type": "markdown", + "id": "69b65a23", + "metadata": {}, + "source": [ + "## Tools end-to-end\n", + "\n", + "Declare a tool once in the canonical (OpenAI strict) shape. Both adapters pick it up; both responses parse into the same `ToolCall`.\n", + "\n", + "This is the **schema-parity contract** the promotion gate later in this notebook checks. If the Anthropic path doesn't produce a `ToolCall` with the same `name` and a JSON body that validates against the same `parameters` schema, the migration fails closed *regardless* of how good the prose is.\n", + "\n", + "> **Heads up: empty `text` is expected here.** When the model decides to call a tool, it returns a `tool_call` instead of prose, so `text` will be `''` on the gpt-5.5 row and `tool_calls` will be populated. That's the model saying \"don't make up the weather, call `get_weather` and feed me the result.\" In a real agent loop you'd run the tool, append a `tool` message with the result, and call the model again; that second response is where the user-facing text comes from. Claude sometimes emits a short narrating text block alongside the `tool_use` block, so the Claude row may show both. That's a known provider difference, not a bug, and the promotion gate ignores it.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "caf6841d", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:45:04.634834Z", + "iopub.status.busy": "2026-06-30T21:45:04.634455Z", + "iopub.status.idle": "2026-06-30T21:45:08.707186Z", + "shell.execute_reply": "2026-06-30T21:45:08.702969Z" + } + }, + "outputs": [], + "source": [ + "GET_WEATHER = ToolSpec(\n", + " name=\"get_weather\",\n", + " description=\"Get the current weather for a city.\",\n", + " parameters={\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"city\": {\"type\": \"string\", \"description\": \"City name, e.g. 'Seattle'\"},\n", + " \"units\": {\"type\": \"string\", \"enum\": [\"c\", \"f\"]},\n", + " },\n", + " \"required\": [\"city\"],\n", + " \"additionalProperties\": False,\n", + " },\n", + ")\n", + "\n", + "conv = Conversation(\n", + " system=\"You can call tools when you need real-world data.\",\n", + " messages=[Message(role=\"user\", content=\"What's the weather in Seattle in Fahrenheit?\")],\n", + " tools=[GET_WEATHER],\n", + ")\n", + "\n", + "for label, adapter, deployment in [\n", + " (\"gpt-5.5\", openai_adapter, OPENAI_DEPLOYMENT),\n", + " (\"claude-opus-4-8\", anth_adapter, ANTH_DEPLOYMENT),\n", + "]:\n", + " r = adapter.run(conv, deployment)\n", + " print(f\"--- {label} ---\")\n", + " print(f\" text : {r.text[:80]!r}\")\n", + " print(f\" tool_calls : {[(c.name, c.arguments) for c in r.tool_calls]}\")\n", + " print()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e18ba332", + "metadata": {}, + "outputs": [], + "source": [ + "# Complete the Claude tool round trip. Preserve the assistant's tool_use block, then\n", + " # append the canonical tool result; AnthropicAdapter turns it into a user tool_result block.\n", + "first_turn = anth_adapter.run(conv, ANTH_DEPLOYMENT)\n", + "if not first_turn.tool_calls:\n", + " raise RuntimeError(\"Claude did not request get_weather; retry with a more explicit tool instruction.\")\n", + "weather_call = first_turn.tool_calls[0]\n", + "continuation = Conversation(\n", + " system=conv.system,\n", + " messages=[\n", + " *conv.messages,\n", + " Message(role=\"assistant\", content=_anthropic_content(first_turn.raw.content)),\n", + " Message(\n", + " role=\"tool\",\n", + " tool_call_id=weather_call.id,\n", + " content=json.dumps({\"city\": \"Seattle\", \"temperature\": 62, \"units\": \"f\"}),\n", + " ),\n", + " ],\n", + " tools=[GET_WEATHER],\n", + ")\n", + "completed_turn = anth_adapter.run(continuation, ANTH_DEPLOYMENT)\n", + "print(completed_turn.text)" + ] + }, + { + "cell_type": "markdown", + "id": "1c9b6d18", + "metadata": {}, + "source": [ + "## Structured outputs\n", + "\n", + "**When to use:** Use this when your OpenAI path depends on `strict: true` JSON Schema outputs and the Claude path must satisfy the same downstream parser.\n", + "\n", + "**What it does:** OpenAI enforces the schema at generation time; Claude is validated after generation and gets one retry if the response does not match.\n", + "\n", + "**How to adapt:** Replace `TICKET_SCHEMA` with your production schema and track `Response.retries` as a portability signal in evals.\n", + "\n", + "OpenAI's `strict: true` `json_schema` is a *hard* guarantee: the model cannot emit invalid JSON. Anthropic has no equivalent. The portable pattern is **validate post-hoc, retry once** with a \"your previous response did not match this schema, re-emit valid JSON only\" follow-up.\n", + "\n", + "The adapter handles this for you. Set `Conversation.json_schema` and call `adapter.run(conv)`. `OpenAIAdapter` sets `response_format` strict mode (zero retries by construction), `AnthropicAdapter` instructs, parses, validates, and retries once on failure. The retry count is returned on `Response.retries`. Call-site code looks the same on both providers.\n", + "\n", + "**Retry rate is itself a signal.** If `Response.retries` is `1` on more than ~5% of your eval set, the prompt likely needs an Anthropic-specific compile. That's a separate pass (per-provider prompt portability). Don't paper over it here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fa5b5765", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:45:08.712075Z", + "iopub.status.busy": "2026-06-30T21:45:08.711712Z", + "iopub.status.idle": "2026-06-30T21:45:12.843058Z", + "shell.execute_reply": "2026-06-30T21:45:12.840856Z" + } + }, + "outputs": [], + "source": [ + "TICKET_SCHEMA = {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"category\": {\"type\": \"string\",\n", + " \"enum\": [\"hardware\", \"software\", \"billing\", \"other\"]},\n", + " \"priority\": {\"type\": \"string\", \"enum\": [\"low\", \"med\", \"high\"]},\n", + " },\n", + " \"required\": [\"category\", \"priority\"],\n", + " \"additionalProperties\": False,\n", + "}\n", + "\n", + "# Same Conversation -> both adapters. No side-channel helpers, no provider branching\n", + "# in user code. The adapter handles strict-mode (OpenAI) vs. validate-and-retry (Anthropic).\n", + "conv = Conversation(\n", + " system=\"Classify support tickets.\",\n", + " messages=[Message(role=\"user\",\n", + " content=\"Classify this support ticket: 'My laptop won't charge.'\")],\n", + " json_schema=TICKET_SCHEMA,\n", + ")\n", + "\n", + "r_oai = openai_adapter.run(conv, OPENAI_DEPLOYMENT)\n", + "r_anth = anth_adapter.run(conv, ANTH_DEPLOYMENT)\n", + "\n", + "print(f\"gpt-5.5 -> {r_oai.text} (retries: {r_oai.retries})\")\n", + "print(f\"claude-opus-4-8 -> {r_anth.text} (retries: {r_anth.retries})\")" + ] + }, + { + "cell_type": "markdown", + "id": "24b6196f", + "metadata": {}, + "source": [ + "## Streaming, normalized\n", + "\n", + "**When to use:** Use this when your UI, logs, or agent loop already consumes streaming events and should not care which provider generated them.\n", + "\n", + "**What it does:** It maps OpenAI chunks and Anthropic typed events into one `StreamEvent` shape for text, tool calls, thinking, and completion.\n", + "\n", + "**How to adapt:** Add only the event kinds your app consumes, and keep provider-only fields inside the normalizer.\n", + "\n", + "Both SDKs stream, but the events look different. OpenAI sends SSE chunks with `choices[0].delta.content`; Anthropic sends typed events whose `delta.type` is `text_delta`, `input_json_delta`, or `thinking_delta`. Wrap both in a single `StreamEvent` interface and downstream UI code stops caring which provider is on." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d63b154c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:45:12.877951Z", + "iopub.status.busy": "2026-06-30T21:45:12.862311Z", + "iopub.status.idle": "2026-06-30T21:45:19.450793Z", + "shell.execute_reply": "2026-06-30T21:45:19.445564Z" + } + }, + "outputs": [], + "source": [ + "from typing import Iterator, Literal\n", + "\n", + "@dataclass\n", + "class StreamEvent:\n", + " kind: Literal[\"text\", \"tool_call_start\", \"tool_call_args\", \"thinking\", \"done\"]\n", + " data: dict\n", + "\n", + "def normalize_openai_stream(resp) -> Iterator[StreamEvent]:\n", + " for chunk in resp:\n", + " if not chunk.choices:\n", + " continue\n", + " delta = chunk.choices[0].delta\n", + " if delta.content:\n", + " yield StreamEvent(\"text\", {\"text\": delta.content})\n", + " for tc in (delta.tool_calls or []):\n", + " if tc.function and tc.function.name:\n", + " yield StreamEvent(\"tool_call_start\",\n", + " {\"id\": tc.id, \"name\": tc.function.name})\n", + " if tc.function and tc.function.arguments:\n", + " yield StreamEvent(\"tool_call_args\", {\"args\": tc.function.arguments})\n", + " yield StreamEvent(\"done\", {})\n", + "\n", + "def normalize_anthropic_stream(resp) -> Iterator[StreamEvent]:\n", + " for event in resp:\n", + " t = getattr(event, \"type\", None)\n", + " if t == \"content_block_start\":\n", + " block = event.content_block\n", + " if block.type == \"tool_use\":\n", + " yield StreamEvent(\"tool_call_start\",\n", + " {\"id\": block.id, \"name\": block.name})\n", + " elif t == \"content_block_delta\":\n", + " d = event.delta\n", + " if d.type == \"text_delta\":\n", + " yield StreamEvent(\"text\", {\"text\": d.text})\n", + " elif d.type == \"input_json_delta\":\n", + " yield StreamEvent(\"tool_call_args\", {\"args\": d.partial_json})\n", + " elif d.type == \"thinking_delta\":\n", + " yield StreamEvent(\"thinking\", {\"text\": d.thinking})\n", + " yield StreamEvent(\"done\", {})\n", + "\n", + "# Same prompt AND same system message on both sides. What we're showing is the\n", + "# difference in event shape, so anything that changes output length (like only one\n", + "# side getting a \"be concise\" instruction) would muddy the comparison.\n", + "SYSTEM = \"Be concise.\"\n", + "prompt = \"List three things to do in Seattle in summer.\"\n", + "\n", + "print(\"--- gpt-5.5 stream ---\")\n", + "oai_stream = openai_client.chat.completions.create(\n", + " model=OPENAI_DEPLOYMENT,\n", + " messages=[{\"role\": \"system\", \"content\": SYSTEM},\n", + " {\"role\": \"user\", \"content\": prompt}],\n", + " max_completion_tokens=200, stream=True,\n", + ")\n", + "for i, ev in enumerate(normalize_openai_stream(oai_stream)):\n", + " if i >= 25: break\n", + " print(f\" {ev.kind:<16} {str(ev.data)[:60]}\")\n", + "\n", + "print(\"\\n--- claude-opus-4-8 stream ---\")\n", + "anth_stream = anth_client.messages.create(\n", + " model=ANTH_DEPLOYMENT, system=SYSTEM,\n", + " messages=[{\"role\": \"user\", \"content\": prompt}],\n", + " max_tokens=200, stream=True,\n", + ")\n", + "for i, ev in enumerate(normalize_anthropic_stream(anth_stream)):\n", + " if i >= 25: break\n", + " print(f\" {ev.kind:<16} {str(ev.data)[:60]}\")" + ] + }, + { + "cell_type": "markdown", + "id": "418a08b0", + "metadata": {}, + "source": [ + "## Run the eval set\n", + "\n", + "An inline 8-row eval set (mix of plain text, tool-use, and JSON shape) so this notebook runs in under a minute. For production, **create your own eval file** with 50–100 prompts that look like your real traffic and load it in place of the inline list. `azure-ai-evaluation`'s `SimilarityEvaluator` / `RelevanceEvaluator` plug straight into the same loop. The harness, gates, and code path don't change; only the data does.\n", + "\n", + "Scoring is three layers, in this order of strictness:\n", + "\n", + "1. **Schema parity** (hard): every row that produced a tool call on gpt-5.5 must produce a `ToolCall` with the same `name` and a JSON body that validates against the same schema on claude-opus-4-8. Argument *values* may differ; structure must not.\n", + "2. **Structured-output retry rate** (hard): JSON-mode rows must succeed in ≤1 retry on the Anthropic path. >10% retry rate is a smell, not a soft fail.\n", + "3. **Quality** (soft): Jaccard token overlap vs `expected` for plain-text rows. Swap in `SimilarityEvaluator` for production. Gate: candidate ≥ baseline − 0.05.\n", + "\n", + "Plus a latency check in the promotion gate below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "58f96af9", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:45:19.456707Z", + "iopub.status.busy": "2026-06-30T21:45:19.456315Z", + "iopub.status.idle": "2026-06-30T21:46:25.151644Z", + "shell.execute_reply": "2026-06-30T21:46:25.149324Z" + } + }, + "outputs": [], + "source": [ + "import time, re\n", + "\n", + "EVAL_ROWS = [\n", + " {\"id\": \"text-1\",\n", + " \"prompt\": \"In one sentence, what does Microsoft Foundry do?\",\n", + " \"expected\": \"Microsoft Foundry is a platform for building, deploying, and managing AI agents and models on Azure.\"},\n", + " {\"id\": \"text-2\",\n", + " \"prompt\": \"Name two benefits of multi-provider model access.\",\n", + " \"expected\": \"Avoiding vendor lock-in and matching the best model to each task.\"},\n", + " {\"id\": \"text-3\",\n", + " \"prompt\": \"What is prompt caching used for?\",\n", + " \"expected\": \"Reusing long static prompt prefixes to cut input cost and latency.\"},\n", + " {\"id\": \"tool-1\",\n", + " \"prompt\": \"What's the weather in Seattle?\",\n", + " \"tools\": [GET_WEATHER],\n", + " \"expected_tool\": \"get_weather\"},\n", + " {\"id\": \"tool-2\",\n", + " \"prompt\": \"Tell me the weather in Tokyo in Celsius.\",\n", + " \"tools\": [GET_WEATHER],\n", + " \"expected_tool\": \"get_weather\"},\n", + " {\"id\": \"json-1\",\n", + " \"prompt\": \"Classify this support ticket: 'My laptop won't charge.'\",\n", + " \"json_schema\": TICKET_SCHEMA},\n", + " {\"id\": \"json-2\",\n", + " \"prompt\": \"Classify this support ticket: 'I was double-billed last month.'\",\n", + " \"json_schema\": TICKET_SCHEMA},\n", + " {\"id\": \"text-4\",\n", + " \"prompt\": \"What does the Claude `tool_result` content block do?\",\n", + " \"expected\": \"Returns the result of a tool call back to the model inside a user message.\"},\n", + "]\n", + "\n", + "def _tokens(s: str) -> set[str]:\n", + " return set(re.findall(r\"[a-z0-9]+\", (s or \"\").lower()))\n", + "\n", + "def jaccard(a: str, b: str) -> float:\n", + " A, B = _tokens(a), _tokens(b)\n", + " return len(A & B) / max(1, len(A | B))\n", + "\n", + "def run_row(adapter, deployment, row) -> dict:\n", + " \"\"\"Every row builds a Conversation and calls adapter.run(). No provider branching.\"\"\"\n", + " t0 = time.perf_counter()\n", + " if \"tools\" in row:\n", + " conv = Conversation(\n", + " system=\"You can call tools when you need real-world data.\",\n", + " messages=[Message(role=\"user\", content=row[\"prompt\"])],\n", + " tools=row[\"tools\"],\n", + " )\n", + " try:\n", + " r = adapter.run(conv, deployment)\n", + " out = {\"tool_calls\": [(c.name, c.arguments) for c in r.tool_calls],\n", + " \"usage\": r.usage}\n", + " except Exception as exc:\n", + " # A malformed tool-call payload is a failed contract, not a crashed gate.\n", + " out = {\"tool_calls\": [], \"usage\": {},\n", + " \"error\": f\"{type(exc).__name__}: {exc}\"}\n", + " elif \"json_schema\" in row:\n", + " conv = Conversation(\n", + " system=\"Be concise and accurate.\",\n", + " messages=[Message(role=\"user\", content=row[\"prompt\"])],\n", + " json_schema=row[\"json_schema\"],\n", + " )\n", + " r = adapter.run(conv, deployment)\n", + " out = {\"json\": json.loads(r.text), \"retries\": r.retries, \"usage\": r.usage}\n", + " else:\n", + " conv = Conversation(\n", + " system=\"Be concise and accurate.\",\n", + " messages=[Message(role=\"user\", content=row[\"prompt\"])],\n", + " )\n", + " r = adapter.run(conv, deployment)\n", + " out = {\"text\": r.text, \"usage\": r.usage}\n", + " out[\"latency_s\"] = round(time.perf_counter() - t0, 2)\n", + " return out\n", + "\n", + "results = {\"openai\": {}, \"anthropic\": {}}\n", + "for row in EVAL_ROWS:\n", + " results[\"openai\"][row[\"id\"]] = run_row(openai_adapter, OPENAI_DEPLOYMENT, row)\n", + " results[\"anthropic\"][row[\"id\"]] = run_row(anth_adapter, ANTH_DEPLOYMENT, row)\n", + "\n", + "print(f\"{'id':<8} {'kind':<6} {'gpt-5.5 lat':<13} {'opus-4-8 lat':<14} preview (candidate)\")\n", + "print(\"-\" * 80)\n", + "for row in EVAL_ROWS:\n", + " o = results[\"openai\"][row[\"id\"]]\n", + " a = results[\"anthropic\"][row[\"id\"]]\n", + " kind = \"tool\" if \"tools\" in row else \"json\" if \"json_schema\" in row else \"text\"\n", + " preview = a.get(\"text\") or a.get(\"json\") or a.get(\"tool_calls\", \"\")\n", + " print(f\"{row['id']:<8} {kind:<6} {o['latency_s']:<13} {a['latency_s']:<14} {str(preview)[:55]}\")" + ] + }, + { + "cell_type": "markdown", + "id": "70259349", + "metadata": {}, + "source": [ + "## Promotion gate, cost, and latency\n", + "\n", + "**When to use:** Use this before any user-facing traffic moves from `gpt-5.5` to `claude-opus-4-8`.\n", + "\n", + "**What it does:** It blocks promotion on parsed-shape regressions first, then checks structured-output retries, quality, and latency.\n", + "\n", + "**How to adapt:** Keep schema/tool parity as hard gates, but replace the toy quality metric and latency threshold with workload-specific evals before production rollout.\n", + "\n", + "The cross-provider gate is stricter than a same-family upgrade. *Parsed-shape parity* fails closed before quality is even considered.\n", + "\n", + "| Gate | Threshold | Why |\n", + "|---|---|---|\n", + "| Schema parity | 100% on tool rows | If 3% of tool calls parse wrong, a 10% quality lift is worthless |\n", + "| Structured retry rate | ≤10% | Higher = prompt needs an Anthropic-specific compile, not a model swap |\n", + "| Quality (Jaccard) | candidate ≥ baseline − 0.05 | Same as a within-family upgrade |\n", + "| P90 latency | ≤ baseline × 1.3 | Budget headroom for the candidate path |\n", + "\n", + "**Cost is intentionally not auto-computed here.** Claude is metered in **Claude Consumption Units** through Azure Marketplace, on a separate page from your OpenAI consumption on Foundry. Fetch your current Foundry-issued rates and plug them in. Run both reports side by side during migration so total spend stays visible." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "be7bff09", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:46:25.157238Z", + "iopub.status.busy": "2026-06-30T21:46:25.156575Z", + "iopub.status.idle": "2026-06-30T21:46:25.194020Z", + "shell.execute_reply": "2026-06-30T21:46:25.193041Z" + } + }, + "outputs": [], + "source": [ + "import statistics\n", + "\n", + "# 1. Schema parity (hard): both providers must emit the expected tool with\n", + "# arguments that validate against its declared JSON Schema.\n", + "tool_rows = [r for r in EVAL_ROWS if \"tools\" in r]\n", + "schema_parity_hits = []\n", + "\n", + "for row in tool_rows:\n", + " o_names = {n for n, _ in results[\"openai\"][row[\"id\"]][\"tool_calls\"]}\n", + " a_names = {n for n, _ in results[\"anthropic\"][row[\"id\"]][\"tool_calls\"]}\n", + " openai_result = results[\"openai\"][row[\"id\"]]\n", + " anthropic_result = results[\"anthropic\"][row[\"id\"]]\n", + " openai_contract = tool_calls_meet_contract(\n", + " row[\"tools\"], row[\"expected_tool\"], openai_result.get(\"tool_calls\", []),\n", + " openai_result.get(\"error\"),\n", + " )\n", + " anthropic_contract = tool_calls_meet_contract(\n", + " row[\"tools\"], row[\"expected_tool\"], anthropic_result.get(\"tool_calls\", []),\n", + " anthropic_result.get(\"error\"),\n", + " )\n", + " schema_parity_hits.append(openai_contract and anthropic_contract and o_names == a_names)\n", + "schema_parity_rate = (sum(schema_parity_hits) / len(schema_parity_hits)\n", + " if schema_parity_hits else 1.0)\n", + "\n", + "# 2. Structured-output retry rate (hard).\n", + "json_rows = [r for r in EVAL_ROWS if \"json_schema\" in r]\n", + "retry_rate = (\n", + " sum(results[\"anthropic\"][r[\"id\"]].get(\"retries\", 0) for r in json_rows)\n", + " / max(1, len(json_rows))\n", + ")\n", + "\n", + "# 3. Quality on plain text (soft): Jaccard token overlap vs `expected`.\n", + "text_rows = [r for r in EVAL_ROWS if \"expected\" in r]\n", + "baseline_q = statistics.mean(\n", + " jaccard(results[\"openai\"][r[\"id\"]][\"text\"], r[\"expected\"]) for r in text_rows\n", + ")\n", + "candidate_q = statistics.mean(\n", + " jaccard(results[\"anthropic\"][r[\"id\"]][\"text\"], r[\"expected\"]) for r in text_rows\n", + ")\n", + "\n", + "# 4. Latency P90 across all rows.\n", + "def p90(xs): return sorted(xs)[max(0, int(0.9 * len(xs)) - 1)]\n", + "base_p90 = p90([results[\"openai\"][r[\"id\"]][\"latency_s\"] for r in EVAL_ROWS])\n", + "cand_p90 = p90([results[\"anthropic\"][r[\"id\"]][\"latency_s\"] for r in EVAL_ROWS])\n", + "\n", + "gates = [\n", + " (\"Schema parity (tool rows)\", schema_parity_rate == 1.0,\n", + " f\"{schema_parity_rate*100:.0f}% match\"),\n", + " (\"Structured retry rate (cap 10%)\", retry_rate <= 0.10,\n", + " f\"{retry_rate*100:.0f}% retries\"),\n", + " (\"Quality (>= baseline - 0.05)\", candidate_q >= baseline_q - 0.05,\n", + " f\"baseline {baseline_q:.2f} | candidate {candidate_q:.2f}\"),\n", + " (\"P90 latency (<= 1.3x baseline)\", cand_p90 <= base_p90 * 1.3,\n", + " f\"baseline {base_p90:.2f}s | candidate {cand_p90:.2f}s\"),\n", + "]\n", + "\n", + "print(\"=\" * 72)\n", + "print(f\" {'GATE':<36} {'STATUS':<8} DETAIL\")\n", + "print(\"-\" * 72)\n", + "for name, passed, detail in gates:\n", + " print(f\" {name:<36} {'PASS' if passed else 'FAIL':<8} {detail}\")\n", + "print(\"-\" * 72)\n", + "overall = all(p for _, p, _ in gates)\n", + "print(f\" OVERALL: {'PASS. Safe to start the side-by-side test on real traffic.' if overall else 'FAIL. Do not promote.'}\")\n", + "print(\"=\" * 72)" + ] + }, + { + "cell_type": "markdown", + "id": "3ffe5fcd", + "metadata": {}, + "source": [ + "## How to read the gate output\n", + "\n", + "A FAIL on this 8-row demo set is *usually* one of three things: small-sample noise, prompt overhead the adapter adds, or a genuine model-behavior difference. Each gate has a typical failure mode and a specific triage step. Run the diagnostic cell below first for any latency FAIL, then use this table for the rest.\n", + "\n", + "| Gate that failed | Most common cause on N=8 | What to do |\n", + "|---|---|---|\n", + "| **Schema parity** | Claude returned a prose hedge instead of calling the tool (e.g. \"I don't have access to live weather…\") | Real provider-behavior difference. Tighten the user prompt with an explicit \"use the tool\" instruction, or accept and add a 1-retry budget on tool rows in production. |\n", + "| **Structured retry rate** | One JSON row came back with a stray code fence or extra prose | Inspect `results[\"anthropic\"][\"json-X\"][\"retries\"]`. >0 retries on >5% of rows means the prompt needs an Anthropic-specific compile (separate pass). |\n", + "| **Quality (Jaccard)** | Only 4 text rows; one different word choice swings the mean by 0.05+ | Re-run the eval cell. If quality flips between PASS and FAIL on three back-to-back runs, it's noise; move to a 50–100 row set and swap in `azure-ai-evaluation.SimilarityEvaluator`. If the gap is consistent, it's real. |\n", + "| **P90 latency** | N=8 means P90 = the 7th-slowest row. One cold start or the heavier JSON prompt dominates. | Run the diagnostic cell below. If P50 is within 1.5× but P90 is over 1.3×, it's tail noise. If P50 is also high, it's a real workload-shape issue (see \"Making a real promotion decision\" below). |\n", + "\n", + "### Why this notebook trips its own gates more than you'd expect\n", + "\n", + "Two structural reasons, both worth understanding before you change the thresholds:\n", + "\n", + "**1. N=8 is below the noise floor of these metrics.** P90 of 8 samples is \"the 7th-slowest row\"; a single slow response can flip it. Jaccard averaged over 4 rows shifts by 0.05+ when Claude picks \"platform\" vs \"service.\" This is a teaching set, not a production set. The fix is more rows, not looser thresholds.\n", + "\n", + "**2. The Anthropic JSON path is genuinely heavier than the OpenAI one.** Because Anthropic has no `strict:true` mode, the adapter appends the JSON schema text to the user message and adds a \"return JSON only\" suffix to the system prompt. That's typically **+150–400 input tokens per JSON row** vs the OpenAI `response_format` path. With 2 of 8 rows being JSON, the extra tokens drag P90 on the Claude side. In production you'd pin the schema text behind `cache_control` so it's effectively free after the first call.\n", + "\n", + "The diagnostic cell below prints **per-row latency for both providers** plus **P50 vs P90 side by side**. Use it to separate the two situations above before deciding whether the FAIL is noise or signal." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f192c044", + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-30T21:46:25.195619Z", + "iopub.status.busy": "2026-06-30T21:46:25.195440Z", + "iopub.status.idle": "2026-06-30T21:46:25.201270Z", + "shell.execute_reply": "2026-06-30T21:46:25.200421Z" + } + }, + "outputs": [], + "source": [ + "# Latency diagnostic: if the P90 gate failed, find which row dominates.\n", + "# Shows per-row latency for both providers plus the P50/P90 spread.\n", + "\n", + "print(f\" {'id':<8} {'kind':<6} {'gpt-5.5':<10} {'opus-4-8':<10} ratio\")\n", + "print(\"-\" * 50)\n", + "for row in EVAL_ROWS:\n", + " rid = row[\"id\"]\n", + " o = results[\"openai\"][rid][\"latency_s\"]\n", + " a = results[\"anthropic\"][rid][\"latency_s\"]\n", + " kind = \"tool\" if \"tools\" in row else \"json\" if \"json_schema\" in row else \"text\"\n", + " ratio = a / o if o > 0 else float(\"inf\")\n", + " print(f\" {rid:<8} {kind:<6} {o:<10.2f} {a:<10.2f} {ratio:.1f}x\")\n", + "\n", + "# Compare medians to means: if P50 is fine but P90 fails, the tail is the problem.\n", + "import statistics\n", + "oai_lat = [results[\"openai\"][r[\"id\"]][\"latency_s\"] for r in EVAL_ROWS]\n", + "anth_lat = [results[\"anthropic\"][r[\"id\"]][\"latency_s\"] for r in EVAL_ROWS]\n", + "print()\n", + "print(f\" P50 gpt-5.5 {statistics.median(oai_lat):.2f}s opus-4-8 {statistics.median(anth_lat):.2f}s\")\n", + "print(f\" P90 gpt-5.5 {base_p90:.2f}s opus-4-8 {cand_p90:.2f}s\")\n", + "print(f\" Ratio P50 = {statistics.median(anth_lat)/statistics.median(oai_lat):.1f}x \"\n", + " f\"Ratio P90 = {cand_p90/base_p90:.1f}x\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "14356407", + "metadata": {}, + "source": [ + "## Making a real promotion decision\n", + "\n", + "If this notebook says **PASS** on a clean run, that means `claude-opus-4-8` cleared every gate on **8 toy prompts**. It does **not** yet mean it's a good fit for your workload. To turn this notebook into a real decision, do these four things in order:\n", + "\n", + "**1. Run a bigger eval set.** Eight rows is not enough for a stable P90; one slow row dominates the metric. Replace `EVAL_ROWS` in the eval-set cell above with **50–100 prompts** that look like your production traffic. The harness, gates, and code path stay the same; only the data changes. For larger sets, create your own eval file (e.g. an `eval.jsonl` next to the notebook) and load it in place of the inline list.\n", + "\n", + "**2. Use prompts that match your actual workload.** If you run a chatbot doing one-line replies, an 8-row long-context legal-analysis eval will mislead you. If you run an agent loop with heavy tool use, gating on plain-text Jaccard won't tell you what you need to know. Bring your real prompts to this harness.\n", + "\n", + "**3. If opus still misses the latency gate, decide consciously, don't auto-fail.** `claude-opus-4-8` is a heavier model than `gpt-5.5`; on short prompts it tends to be noticeably slower per call without extended thinking. Measure your own workload; the gap varies materially by prompt shape, region, and load. Whether that's a blocker depends on workload shape:\n", + "\n", + "| Workload | What usually happens | Path forward |\n", + "|---|---|---|\n", + "| Short prompts, short answers | opus loses on latency | Relax the gate (e.g. `<= 2.0×`), keep gpt-5.5 for this path, or accept the trade-off |\n", + "| Long context (10K+ input tokens) | gap closes; often within 1.2× | Pass as-is |\n", + "| Heavy reasoning where gpt-5.5 already runs `reasoning_effort=high` | roughly a wash | Pass as-is |\n", + "| Tool-heavy agent loops | comparable end-to-end; opus often faster | Pass as-is |\n", + "| Streaming chat UX | time-to-first-token is close; only total time differs | Gate on TTFT, not total |\n", + "\n", + "**4. Test streaming separately when UX is what's actually on the line.** For any user-facing chat, perceived latency is **time-to-first-token**, not total response time. Use the streaming cell earlier in this notebook to measure TTFT directly; the streaming gap between providers is much smaller than the total-time gap.\n", + "\n", + "**Bottom line:** a green gate on these 8 demo rows is permission to start a real side-by-side test with your own prompts. It is not permission to flip `PROVIDER=anthropic` in production." + ] + }, + { + "cell_type": "markdown", + "id": "a431653d", + "metadata": {}, + "source": [ + "## Common errors\n", + "\n", + "| Symptom | Cause | Fix |\n", + "|---|---|---|\n", + "| `401` on Anthropic calls | Wrong Entra audience (`cognitiveservices.azure.com` instead of `ai.azure.com`) | Build a separate token provider per audience (see the \"Configure\" code cell that builds the two clients) |\n", + "| `404` from Anthropic SDK | Used the stock `Anthropic` class against the Foundry URL | Use `AnthropicFoundry` |\n", + "| `Marketplace agreement required` on first deploy | Tenant has not accepted the Anthropic Marketplace agreement | Have an admin accept once for the tenant |\n", + "| `429` storms | PAYG opus baseline starts low (single-digit RPM range in many regions) | Request an increase from your Foundry portal's quota page; SDK retries from your OpenAI wrapper do **not** carry over |\n", + "| Anthropic returns prose, not JSON | No strict mode | Use the validate-and-retry helper from the structured outputs section |\n", + "| `thinking` block leaks to end users | Rendering the whole `content` array | Filter `b.type == \"text\"` before display |\n", + "| Tool calls silently dropped after porting | `tool_result` sent as `role:\"tool\"` instead of a user message with a `tool_result` block | Use the adapter; never wrap tool results by hand |\n", + "\n", + "## Best practices\n", + "\n", + "- **Keep the adapter even after you've fully switched over.** The next provider arrives faster than the last one did.\n", + "- **Don't optimize the prompt yet.** Prove parity on the unchanged prompt first; per-provider prompt tuning is a separate pass.\n", + "- **Use `cache_control` on system prompts >2K tokens.** Cache reads are excluded from input-TPM and are significantly cheaper than uncached input on hits. Confirm the exact discount in your current Foundry pricing for Claude.\n", + "- **Start `output_config.effort` at `medium`.** Raise to `high` for agent loops; only raise further if your eval shows a measurable quality lift; thinking tokens are billed.\n", + "- **Strip `thinking` blocks before render.** They are for the model, not the user.\n", + "- **Run two cost reports during migration.** Claude rolls up under Azure Marketplace (CCU); your OpenAI deployments on Foundry roll up under Foundry consumption. Reconcile both.\n", + "\n", + "## Rollout\n", + "\n", + "A four-stage rollout, with explicit rollback triggers:\n", + "\n", + "1. **Side-by-side test.** Adapter live, `PROVIDER=openai` everywhere (users still see gpt-5.5). Send every Nth request *also* to claude-opus-4-8 in the background; score offline. Run ≥ 1 week. No customer impact.\n", + "2. **Gradual switch.** Start sending real user traffic to `PROVIDER=anthropic`: 5% → 25% → 50% → 100%, ≥ 1 week per step. Watch the gates from the promotion-gate section *plus* operational metrics.\n", + "3. **New default.** Flip the cohort default to `claude-opus-4-8`. Keep the gpt-5.5 deployment alive as the rollback target.\n", + "4. **Retire.** After 30 clean days at 100%, delete or repurpose the gpt-5.5 deployment.\n", + "\n", + "**Roll back if:** tool-shape parity <99.5% over any hour, structured-output retry rate >10% over any hour, P99 latency >2× baseline for >15 minutes, or any 5xx rate >0.5% over 5 minutes." + ] + }, + { + "cell_type": "markdown", + "id": "170a0c99", + "metadata": {}, + "source": [ + "## Hand it to a coding agent\n", + "\n", + "When you're ready to apply this to a real codebase, paste this into your agent:\n", + "\n", + "> You are migrating this app from `gpt-5.5` to `claude-opus-4-8`. Both are Foundry deployments on the same resource. Do the smallest set of changes that lets `PROVIDER=anthropic` work end-to-end behind a feature flag.\n", + ">\n", + "> 1. Add `anthropic>=0.55` to requirements; do not bump other versions. `AnthropicFoundry` ships in that package.\n", + "> 2. Introduce a `Conversation` dataclass (system, messages[], tools[], optional thinking + effort) and refactor every chat completion call site to build one and call `adapter.run(conv)`.\n", + "> 3. Add `OpenAIAdapter` and `AnthropicAdapter` behind a `PROVIDER` env var. Default `PROVIDER=openai`.\n", + "> 4. Translate tool schemas using `openai_tools_to_anthropic()`; wrap tool results per provider.\n", + "> 5. Post-hoc JSON-schema validation on the Anthropic path with a retry budget of 1.\n", + "> 6. Route all stream consumers through the `StreamEvent` normalizer.\n", + "> 7. Add a second Entra token provider with audience `https://ai.azure.com/.default` for the Anthropic path. Keep the existing `cognitiveservices.azure.com` provider for the OpenAI path.\n", + "> 8. Update network allowlists for the Foundry host.\n", + "> 9. Run this notebook in CI; gate merges on PASS.\n", + ">\n", + "> Do **not**: modify business prompts, change retry/timeout defaults, touch the gpt-5.5 deployment, delete `gpt-5.5`, or use the stock `Anthropic` SDK class (use `AnthropicFoundry`).\n", + "\n", + "## Cleanup\n", + "\n", + "Nothing in this notebook created resources. **Do not delete the `gpt-5.5` deployment yet.** Keep it alive as the rollback target through the four-stage rollout above. Once you have 30 clean days at 100% on claude-opus-4-8, decommission `gpt-5.5` and update cost dashboards to drop the OpenAI-on-Foundry columns in favor of the Marketplace / CCU column.\n", + "\n", + "---\n", + "\n", + "### Takeaway\n", + "\n", + "A cross-provider migration is one adapter, two SDKs, and one extra Entra audience. The adapter is the artifact worth keeping; the next provider arrives faster than the last one did.\n", + "\n", + "### See also\n", + "\n", + "- [Microsoft Learn: Use Claude models in Foundry Models](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-claude?tabs=python)\n", + "- [Microsoft Learn: Claude models in Foundry Models (concepts)](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/claude-models?tabs=pay-go)\n", + "- [Azure-Samples/claude](https://github.com/Azure-Samples/claude): `azd up` starter that provisions the resource, deployments, and Entra wiring" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv (3.13.14.final.0)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.14" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/registry.yaml b/registry.yaml index 906afe8..3cc4edc 100644 --- a/registry.yaml +++ b/registry.yaml @@ -14,6 +14,36 @@ # - date: Publication date (YYYY-MM-DD) # - tags: List of topic tags +- slug: migrate-gpt-4o-mini-to-gpt-5-1 + path: notebooks/migrate-gpt-4o-mini-to-gpt-5-1.ipynb + title: "Migrate gpt-4o-mini to gpt-5.1" + description: "Port a Chat Completions app from gpt-4o-mini to gpt-5.1: rename max_tokens to max_completion_tokens, remove unsupported sampling, and use reasoning_effort and verbosity." + date: "2026-07-07" + authors: + - github: meerakurup + tags: + - models + - inference + - chat-completions + - reasoning + - responses + - azure-openai + +- slug: migrate-gpt-5-5-to-claude-opus-4-8 + path: notebooks/migrate-gpt-5-5-to-claude-opus-4-8.ipynb + title: "Migrate GPT-5.5 to Claude Opus 4.8" + description: "Move a Chat Completions workload from GPT-5.5 to Claude Opus 4.8 on Microsoft Foundry with one adapter, side-by-side eval gates, structured output retries, streaming normalization, and rollout checks." + date: "2026-06-30" + authors: + - github: meerakurup + tags: + - models + - inference + - evaluation + - sdk + - azure-openai + - anthropic + - slug: migrate-oyd-to-foundry-iq path: notebooks/migrate-oyd-to-foundry-iq.ipynb title: "Migrate Azure OpenAI On Your Data to Foundry IQ"