[Azure AI Foundry] Add logs datastream to support prompt response log

packages/azure_ai_foundry/_dev/build/docs/README.md

@@ -4,6 +4,37 @@ Azure AI Foundry provides a comprehensive suite of AI services that enable devel
 
 ## Data streams
 
+### Logs
+
+The Azure AI Foundry logs data stream captures the gateway log events.
+
+These are the supported Azure log categories:
+
+| Data Stream |       Log Category       |
+|:-----------:|:------------------------:|
+|    logs     | ApiManagementGatewayLogs |
+
+#### Requirements and setup
+
+Refer to the [Azure Logs](https://docs.elastic.co/integrations/azure) page for more information on how to set up and use this integration.
+
+#### API Gateway Logs
+
+The API Management services provide the advanced logging capabilities. The `ApiManagementGatewayLogs` category comes under the advanced logging. This is not directly available in the Azure AI Foundry service itself. You have to set up the API Management services in Azure to access the Azure AI Foundry models. When the setup is complete, add the diagnostic setting for the API Management service.
+
+For more information on how to implement the comprehensive solution using API Management services to monitor the Azure AI Foundry services, check the [AI Foundry API](https://learn.microsoft.com/en-us/azure/api-management/azure-ai-foundry-api) page.
+
+**Diagnostic settings**
+
+- Enable the category `Logs related to ApiManagement Gateway` to stream the logs to the event hub.
+
+```text
+   ┌──────────────────┐      ┌──────────────┐     ┌─────────────────┐
+   │   APIM service   │      │  Diagnostic  │     │    Event Hub    │
+   │    <<source>>    │─────▶│   settings   │────▶│ <<destination>> │
+   └──────────────────┘      └──────────────┘     └─────────────────┘
+```
+
 ### Metrics
 
 The metrics data stream collects the cognitive service metrics that is specific to the Azure AI Foundry service.
@@ -40,6 +71,16 @@ If no resource filter is specified, then all Azure AI Foundry services inside th
 
 The primary aggregation value will be retrieved for all the metrics contained in the namespaces. The aggregation options are `avg`, `sum`, `min`, `max`, `total`, `count`.
 
+#### Logs Reference
+
+{{event "logs"}}
+
+**ECS Field Reference**
+
+For more details on ECS fields, check the [ECS Field Reference](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) documentation.
+
+{{fields "logs"}}
+
 #### Metrics Reference
 
 The Azure AI Foundry metrics provide insights into the performance and usage of your AI resources. These metrics help in monitoring and optimizing your deployments.

packages/azure_ai_foundry/changelog.yml

@@ -1,3 +1,8 @@
+- version: "0.2.0"
+  changes:
+    - description: Add support for API Management Gateway logs.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/14346
 - version: "0.1.1"
   changes:
     - description: Fix overview dashboard visualization labels.

packages/azure_ai_foundry/data_stream/logs/_dev/test/pipeline/test-ai-foundry-gateway.log

@@ -0,0 +1,4 @@
+{"DeploymentVersion":"0.44.18652.0","Level":4,"Sku":"Basic","callerIpAddress":"89.160.20.156","category":"GatewayLogs","correlationId":"99789635-18d0-480f-8182-a3a3c477a6a4","durationMs":176,"isRequestSuccess":true,"location":"East US","operationName":"Microsoft.ApiManagement/GatewayLogs","properties":{"apiId":"azure-ai-foundry-apim-api","apiRevision":"1","backendMethod":"POST","backendProtocol":"HTTP/1.1","backendRequestBody":"{\"model\": \"gpt-chat-pilot\", \"messages\": [{\"role\": \"user\", \"content\": \"capital of tamilnadu?\"}]}","backendResponseBody":"{\"choices\":[{\"content_filter_results\":{\"custom_blocklists\":[],\"hate\":{\"filtered\":true,\"severity\":\"low\"},\"profanity\":{\"filtered\":false,\"detected\":false},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}},\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"The capital of Tamil Nadu is Chennai.\",\"role\":\"assistant\"}}],\"created\":1719901891,\"id\":\"chatcmpl-9gRL14hGa8nQstOJKvLjh7EyulsnT\",\"model\":\"gpt-35-turbo\",\"object\":\"chat.completion\",\"prompt_filter_results\":[{\"prompt_index\":0,\"content_filter_results\":{\"custom_blocklists\":[],\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"profanity\":{\"filtered\":false,\"detected\":false},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}],\"system_fingerprint\":null,\"usage\":{\"completion_tokens\":8,\"prompt_tokens\":14,\"total_tokens\":22}}\n","backendResponseCode":200,"backendTime":176,"backendUrl":"https://obs-aifoundry-test-01.aifoundry.azure.com/aifoundry/deployments/gpt-chat-pilot/chat/completions?api-version=2024-02-15-preview","cache":"none","clientProtocol":"HTTP/1.1","clientTlsVersion":"1.3","method":"POST","operationId":"ChatCompletions_Create","requestBody":"{\"model\": \"gpt-chat-pilot\", \"messages\": [{\"role\": \"user\", \"content\": \"capital of tamilnadu?\"}]}","requestSize":536,"responseBody":"{\"choices\":[{\"content_filter_results\":{\"custom_blocklists\":[],\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"profanity\":{\"filtered\":false,\"detected\":false},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}},\"finish_reason\":\"stop\",\"index\":0,\"logprobs\":null,\"message\":{\"content\":\"The capital of Tamil Nadu is Chennai.\",\"role\":\"assistant\"}}],\"created\":1719901891,\"id\":\"chatcmpl-9gRL14hGa8nQstOJKvLjh7EyulsnT\",\"model\":\"gpt-35-turbo\",\"object\":\"chat.completion\",\"prompt_filter_results\":[{\"prompt_index\":0,\"content_filter_results\":{\"custom_blocklists\":[],\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"profanity\":{\"filtered\":false,\"detected\":false},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}],\"system_fingerprint\":null,\"usage\":{\"completion_tokens\":8,\"prompt_tokens\":14,\"total_tokens\":22}}\n","responseCode":200,"responseSize":1604,"url":"https://azure-aifoundry-apm.azure-api.net/deployments/gpt-chat-pilot/chat/completions?api-version=2024-02-15-preview"},"resourceId":"/SUBSCRIPTIONS/12345CABCB4-86E8-404F-A3D2-1DC9982F45CA/RESOURCEGROUPS/OBS-AIFOUNDRY-APIM-TEST/PROVIDERS/MICROSOFT.APIMANAGEMENT/SERVICE/AZURE-AIFOUNDRY-APM","resultType":"Succeeded","time":"2024-07-02T06:31:31.4465902Z","truncated":0}
+{"DeploymentVersion":"0.44.18652.0","Level":4,"Sku":"Basic","callerIpAddress":"81.2.69.144","category":"GatewayLogs","correlationId":"0aa398fe-7b98-49aa-8352-52ce04ddc34a","durationMs":453,"isRequestSuccess":false,"location":"East US","operationName":"Microsoft.ApiManagement/GatewayLogs","properties":{"apiId":"azure-ai-foundry-apim-api","apiRevision":"1","backendMethod":"POST","backendProtocol":"HTTP/1.1","backendRequestBody":"{\"model\": \"gpt-chat-pilot\", \"messages\": [{\"role\": \"user\", \"content\": \"test\"}]}","backendResponseBody":"{\"error\":{\"message\":\"The response was filtered due to the prompt triggering Azure AI Foundry's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766\",\"type\":null,\"param\":\"prompt\",\"code\":\"content_filter\",\"status\":400,\"innererror\":{\"code\":\"ResponsibleAIPolicyViolation\",\"content_filter_result\":{\"custom_blocklists\":[{\"filtered\":true,\"id\":\"CustomBlockList601\"}],\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"profanity\":{\"filtered\":false,\"detected\":false},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}}}","backendResponseCode":400,"backendTime":452,"backendUrl":"https://obs-aifoundry-test-01.aifoundry.azure.com/aifoundry/deployments/gpt-chat-pilot/chat/completions?api-version=2024-02-15-preview","cache":"none","clientProtocol":"HTTP/1.1","clientTlsVersion":"1.3","method":"POST","operationId":"ChatCompletions_Create","requestBody":"{\"model\": \"gpt-chat-pilot\", \"messages\": [{\"role\": \"user\", \"content\": \"test\"}]}","requestSize":519,"responseBody":"{\"error\":{\"message\":\"The response was filtered due to the prompt triggering Azure AI Foundry's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766\",\"type\":null,\"param\":\"prompt\",\"code\":\"content_filter\",\"status\":400,\"innererror\":{\"code\":\"ResponsibleAIPolicyViolation\",\"content_filter_result\":{\"custom_blocklists\":[{\"filtered\":true,\"id\":\"CustomBlockList601\"}],\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"profanity\":{\"filtered\":false,\"detected\":false},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}}}","responseCode":400,"responseSize":1360,"url":"https://azure-aifoundry-apm.azure-api.net/deployments/gpt-chat-pilot/chat/completions?api-version=2024-02-15-preview"},"resourceId":"/SUBSCRIPTIONS/12345CABCB4-86E8-404F-A3D2-1DC9982F45CA/RESOURCEGROUPS/OBS-AIFOUNDRY-APIM-TEST/PROVIDERS/MICROSOFT.APIMANAGEMENT/SERVICE/AZURE-AIFOUNDRY-APM","resultType":"Failed","time":"2024-07-02T06:14:56.2371877Z","truncated":0}
+{"DeploymentVersion":"0.47.21405.0","Level":4,"Sku":"Basic","callerIpAddress":"81.2.69.144","category":"GatewayLogs","correlationId":"503d647d-3f8c-45b4-a9df-ec02ec01e988","durationMs":811,"isRequestSuccess":false,"location":"East US","operationName":"Microsoft.ApiManagement/GatewayLogs","properties":{"apiId":"azure-ai-foundry-apim-api","apiRevision":"1","backendMethod":"POST","backendProtocol":"HTTP/1.1","backendRequestBody":"{\"model\": \"gpt-chat-pilot\", \"messages\": [{\"role\": \"user\", \"content\": \"virus\"}]}","backendResponseBody":"{\"error\":{\"message\":\"The response was filtered due to the prompt triggering Azure AI Foundry's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766\",\"type\":null,\"param\":\"prompt\",\"code\":\"content_filter\",\"status\":400,\"innererror\":{\"code\":\"ResponsibleAIPolicyViolation\",\"content_filter_result\":{\"custom_blocklists\":[{\"filtered\":true,\"id\":\"CustomBlockList601\"}],\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"jailbreak\":{\"filtered\":false,\"detected\":false},\"profanity\":{\"filtered\":false,\"detected\":false},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":true,\"severity\":\"medium\"}}}}}","backendResponseCode":400,"backendTime":810,"backendUrl":"https://obs-aifoundry-test-01.aifoundry.azure.com/aifoundry/deployments/gpt-chat-pilot/chat/completions?api-version=2024-02-15-preview","cache":"none","clientProtocol":"HTTP/1.1","clientTlsCipherSuite":"TLS_UNK0_WITH_AES_256_GCM_SHA384","clientTlsVersion":"1.3","method":"POST","operationId":"ChatCompletions_Create","requestBody":"{\"model\": \"gpt-chat-pilot\", \"messages\": [{\"role\": \"user\", \"content\": \"virus\"}]}","requestSize":496,"responseBody":"{\"error\":{\"message\":\"The response was filtered due to the prompt triggering Azure AI Foundry's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766\",\"type\":null,\"param\":\"prompt\",\"code\":\"content_filter\",\"status\":400,\"innererror\":{\"code\":\"ResponsibleAIPolicyViolation\",\"content_filter_result\":{\"custom_blocklists\":[{\"filtered\":true,\"id\":\"CustomBlockList601\"}],\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"jailbreak\":{\"filtered\":false,\"detected\":false},\"profanity\":{\"filtered\":false,\"detected\":false},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":false,\"severity\":\"safe\"}}}}}","responseCode":400,"responseSize":1471,"url":"https://azure-aifoundry-apm.azure-api.net/deployments/gpt-chat-pilot/chat/completions?api-version=2024-02-15-preview"},"resourceId":"/SUBSCRIPTIONS/12345CABCB4-86E8-404F-A3D2-1DC9982F45CA/RESOURCEGROUPS/OBS-AIFOUNDRY-APIM-TEST/PROVIDERS/MICROSOFT.APIMANAGEMENT/SERVICE/AZURE-AIFOUNDRY-APM","resultType":"Failed","time":"2025-03-05T10:39:40.2260356Z","truncated":0}
+{"DeploymentVersion":"0.48.2706.0","Level":4,"Sku":"StandardV2","callerIpAddress":"81.2.69.144","category":"GatewayLogs","correlationId":"134005c6-ef24-4eb3-ab74-08fad5176d5f","durationMs":336,"isRequestSuccess":false,"location":"East US","operationName":"Microsoft.ApiManagement/GatewayLogs","properties":{"apiId":"azure-ai-foundry-apim-api","apiRevision":"1","backendId":"azure-ai-foundry-apim-api-ai-endpoint","backendMethod":"POST","backendProtocol":"HTTP/1.1","backendRequestBody":"{\"model\": \"gpt-4o\", \"messages\": [{\"role\": \"user\", \"content\": \"how to kill a dog?\"}]}","backendResponseBody":"{\"error\":{\"message\":\"The response was filtered due to the prompt triggering Azure AI Foundry's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766\",\"type\":null,\"param\":\"prompt\",\"code\":\"content_filter\",\"status\":400,\"innererror\":{\"code\":\"ResponsibleAIPolicyViolation\",\"content_filter_result\":{\"custom_blocklists\":{\"filtered\":false},\"hate\":{\"filtered\":false,\"severity\":\"safe\"},\"jailbreak\":{\"filtered\":false,\"detected\":false},\"profanity\":{\"filtered\":false,\"detected\":false},\"self_harm\":{\"filtered\":false,\"severity\":\"safe\"},\"sexual\":{\"filtered\":false,\"severity\":\"safe\"},\"violence\":{\"filtered\":true,\"severity\":\"medium\"}}}}}","backendResponseCode":400,"backendTime":624,"backendUrl":"https://ai-muthuhub687016784742.services.ai.azure.com/models/chat/completions?api-version=2024-05-01-preview","cache":"none","clientProtocol":"HTTP/1.1","method":"POST","operationId":"getChatCompletions","responseCode":400,"responseSize":1311,"url":"https://azure-ai-foundry-apim.azure-api.net/aifoundry/models/chat/completions?api-version=2024-05-01-preview"},"resourceId":"/SUBSCRIPTIONS/12345CABCB4-86E8-404F-A3D2-1DC9982F45CA/RESOURCEGROUPS/OBS-AIFOUNDRY-APIM-TEST/PROVIDERS/MICROSOFT.APIMANAGEMENT/SERVICE/AZURE-AI-FOUNDRY-APIM","resultType":"Failed","serviceName":"azure-ai-foundry-apim","time":"2025-06-23T11:21:53.3239304Z","truncated":0 }