[Azure AI Foundry] Add logs datastream to support prompt response log

[muthu-mps]

• Enhancement

Proposed commit message

This PR adds logs data stream to the Azure AI Foundry integration. The current category include the prompt and response logging using the API Management Services in Azure.

The Audit and RequestResponse log support will get added as part of different PR.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Related issues

Screenshots

[Copilot]

Pull Request Overview

This pull request enhances the Azure AI Foundry integration by adding a logs data stream to support prompt response logging from the API Management Gateway. Key changes include a version bump in the manifest, addition of new screenshot and input configurations for logs, extensive additions to the logs field definitions and ingest pipelines, and updated documentation for logs and metrics.

Reviewed Changes

Copilot reviewed 16 out of 19 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
packages/azure_ai_foundry/manifest.yml	Updated version, added logs data stream and input with a potential naming inconsistency in the input title.
packages/azure_ai_foundry/docs/README.md	Extended documentation with new section on logs and example event, with a slight mismatch in log category names.
packages/azure_ai_foundry/data_stream/logs/*	Multiple new files for sample events, manifest, fields, and ingest pipeline definitions to support the logs data stream.
packages/azure_ai_foundry/changelog.yml	Updated changelog reflecting the new logs support and Kibana constraint update.
Other files (agent, templates, test files)	Updated to support the new logs configurations via templates and test expectations.

Comments suppressed due to low confidence (3)

packages/azure_ai_foundry/manifest.yml:54

• The input title currently reads "Collect Azure OpenAI logs from Event Hub", which appears inconsistent with the Azure AI Foundry integration. Consider renaming it to "Collect Azure AI Foundry logs from Event Hub" for clarity.

      - type: "azure-eventhub"

packages/azure_ai_foundry/docs/README.md:15

• There is a naming inconsistency for the log category: the documentation lists "ApiManagementGatewayLogs" while sample events and ingest pipeline logic refer to "GatewayLogs". Consider aligning these names throughout the integration.

|    logs     | ApiManagementGatewayLogs |

packages/azure_ai_foundry/data_stream/logs/elasticsearch/ingest_pipeline/default.yml:27

• Verify that the condition in this drop processor uses the intended log category name; if the documentation should use 'ApiManagementGatewayLogs' instead of 'GatewayLogs', update the condition or the documentation for consistency.

      if: "ctx.azure.ai_foundry.category != 'GatewayLogs'"

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: da6c86f

History

• 💚 Build #27739 succeeded 495c2da

cc @muthu-mps

[elastic-sonarqube]

Quality Gate failed

Failed conditions
79.9% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube