Spyre in use or not

[ALutz273]

💡 Feature Description

As a system administrator, I want a CLI command that shows which Spyre AI accelerator cards are currently in use and which are free — without entering any container so that I can quickly assess card availability for maintenance, troubleshooting, or workload planning from the host level.

Currently, determining which Spyre card is assigned to which container requires manually entering each container and inspecting its environment variables (e.g. SPYRE_DEVICE or equivalent). On systems with multiple containers and multiple Spyre cards, this is time-consuming and error-prone. A host-level command (e.g. spyre-status, project-ai-services status cards, or a similar CLI entry point) should provide a consolidated, human-readable overview of card allocation across all running inference services.

❗ Why This Matters

Without a host-level visibility tool, routine operations like:

• Scheduling maintenance on a specific Spyre card
• Identifying free cards for additional deployments
• Diagnosing resource contention between inference containers

...require iterating through all running containers one by one. This significantly increases operational overhead and the risk of errors during maintenance windows. A single command from the host system would dramatically improve observability and operational safety.

🧩 Requirements

• A command (or subcommand of an existing CLI) that runs outside any container and reports the status of all Spyre AI accelerator cards on the host
• Output should clearly show:
  • Card identifier (e.g. slot, PCI address, or device name)
  • Current status: in use / free
  • If in use: which container/service is using it
• Should work without requiring the user to exec into any container
• Human-readable default output; optional structured output (e.g. --json) for scripting/automation
• Should function on RHEL/ppc64le host environments where project-ai-services is deployed

📚 Additional Details

• Current workaround: exec into each container individually and check environment variables to determine card assignment — does not scale with multiple containers and multiple cards
• Typical use case: before scheduling maintenance on a specific Spyre card, an admin needs to confirm no active inference workload is bound to it
• Environment: IBM Spyre AI accelerator cards on IBM Power (ppc64le), RHEL host, containers managed by project-ai-services
• Related: inspection of device assignment is currently only possible from within the container scope (environment variable level), not from the host orchestration layer

[mkumatag]

Thanks for the input. This looks like a valid concern, especially when we're dealing with more cards assigned and deploying multiple containers using those cards. However, some of these functionalities may require additional modifications to the underlying programs that were designed. We are working with an external team to determine the feasibility or find a workaround at the AI-Services project itself.