Add KubernetesPodExecOperator for existing Kubernetes Pods

[AlejandroMorgante]

Add KubernetesPodExecOperator to execute commands in a running container of an existing Kubernetes Pod without creating, restarting, or deleting it. This supports pre-warmed or externally managed Pods where avoiding startup latency is important.

The operator validates the target Pod and container, streams stdout and stderr, waits for the command exit code, and reliably closes the exec connection. Optional stdout XCom is disabled by default and protected by a configurable size limit to avoid retaining unbounded output in worker memory.

Provider metadata, RBAC guidance, operator documentation, unit tests, and an end-to-end system test are included.

Tests:

• KubernetesPodExecOperator unit tests (39 passed)
• KubernetesPodExecOperator system test (1 passed)
• Targeted provider mypy and Ruff checks
• Provider metadata and documentation validation

---

Was generative AI tooling used to co-author this PR?

• Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

[kaxil]

Add KubernetesPodExecOperator to execute commands in a running container of an existing Kubernetes Pod without creating, restarting, or deleting it. This supports pre-warmed and API-visible static Pods where avoiding startup latency is important.

The operator validates the target Pod and container, streams stdout and stderr, supports optional stdout XCom, reports non-zero exit codes, and closes the exec connection when the task is killed. Provider metadata, RBAC guidance, and operator documentation are included.

Tests:

• pytest providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod_exec.py -q (32 passed)
• Provider mypy check
• Provider metadata and documentation validation
• Pre-commit and manual checks

Was generative AI tooling used to co-author this PR?

• Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

What's the User story and real-world use cases for this?

[AlejandroMorgante]

@kaxil

What's the User story and real-world use cases for this?

Thanks for asking. The user story is:

As an Airflow user who already manages a long-running Kubernetes Pod, I want to execute commands in one
of its containers as Airflow tasks, without Airflow creating, restarting, or deleting that Pod.

Our concrete use case is running dbt on EKS. Today, our workflow starts a new Pod, initializes the dbt project and its dependencies, and then executes a command such as dbt run. A significant part of the execution time is spent scheduling the Pod and initializing the environment.

We want to manage that Pod separately and keep it warm, then use Airflow to execute ad hoc commands such as dbt run, dbt test, or dbt build. This avoids the repeated cold-start and initialization cost.

More generally, this is useful whenever a Pod’s lifecycle is managed outside the task and its environment is already initialized. The operator exposes the equivalent of kubectl exec as an Airflow task, including Airflow logging and optional XCom output, without assuming ownership of the Pod lifecycle.

[AlejandroMorgante]

@SameerMesiah97 Could you please review it when you have a chance? Thank you!

[SameerMesiah97]

I am 50/50 on this new operator for 3 main reasons:

1. I think this new operator diverges from the existing operators in the sense that it is trying to orchestrate externally managed resources that have their own lifecycle and control plane. This introduces some unclear lifecycle and ownership semantics. For example, what happens if the Pod/container is restarted or replaced while the task is executing, or if the exec connection is lost and the Airflow task retries? Could the original command still be running while the retry starts another exec?

2. Should we be introducing a new operator i.e. KubernetesPodExecOperator or could we consider supporting externally managed Pods as an execution mode of KubernetesPodOperator instead? I can see an argument for keeping them separate because KPO owns the Pod lifecycle whereas this operator would only own the exec lifecycle, but both ultimately represent executing an Airflow task inside a Kubernetes container. I think it would be useful to establish why a separate public operator is preferable before introducing another Kubernetes execution abstraction.

3. I have to build on @kaxil's reservations regarding potential use cases by questioning whether an operator strictly for kubectl exec commands is the right abstraction. What about fetching logs or waiting on/monitoring an externally managed Pod (this may be a senor or trigger; this needs discussion too) ? If the broader use case is for Airflow to interact with Pods whose lifecycle it does not own, then exec feels like only one specific operation within that model. Perhaps, a broader operator that handles several commands may be better.

These points could be addresesd in the description. For an atypical operator in a high blast radius provider, I think we need to really nail the rationale.

[AlejandroMorgante]

Thanks for the feedback. Airflow commonly acts as a client of infrastructure it does not own: SSHOperator does not manage the server, SQL operators do not manage the database, and SsmRunCommandOperator does not manage the target instances. They own only the operation being executed. This operator follows the same model: Airflow owns the exec session, while another system owns the Pod lifecycle.

Although this can be implemented with BashOperator and kubectl exec, that moves additional integration logic to every user: installing and maintaining kubectl on the workers, distributing kubeconfig and cloud authentication helpers, handling shell quoting, and mapping Kubernetes errors and output into Airflow. This operator provides that behavior through KubernetesHook and Airflow connections. The provider already uses the pods/exec API internally, so this is not a new Kubernetes mechanism.

Keeping this provider-neutral also allows GKE- or EKS-specific operators to reuse it and add only their cloud authentication and cluster configuration, following the existing provider layering. I prefer keeping it separate from KPO because KPO owns the complete Pod lifecycle, while this operator deliberately treats the Pod as externally managed.

If the Pod disappears, restarts, or the connection fails, the task fails. A retry creates a new exec session; Airflow never creates, replaces, or deletes the Pod.

[SameerMesiah97]

Although this can be implemented with BashOperator and kubectl exec, that moves additional integration logic to every user: installing and maintaining kubectl on the workers, distributing kubeconfig and cloud authentication helpers, handling shell quoting, and mapping Kubernetes errors and output into Airflow. This operator provides that behavior through KubernetesHook and Airflow connections. The provider already uses the pods/exec API internally, so this is not a new Kubernetes mechanism.

I think this is the strongest argument here. If using this operator saves time and effort for users who want this specific functionality, then I believe it is justified. But like I mentioned above, why only exec? Why not other commands like log or watch? It seems premature and restrictive to narrow it to just exec?

[AlejandroMorgante]

Combining exec, logs, and watch would give one operator several different responsibilities and completion contracts.

pods/exec already allows this operator to start any command available inside the container, stream its stdout and stderr, and wait for its exit code. In contrast, pods/log only consumes a log stream, while watch observes resource state and requires a condition to determine when the task should finish.

This also differs from KubernetesPodOperator: KPO creates a Pod and configures the workload as the container's main process. It then monitors the Pod lifecycle.KubernetesPodExecOperator starts a secondary process through pods/exec, waits only for that process, and leaves the existing Pod running.

[AlejandroMorgante]

@SameerMesiah97 Could you help me with a code review?

I’ve already tested it with the end-to-end system test and an Apache Magpie review.

Thank you! 🙏

[SameerMesiah97]

Looks good.