Fix airflow celery stop crashing on stale PID file

[Eason09053360]

When a Celery worker dies without cleaning up its PID file (SIGKILL, OOM, host reboot), airflow celery stop reads the dead PID, crashes with an uncaught psutil.NoSuchProcess traceback, and never reaches the PID-file removal at the end of stop_worker. Every subsequent stop fails the same way until the operator deletes the file by hand.

$ echo 999999 > stale.pid
$ airflow celery stop --pid stale.pid
  File ".../celery_command.py", line 386, in stop_worker
    worker_process = psutil.Process(pid)
psutil.NoSuchProcess: process PID not found (pid=999999)
$ echo $?
1
$ ls stale.pid
stale.pid

After this change the command logs a warning, removes the stale file and exits 0, which matches the existing behaviour when no PID file exists at all (stop is idempotent).

$ airflow celery stop --pid stale.pid
[warning] Worker process with PID 999999 is not running, PID file .../stale.pid is stale.
$ echo $?
0
$ ls stale.pid
ls: stale.pid: No such file or directory

The worker start path already treats NoSuchProcess as "stale PID file, clean it up" in check_if_pidfile_process_is_running; the stop path was the only reader that did not.

Only NoSuchProcess is handled on purpose. psutil.AccessDenied means a live process the caller cannot signal (for example a worker started as root and stop run as another user); removing its PID file would be wrong, so that case still surfaces as an error.

Tests

test_if_right_pid_is_read is parametrized over three cases: worker running, PID file stale, and worker exiting between psutil.Process() and terminate(). The last two fail without the fix.

airflow edge stop has the same shape of bug; it lives in a different provider and will be addressed separately.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Fable 5.1)

Generated-by: Claude Code (Fable 5.1) following the guidelines

[rjgoyln]

LGTM overall! I verified the new parametrization against the unpatched stop_worker: both added cases fail with psutil.NoSuchProcess without the fix and pass with it. The full file also passes (30 passed, 2 skipped). remove_existing_pidfile already handles ENOENT, so the stale-PID path correctly remains idempotent.

Two non-blocking notes below, both on the new test.

Just my thoughts — feel free to resolve if you think otherwise.

---

Drafted-by: Claude Code (Opus 5); reviewed by @rjgoyln before posting

[dheerajturaga]

Looks good overall. Minor, non-blocking suggestion: assert the new stale-PID warning with structured caplog for both NoSuchProcess cases.