Keep polling DataFusion pipeline state when the run is not visible yet

[Kunal8954]

closes: #72385

What's the problem

CloudDataFusionStartPipelineOperator starts a pipeline and then polls DataFusionHook.wait_for_pipeline_state() for the run's status. Right after the start call, CDAP hasn't registered the run yet and returns 404 for the first poll or two.

#60688 changed the exception handling, causing the 404 to no longer be caught:

	Raised by _check_response_status_and_data	Caught by wait_for_pipeline_state
Before	AirflowNotFoundException	AirflowException
After	HTTPError	KeyError

HTTPError isn't a KeyError, so the 404 escapes the polling loop and fails the task immediately.

The fix

Changed the exception handling to:

except (HTTPError, KeyError):
    pass

---

<sub>Disclosure: this PR was prepared with the assistance of Claude Code (the commit carries a `Co-Authored-By` trailer). The provider's datafusion unit tests were run before and after the change — the new 404 test fails on `main` and passes with the fix, and the rest of the file is unchanged — along with `ruff check` and `ruff format --check`. I have reviewed the change and take responsibility for it.</sub>

[tgmof]

I think it (kinda) makes sense! But additionally, I would do some AI-review of PR #60688 because I'm guessing it's only the tip of the iceberg if so much exception were changed in 1 PR, probably a bunch of try/except are going to fail in one situation or another. Moreover I would challenge if KeyError is possible at all and if one should retry on KeyError at all...

[Kunal8954]

Yeah, fair point — let me split this into two things.

The HTTPError catch is the actual fix here. That's the regression the issue is
about: _check_response_status_and_data started raising HTTPError on a 404, but
the loop was still only catching KeyError, so the "give it a second, the run
isn't registered yet" tolerance from #10031 quietly broke.

The KeyError catch was already there on main before this PR (also came in with
#60688) — I left it alone rather than ripping it out, since that felt like a
separate change from what the issue is reporting. But you're asking the right
question. I don't actually have a case where get_pipeline_workflow returns a
200 with no "status" key in it, so I can't say for sure it's reachable. If you've
seen it happen, or have a hunch where, I'm happy to dig in.

Auditing all of #60688 for similar issues is a good shout, but honestly feels
like its own effort rather than something to fold into this PR. If a maintainer
wants that done I'm up for picking it up separately, just didn't want to
scope-creep this one.

[tgmof]

Yeah, fair point — let me split this into two things.

The HTTPError catch is the actual fix here. That's the regression the issue is about: _check_response_status_and_data started raising HTTPError on a 404, but the loop was still only catching KeyError, so the "give it a second, the run isn't registered yet" tolerance from #10031 quietly broke.

The KeyError catch was already there on main before this PR (also came in with #60688) — I left it alone rather than ripping it out, since that felt like a separate change from what the issue is reporting. But you're asking the right question. I don't actually have a case where get_pipeline_workflow returns a 200 with no "status" key in it, so I can't say for sure it's reachable. If you've seen it happen, or have a hunch where, I'm happy to dig in.

Auditing all of #60688 for similar issues is a good shout, but honestly feels like its own effort rather than something to fold into this PR. If a maintainer wants that done I'm up for picking it up separately, just didn't want to scope-creep this one.

💯Asbolutely, focus should definitely remain on fixing the issue, thanks a lot for the PR! 👌