Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions providers/amazon/docs/operators/redshift/redshift_data.rst
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,16 @@ the statement id won't be there for the next retry, and the operator will submit
instead of reconnecting. Avoid running cleanup on a schedule shorter than your longest
``retry_delay``.

Clearing a task is treated the same as a retry, which matters specifically for a task whose
statement already succeeded: clearing does not delete the stored statement id, so the next attempt
reads it back and returns immediately without submitting the SQL again. See
:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
``[state_store] clear_on_success`` setting that restores "clearing always resubmits."

This is most reliable for deferred tasks (``deferrable=True``); clearing a task that's actively
polling synchronously can cancel the statement via ``on_kill`` before the next attempt gets a
chance to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.

To opt out and always submit fresh SQL on retry, set ``durable=False``:

.. code-block:: python
Expand Down
10 changes: 10 additions & 0 deletions providers/apache/spark/docs/operators.rst
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,16 @@ See :doc:`connections/spark-submit` for how to configure these fields.
Crash recovery in cluster mode requires Airflow 3.3+ (``task_state_store`` support). On earlier
versions the operator falls back to the previous behavior of always submitting fresh.

Clearing a task is treated the same as a retry, which matters specifically for a task whose driver
already succeeded: clearing does not delete the stored driver ID, so the next attempt reads it
back and returns immediately without resubmitting. See
:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
``[state_store] clear_on_success`` setting that restores "clearing always resubmits."

This is most reliable for deferred tasks (``deferrable=True``); clearing a task that's actively
polling synchronously can cancel the driver via ``on_kill`` before the next attempt gets a chance
to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.

Tracking driver status via Kubernetes API
""""""""""""""""""""""""""""""""""""""""""

Expand Down
10 changes: 10 additions & 0 deletions providers/databricks/docs/operators/run_now.rst
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,16 @@ when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is
run id won't be there for the next retry, and the operator will trigger a fresh run instead of
reconnecting. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``.

Clearing a task is treated the same as a retry, which matters specifically for a task whose run
already succeeded: clearing does not delete the stored run id, so the next attempt reads it back
and returns immediately without triggering a new run. See
:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
``[state_store] clear_on_success`` setting that restores "clearing always resubmits."

This is most reliable for deferred tasks (``deferrable=True``); clearing a task that's actively
polling synchronously can cancel the run via ``on_kill`` before the next attempt gets a chance to
reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.

To opt out and always trigger a fresh run on retry, set ``durable=False``:

.. code-block:: python
Expand Down
10 changes: 10 additions & 0 deletions providers/databricks/docs/operators/submit_run.rst
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,16 @@ when someone runs ``airflow state-store clean``. If a task's ``retry_delay`` is
run id won't be there for the next retry, and the operator will submit a fresh run instead of
reconnecting. Avoid running cleanup on a schedule shorter than your longest ``retry_delay``.

Clearing a task is treated the same as a retry, which matters specifically for a task whose run
already succeeded: clearing does not delete the stored run id, so the next attempt reads it back
and returns immediately without submitting a new run. See
:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
``[state_store] clear_on_success`` setting that restores "clearing always resubmits."

This is most reliable for deferred tasks (``deferrable=True``); clearing a task that's actively
polling synchronously can cancel the run via ``on_kill`` before the next attempt gets a chance to
reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.

To opt out and always submit a fresh run on retry, set ``durable=False``:

.. code-block:: python
Expand Down
10 changes: 10 additions & 0 deletions providers/snowflake/docs/operators/snowflake.rst
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,16 @@ between, the handles won't be there for the next retry, and the operator will su
fresh instead of reconnecting. Avoid running cleanup on a schedule shorter than your longest
``retry_delay``.

Clearing a task is treated the same as a retry, which matters specifically for a task whose
statements already succeeded: clearing does not delete the stored handles, so the next attempt
reads them back and returns immediately without submitting the SQL again. See
:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
``[state_store] clear_on_success`` setting that restores "clearing always resubmits."

This is most reliable for deferred tasks (``deferrable=True``); clearing a task that's actively
polling synchronously can cancel the statements via ``on_kill`` before the next attempt gets a
chance to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.

To opt out and always submit fresh SQL on retry, set ``durable=False``:

.. code-block:: python
Expand Down