Before this patch, we ran some reconciliation when a form is edited to
ensure that existing SubmissionValueVariable instances don't have
'null' form_variable foreign keys because the FormVariable set of a
form was updated after editing (a) step(s) - the idea here was to be
eventually consistent. This turns out not to be necessary (if we can
trust our test suite) because the load_submission_value_variables_state
method on the Submission operates on the variable keys rather than
the FK relation and is able to properly resolve everything. This is
also the interface that developers should use when accessing the
submission values and it appears to be done properly in the
registration backends, otherwise tests would likely fail.

This re-coupling was extended in #4900, after it was noticed in #4824
that the re-coupling didn't happen for other forms that use the same
re-usable form definition. At the time, we didn't understand how this
seemingly didn't cause issues or at least didn't result in issues
being reported to us, but we can now conclude that it just wasn't a
problem in the first place because the proper interfaces/service layer
are/were being used and everything is done/reconciled in-memory when
comparing/populating submission variables with form variables.

A further cleanup step will then also be to remove this FK field from
the submission value variable model, as it is not necessary.

Offloading this to celery results in more, separate tasks each competing
for database locks and can lead to integrity errors.

Changing the business logic of form variable creation/update to
make sure a FormDefinition write results in a sync action gives
a single trigger for these kind of changes, and it's responsible
for updating all the forms that are affected by it.

Since multiple FormDefinition/FormStep writes happen in parallel
because of parallel client requests, we can only update or
create variables and can't delete variables that are not present
in our current form definition, as they may belong to another
request. This shouldn't immediately cause problems, as old,
unused variables don't have an 'execution path'. We piggy
back on the variables bulk update endpoint/transaction to
clean these up, as that call is made after all the form step
persistence succeeded so we know that everything is resolved
by then.

The left-over variables problem only exists when updating a
form step to use a different form definition. It's not
relevant when a form definition is updated through the
admin or a form step is added to a form.

Offloading this to celery results in more, separate tasks each competing
for database locks and can lead to integrity errors.

Changing the business logic of form variable creation/update to
make sure a FormDefinition write results in a sync action gives
a single trigger for these kind of changes, and it's responsible
for updating all the forms that are affected by it.

Since multiple FormDefinition/FormStep writes happen in parallel
because of parallel client requests, we can only update or
create variables and can't delete variables that are not present
in our current form definition, as they may belong to another
request. This shouldn't immediately cause problems, as old,
unused variables don't have an 'execution path'. We piggy
back on the variables bulk update endpoint/transaction to
clean these up, as that call is made after all the form step
persistence succeeded so we know that everything is resolved
by then.

The left-over variables problem only exists when updating a
form step to use a different form definition. It's not
relevant when a form definition is updated through the
admin or a form step is added to a form.

… endpoint

The bulk update endpoint may no longer manage the variables for form
steps, as that's taken care of by the form step endpoint now. But,
user defined variables must still be managed.

Instead of triggering celery tasks, perform the variable state
synchronization for a form in the bulk update endpoint. This
endpoint now still validates the state of all component/user
defined variables, but no longer drops all the variables and
re-creates them, instead it only touches user defined
variables and leaves the form step variables alone.

Since this is called after the steps have been sorted out,
a complete view of left-over variables from earlier
form definitions is available and those can be cleaned
up safely now.

Instead of triggering celery tasks, perform the variable state
synchronization for a form in the bulk update endpoint. This
endpoint now still validates the state of all component/user
defined variables, but no longer drops all the variables and
re-creates them, instead it only touches user defined
variables and leaves the form step variables alone.

Since this is called after the steps have been sorted out,
a complete view of left-over variables from earlier
form definitions is available and those can be cleaned
up safely now.

Offloading this to celery results in more, separate tasks each competing
for database locks and can lead to integrity errors.

Changing the business logic of form variable creation/update to
make sure a FormDefinition write results in a sync action gives
a single trigger for these kind of changes, and it's responsible
for updating all the forms that are affected by it.

Since multiple FormDefinition/FormStep writes happen in parallel
because of parallel client requests, we can only update or
create variables and can't delete variables that are not present
in our current form definition, as they may belong to another
request. This shouldn't immediately cause problems, as old,
unused variables don't have an 'execution path'. We piggy
back on the variables bulk update endpoint/transaction to
clean these up, as that call is made after all the form step
persistence succeeded so we know that everything is resolved
by then.

The left-over variables problem only exists when updating a
form step to use a different form definition. It's not
relevant when a form definition is updated through the
admin or a form step is added to a form.

… endpoint

The bulk update endpoint may no longer manage the variables for form
steps, as that's taken care of by the form step endpoint now. But,
user defined variables must still be managed.

Instead of triggering celery tasks, perform the variable state
synchronization for a form in the bulk update endpoint. This
endpoint now still validates the state of all component/user
defined variables, but no longer drops all the variables and
re-creates them, instead it only touches user defined
variables and leaves the form step variables alone.

Since this is called after the steps have been sorted out,
a complete view of left-over variables from earlier
form definitions is available and those can be cleaned
up safely now.

We can express everything now in terms of synchronize_for form_definition.

Before this patch, we ran some reconciliation when a form is edited to
ensure that existing SubmissionValueVariable instances don't have
'null' form_variable foreign keys because the FormVariable set of a
form was updated after editing (a) step(s) - the idea here was to be
eventually consistent. This turns out not to be necessary (if we can
trust our test suite) because the load_submission_value_variables_state
method on the Submission operates on the variable keys rather than
the FK relation and is able to properly resolve everything. This is
also the interface that developers should use when accessing the
submission values and it appears to be done properly in the
registration backends, otherwise tests would likely fail.

This re-coupling was extended in #4900, after it was noticed in #4824
that the re-coupling didn't happen for other forms that use the same
re-usable form definition. At the time, we didn't understand how this
seemingly didn't cause issues or at least didn't result in issues
being reported to us, but we can now conclude that it just wasn't a
problem in the first place because the proper interfaces/service layer
are/were being used and everything is done/reconciled in-memory when
comparing/populating submission variables with form variables.

A further cleanup step will then also be to remove this FK field from
the submission value variable model, as it is not necessary.

Offloading this to celery results in more, separate tasks each competing
for database locks and can lead to integrity errors.

Changing the business logic of form variable creation/update to
make sure a FormDefinition write results in a sync action gives
a single trigger for these kind of changes, and it's responsible
for updating all the forms that are affected by it.

Since multiple FormDefinition/FormStep writes happen in parallel
because of parallel client requests, we can only update or
create variables and can't delete variables that are not present
in our current form definition, as they may belong to another
request. This shouldn't immediately cause problems, as old,
unused variables don't have an 'execution path'. We piggy
back on the variables bulk update endpoint/transaction to
clean these up, as that call is made after all the form step
persistence succeeded so we know that everything is resolved
by then.

The left-over variables problem only exists when updating a
form step to use a different form definition. It's not
relevant when a form definition is updated through the
admin or a form step is added to a form.

… endpoint

The bulk update endpoint may no longer manage the variables for form
steps, as that's taken care of by the form step endpoint now. But,
user defined variables must still be managed.

Instead of triggering celery tasks, perform the variable state
synchronization for a form in the bulk update endpoint. This
endpoint now still validates the state of all component/user
defined variables, but no longer drops all the variables and
re-creates them, instead it only touches user defined
variables and leaves the form step variables alone.

Since this is called after the steps have been sorted out,
a complete view of left-over variables from earlier
form definitions is available and those can be cleaned
up safely now.

We can express everything now in terms of synchronize_for form_definition.

Before this patch, we ran some reconciliation when a form is edited to
ensure that existing SubmissionValueVariable instances don't have
'null' form_variable foreign keys because the FormVariable set of a
form was updated after editing (a) step(s) - the idea here was to be
eventually consistent. This turns out not to be necessary (if we can
trust our test suite) because the load_submission_value_variables_state
method on the Submission operates on the variable keys rather than
the FK relation and is able to properly resolve everything. This is
also the interface that developers should use when accessing the
submission values and it appears to be done properly in the
registration backends, otherwise tests would likely fail.

This re-coupling was extended in #4900, after it was noticed in #4824
that the re-coupling didn't happen for other forms that use the same
re-usable form definition. At the time, we didn't understand how this
seemingly didn't cause issues or at least didn't result in issues
being reported to us, but we can now conclude that it just wasn't a
problem in the first place because the proper interfaces/service layer
are/were being used and everything is done/reconciled in-memory when
comparing/populating submission variables with form variables.

A further cleanup step will then also be to remove this FK field from
the submission value variable model, as it is not necessary.

Offloading this to celery results in more, separate tasks each competing
for database locks and can lead to integrity errors.

Changing the business logic of form variable creation/update to
make sure a FormDefinition write results in a sync action gives
a single trigger for these kind of changes, and it's responsible
for updating all the forms that are affected by it.

Since multiple FormDefinition/FormStep writes happen in parallel
because of parallel client requests, we can only update or
create variables and can't delete variables that are not present
in our current form definition, as they may belong to another
request. This shouldn't immediately cause problems, as old,
unused variables don't have an 'execution path'. We piggy
back on the variables bulk update endpoint/transaction to
clean these up, as that call is made after all the form step
persistence succeeded so we know that everything is resolved
by then.

The left-over variables problem only exists when updating a
form step to use a different form definition. It's not
relevant when a form definition is updated through the
admin or a form step is added to a form.

… endpoint

The bulk update endpoint may no longer manage the variables for form
steps, as that's taken care of by the form step endpoint now. But,
user defined variables must still be managed.

Instead of triggering celery tasks, perform the variable state
synchronization for a form in the bulk update endpoint. This
endpoint now still validates the state of all component/user
defined variables, but no longer drops all the variables and
re-creates them, instead it only touches user defined
variables and leaves the form step variables alone.

Since this is called after the steps have been sorted out,
a complete view of left-over variables from earlier
form definitions is available and those can be cleaned
up safely now.

We can express everything now in terms of synchronize_for form_definition.

**Removed the submission value variable recoupling**

Before this patch, we ran some reconciliation when a form is edited to
ensure that existing SubmissionValueVariable instances don't have
'null' form_variable foreign keys because the FormVariable set of a
form was updated after editing (a) step(s) - the idea here was to be
eventually consistent. This turns out not to be necessary (if we can
trust our test suite) because the load_submission_value_variables_state
method on the Submission operates on the variable keys rather than
the FK relation and is able to properly resolve everything. This is
also the interface that developers should use when accessing the
submission values and it appears to be done properly in the
registration backends, otherwise tests would likely fail.

This re-coupling was extended in #4900, after it was noticed in #4824
that the re-coupling didn't happen for other forms that use the same
re-usable form definition. At the time, we didn't understand how this
seemingly didn't cause issues or at least didn't result in issues
being reported to us, but we can now conclude that it just wasn't a
problem in the first place because the proper interfaces/service layer
are/were being used and everything is done/reconciled in-memory when
comparing/populating submission variables with form variables.

A further cleanup step will then also be to remove this FK field from
the submission value variable model, as it is not necessary.

**Run form variable sync in same transaction**

Offloading this to celery results in more, separate tasks each competing
for database locks and can lead to integrity errors.

Changing the business logic of form variable creation/update to
make sure a FormDefinition write results in a sync action gives
a single trigger for these kind of changes, and it's responsible
for updating all the forms that are affected by it.

Since multiple FormDefinition/FormStep writes happen in parallel
because of parallel client requests, we can only update or
create variables and can't delete variables that are not present
in our current form definition, as they may belong to another
request. This shouldn't immediately cause problems, as old,
unused variables don't have an 'execution path'. We piggy
back on the variables bulk update endpoint/transaction to
clean these up, as that call is made after all the form step
persistence succeeded so we know that everything is resolved
by then.

The left-over variables problem only exists when updating a
form step to use a different form definition. It's not
relevant when a form definition is updated through the
admin or a form step is added to a form.

**Add test for expected new behaviour of bulk variable update endpoint**

The bulk update endpoint may no longer manage the variables for form
steps, as that's taken care of by the form step endpoint now. But,
user defined variables must still be managed.

**Use the form variables bulk update for consistency**

Instead of triggering celery tasks, perform the variable state
synchronization for a form in the bulk update endpoint. This
endpoint now still validates the state of all component/user
defined variables, but no longer drops all the variables and
re-creates them, instead it only touches user defined
variables and leaves the form step variables alone.

Since this is called after the steps have been sorted out,
a complete view of left-over variables from earlier
form definitions is available and those can be cleaned
up safely now.

Backport-of: #5064

**Removed the submission value variable recoupling**

Before this patch, we ran some reconciliation when a form is edited to
ensure that existing SubmissionValueVariable instances don't have
'null' form_variable foreign keys because the FormVariable set of a
form was updated after editing (a) step(s) - the idea here was to be
eventually consistent. This turns out not to be necessary (if we can
trust our test suite) because the load_submission_value_variables_state
method on the Submission operates on the variable keys rather than
the FK relation and is able to properly resolve everything. This is
also the interface that developers should use when accessing the
submission values and it appears to be done properly in the
registration backends, otherwise tests would likely fail.

This re-coupling was extended in #4900, after it was noticed in #4824
that the re-coupling didn't happen for other forms that use the same
re-usable form definition. At the time, we didn't understand how this
seemingly didn't cause issues or at least didn't result in issues
being reported to us, but we can now conclude that it just wasn't a
problem in the first place because the proper interfaces/service layer
are/were being used and everything is done/reconciled in-memory when
comparing/populating submission variables with form variables.

A further cleanup step will then also be to remove this FK field from
the submission value variable model, as it is not necessary.

**Run form variable sync in same transaction**

Offloading this to celery results in more, separate tasks each competing
for database locks and can lead to integrity errors.

Changing the business logic of form variable creation/update to
make sure a FormDefinition write results in a sync action gives
a single trigger for these kind of changes, and it's responsible
for updating all the forms that are affected by it.

Since multiple FormDefinition/FormStep writes happen in parallel
because of parallel client requests, we can only update or
create variables and can't delete variables that are not present
in our current form definition, as they may belong to another
request. This shouldn't immediately cause problems, as old,
unused variables don't have an 'execution path'. We piggy
back on the variables bulk update endpoint/transaction to
clean these up, as that call is made after all the form step
persistence succeeded so we know that everything is resolved
by then.

The left-over variables problem only exists when updating a
form step to use a different form definition. It's not
relevant when a form definition is updated through the
admin or a form step is added to a form.

**Add test for expected new behaviour of bulk variable update endpoint**

The bulk update endpoint may no longer manage the variables for form
steps, as that's taken care of by the form step endpoint now. But,
user defined variables must still be managed.

**Use the form variables bulk update for consistency**

Instead of triggering celery tasks, perform the variable state
synchronization for a form in the bulk update endpoint. This
endpoint now still validates the state of all component/user
defined variables, but no longer drops all the variables and
re-creates them, instead it only touches user defined
variables and leaves the form step variables alone.

Since this is called after the steps have been sorted out,
a complete view of left-over variables from earlier
form definitions is available and those can be cleaned
up safely now.

Backport-of: #5064