Fix deadlocks and races when making form variables state consistent #5064
+523
−876
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sergei-maertens
added
the
needs-backport
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #5064 +/- ##
=======================================
Coverage 96.72% 96.72%
=======================================
Files 770 770
Lines 26542 26538 -4
Branches 3460 3457 -3
=======================================
- Hits 25672 25670 -2
+ Misses 607 606 -1
+ Partials 263 262 -1 ☔ View full report in Codecov by Sentry. |
sergei-maertens
force-pushed
the
issue/5058-fix-deadlocks-and-races
branch
from
de3d480 to
7dec89c
Compare
sergei-maertens
commented
Jan 30, 2025
sergei-maertens
commented
Jan 30, 2025
| @@ -64,75 +65,6 @@ def test_staff_required(self): | |||
|
|
|||
| self.assertEqual(status.HTTP_403_FORBIDDEN, response.status_code) | |||
|
|
|||
| def test_bulk_create_and_update(self): | |||
There was a problem hiding this comment.
this is covered entirely in the new test now
sergei-maertens
force-pushed
the
issue/5058-fix-deadlocks-and-races
branch
2 times, most recently
from
3e1c78c to
0a2fa51
Compare
Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary.
Offloading this to celery results in more, separate tasks each competing for database locks and can lead to integrity errors. Changing the business logic of form variable creation/update to make sure a FormDefinition write results in a sync action gives a single trigger for these kind of changes, and it's responsible for updating all the forms that are affected by it. Since multiple FormDefinition/FormStep writes happen in parallel because of parallel client requests, we can only update or create variables and can't delete variables that are not present in our current form definition, as they may belong to another request. This shouldn't immediately cause problems, as old, unused variables don't have an 'execution path'. We piggy back on the variables bulk update endpoint/transaction to clean these up, as that call is made after all the form step persistence succeeded so we know that everything is resolved by then. The left-over variables problem only exists when updating a form step to use a different form definition. It's not relevant when a form definition is updated through the admin or a form step is added to a form.
… endpoint The bulk update endpoint may no longer manage the variables for form steps, as that's taken care of by the form step endpoint now. But, user defined variables must still be managed.
Instead of triggering celery tasks, perform the variable state synchronization for a form in the bulk update endpoint. This endpoint now still validates the state of all component/user defined variables, but no longer drops all the variables and re-creates them, instead it only touches user defined variables and leaves the form step variables alone. Since this is called after the steps have been sorted out, a complete view of left-over variables from earlier form definitions is available and those can be cleaned up safely now.
We can express everything now in terms of synchronize_for form_definition.
sergei-maertens
force-pushed
the
issue/5058-fix-deadlocks-and-races
branch
from
0a2fa51 to
d1fdb80
Compare
stevenbal
approved these changes
Jan 31, 2025
| @@ -308,15 +308,18 @@ class SubmissionValueVariable(models.Model): | |||
| help_text=_("The submission to which this variable value is related"), | |||
| on_delete=models.CASCADE, | |||
| ) | |||
| # XXX DeprecationWarning - this foreign key field actually doesn't serve any purpose | |||
There was a problem hiding this comment.
Should it be hidden from the admin interface?
There was a problem hiding this comment.
I intend to make a PR to the master branch only (so no backport) that removes the field, but I'm curious to see what will blow up because of it 😬
Otherwise it's probably a good suggestion to delete it, but afaik admins don't have access to this now anyway, only superusers.
Comment on lines
+61
to
+63
| Note that we *don't* remove variables related to other form definitions, as | ||
| multiple isolated transactions for different form definitions can happen at the | ||
| same time. |
There was a problem hiding this comment.
For my understanding: did the deadlock issue occur when two admins were modifying two different forms that shared a reusable form definition? Is this fixed now because it no longer does the recoupling via celery?
There was a problem hiding this comment.
I think the problem was that the task was scheduled to run 60s later and other write actions were happening and multiple transactions were trying to lock the same records in the table and competing for resources. Celery is definitely a big factor in this issue.
Laurens also saw issues when saving different forms in the admin in different tabs, so you didn't even need multiple different users doing stuff at the same time, but one customer was apparently building forms with 4 different people so that will have manifested much more obviously.
sergei-maertens
added a commit
that referenced
this pull request
Jan 31, 2025
**Removed the submission value variable recoupling** Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary. **Run form variable sync in same transaction** Offloading this to celery results in more, separate tasks each competing for database locks and can lead to integrity errors. Changing the business logic of form variable creation/update to make sure a FormDefinition write results in a sync action gives a single trigger for these kind of changes, and it's responsible for updating all the forms that are affected by it. Since multiple FormDefinition/FormStep writes happen in parallel because of parallel client requests, we can only update or create variables and can't delete variables that are not present in our current form definition, as they may belong to another request. This shouldn't immediately cause problems, as old, unused variables don't have an 'execution path'. We piggy back on the variables bulk update endpoint/transaction to clean these up, as that call is made after all the form step persistence succeeded so we know that everything is resolved by then. The left-over variables problem only exists when updating a form step to use a different form definition. It's not relevant when a form definition is updated through the admin or a form step is added to a form. **Add test for expected new behaviour of bulk variable update endpoint** The bulk update endpoint may no longer manage the variables for form steps, as that's taken care of by the form step endpoint now. But, user defined variables must still be managed. **Use the form variables bulk update for consistency** Instead of triggering celery tasks, perform the variable state synchronization for a form in the bulk update endpoint. This endpoint now still validates the state of all component/user defined variables, but no longer drops all the variables and re-creates them, instead it only touches user defined variables and leaves the form step variables alone. Since this is called after the steps have been sorted out, a complete view of left-over variables from earlier form definitions is available and those can be cleaned up safely now. Backport-of: #5064
sergei-maertens
added a commit
that referenced
this pull request
Jan 31, 2025
**Removed the submission value variable recoupling** Before this patch, we ran some reconciliation when a form is edited to ensure that existing SubmissionValueVariable instances don't have 'null' form_variable foreign keys because the FormVariable set of a form was updated after editing (a) step(s) - the idea here was to be eventually consistent. This turns out not to be necessary (if we can trust our test suite) because the load_submission_value_variables_state method on the Submission operates on the variable keys rather than the FK relation and is able to properly resolve everything. This is also the interface that developers should use when accessing the submission values and it appears to be done properly in the registration backends, otherwise tests would likely fail. This re-coupling was extended in #4900, after it was noticed in #4824 that the re-coupling didn't happen for other forms that use the same re-usable form definition. At the time, we didn't understand how this seemingly didn't cause issues or at least didn't result in issues being reported to us, but we can now conclude that it just wasn't a problem in the first place because the proper interfaces/service layer are/were being used and everything is done/reconciled in-memory when comparing/populating submission variables with form variables. A further cleanup step will then also be to remove this FK field from the submission value variable model, as it is not necessary. **Run form variable sync in same transaction** Offloading this to celery results in more, separate tasks each competing for database locks and can lead to integrity errors. Changing the business logic of form variable creation/update to make sure a FormDefinition write results in a sync action gives a single trigger for these kind of changes, and it's responsible for updating all the forms that are affected by it. Since multiple FormDefinition/FormStep writes happen in parallel because of parallel client requests, we can only update or create variables and can't delete variables that are not present in our current form definition, as they may belong to another request. This shouldn't immediately cause problems, as old, unused variables don't have an 'execution path'. We piggy back on the variables bulk update endpoint/transaction to clean these up, as that call is made after all the form step persistence succeeded so we know that everything is resolved by then. The left-over variables problem only exists when updating a form step to use a different form definition. It's not relevant when a form definition is updated through the admin or a form step is added to a form. **Add test for expected new behaviour of bulk variable update endpoint** The bulk update endpoint may no longer manage the variables for form steps, as that's taken care of by the form step endpoint now. But, user defined variables must still be managed. **Use the form variables bulk update for consistency** Instead of triggering celery tasks, perform the variable state synchronization for a form in the bulk update endpoint. This endpoint now still validates the state of all component/user defined variables, but no longer drops all the variables and re-creates them, instead it only touches user defined variables and leaves the form step variables alone. Since this is called after the steps have been sorted out, a complete view of left-over variables from earlier form definitions is available and those can be cleaned up safely now. Backport-of: #5064
e2e tests otherwise run against 3.0.0 which isn't compatible with the current state of this bugfix branch.
sergei-maertens
force-pushed
the
issue/5058-fix-deadlocks-and-races
branch
from
45d62e0 to
36e0a90
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #5058
Changes
This is essentially a rewrite to make everything consistent in a database transaction and not use Celery tasks which have race and concurrency issues.
Checklist
Check off the items that are completed or not relevant.
Impact on features
Release management
I have updated the translations assets (you do NOT need to provide translations)
./bin/makemessages_js.sh./bin/compilemessages_js.shDockerfile/scripts
./binfolderCommit hygiene