Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#5077] Improve performance in loading logic rules in admin #5078

Merged

Conversation

vaszig
Copy link
Contributor

@vaszig vaszig commented Feb 4, 2025

Closes #5077

Changes

  • Logic rules list in the admin was really slow as the logic rules increase because of the same queries happening for the form and the form step. Some joins were added in order to reduce the amount of the queries.

Checklist

Check off the items that are completed or not relevant.

  • Impact on features

    • Checked copying a form
    • Checked import/export of a form
    • Config checks in the configuration overview admin page
    • Problem detection in the admin email digest is handled
  • Release management

    • I have labelled the PR as "needs-backport" accordingly
  • I have updated the translations assets (you do NOT need to provide translations)

    • Ran ./bin/makemessages_js.sh
    • Ran ./bin/compilemessages_js.sh
  • Dockerfile/scripts

    • Updated the Dockerfile with the necessary scripts from the ./bin folder
  • Commit hygiene

    • Commit messages refer to the relevant Github issue
    • Commit messages explain the "why" of change, not the how

@vaszig vaszig marked this pull request as draft February 4, 2025 13:04
@vaszig
Copy link
Contributor Author

vaszig commented Feb 4, 2025

Logic rules

Before:

logic-rules-performance-old

After (select_related):

logic-rules-performance-new

After (prefetch_related):

logic-rules-performance-prefetch

Variables

Before:

variables-list-performance-old

After (prefetch_related):

variables-list-performance-prefetch

Copy link

codecov bot commented Feb 4, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.73%. Comparing base (7c10f99) to head (7a56bce).
Report is 6 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #5078   +/-   ##
=======================================
  Coverage   96.73%   96.73%           
=======================================
  Files         771      771           
  Lines       26597    26619   +22     
  Branches     3460     3463    +3     
=======================================
+ Hits        25729    25751   +22     
  Misses        606      606           
  Partials      262      262           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@vaszig vaszig marked this pull request as ready for review February 4, 2025 13:16
Copy link
Member

@sergei-maertens sergei-maertens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the variables bulk/list endpoint may have similar issues, iirc that one also showed up in APM. You can give it the same treatment (notably the FormVariable.form and FormVariable.form_definition foreign keys may cause n+1 queries here)

Comment on lines 595 to 596
# See github issue https://github.com/open-formulieren/open-forms/issues/5077
logic_rules = form.formlogic_set.select_related(
"form", "trigger_from_step__form"
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you tried prefetch_related too instead of select_related?

Select related performs an inner join and results in more data going over the wire between database and application, especially here because the joined form will always be same as the form we already retrieved (meaning that 55 logic rules result in 55 times the same form being joined).

For the trigger_from_step, I expect a similar situation - it's more likely that multiple different logic rules have the same trigger_from_step, so joining can be slower than doing a prefetch. And of course, the related form of the trigger_from_step has the same problem as the form FK of the logic rule.

I think that all three of these fields (form, trigger_from_step and trigger_from_step__form) might benefit from a prefetch - it will result in a couple more queries, but it will be a fixed amount and the data going over the wire could be a lot less, so I'm interested in the performance profile of that approach.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't notice any difference concerning the update endpoints for both logic-rules and variables (tried with both joins and prefetch)

Copy link
Member

@sergei-maertens sergei-maertens Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You specifically mention the update endpoints, but I'm interested in the list endpoints in particular :P

However, looking at your screenshots, prefetch_related seems to be a clear winner!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the variables bulk/list endpoint may have similar issues, iirc that one also showed up in APM. You can give it the same treatment (notably the FormVariable.form and FormVariable.form_definition foreign keys may cause n+1 queries here)

This is why I mentioned it.I understood that we wanted to see what's going on with these as well. But clearly wrong place to write this, it should be as a reply to the above.

@vaszig vaszig force-pushed the fix/5077-improve-performance-in-logic-rules-list-qs branch from 45b71ff to 46103a7 Compare February 5, 2025 14:39
@vaszig vaszig marked this pull request as draft February 5, 2025 14:49
@vaszig vaszig marked this pull request as ready for review February 5, 2025 14:57
Had to apply prefetch here as too many queries (n+1) were taking place
for the variables and logic-rules list endpoints and affected the
performance.
@vaszig vaszig force-pushed the fix/5077-improve-performance-in-logic-rules-list-qs branch from 46103a7 to 7a56bce Compare February 6, 2025 12:27
Copy link
Member

@sergei-maertens sergei-maertens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚡ LETSGO

@sergei-maertens sergei-maertens merged commit 9894ae6 into master Feb 6, 2025
32 checks passed
@sergei-maertens sergei-maertens deleted the fix/5077-improve-performance-in-logic-rules-list-qs branch February 6, 2025 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve performance of loading logic rules in admin
2 participants