recalc user statistics using sql

[arily]

Describe your changes

• recalc user statistics using sql
  we are running this code for few months, and eliminated few bugs during testing.

• atomic updates on beatmap statistics
  I don't feel boatmap play count needs to be accurate but if needed we can also count from submitted scores.

benefits

• will dramatically speed up score submission.
  for our server (ppy.sb), in extreme cases (multiplay, multiple user, heavy user), user will experience score submission speed up from few minutes to seconds.
  I believe this change could also in theory make removing the implemented global submission lock possible, therefore further increase parallel performance for multiple user submitting scores.
  Such lock is for prevent increasing user stat twice for same score, bpy will / should check checksum to not allow multiple same score inserted into db.

• handles db changes naturally.
  other than using our recalc utils (mentioned below), all stats will be updated after user submit any score, if proceed any database modifications eg. delete score, rank custom maps, recalc score stats.

I believe some sort of indexes are required for the calculation to run smoothly.
we have 2 compound indexes:

• map-leaderboard (map_md5, mode, status)
• user-best (userid, mode, status, pp)

these are for both recalc and our frontend.

CREATE INDEX `map-leaderboard` ON `scores` (`map_md5`,`mode`,`status`);--> statement-breakpoint
CREATE INDEX `user-best` ON `scores` (`userid`,`mode`,`status`,`pp`);--> statement-breakpoint

Related Issues / Projects

we also have offline recalc utility if anything messed up:
https://github.com/ppy-sb/nyamatrix

Checklist

• I've manually tested my code

[marcandreher]

Indeed good stuff, but I prefer to not change the db structure atm, or there is a auto migration that adds that?

[arily]

Indeed good stuff, but I prefer to not change the db structure atm, or there is a auto migration that adds that?

No db structures has been changed. Just add few indexes speed up queries.
The mentioned sql is all we needed.