TrueSkill comments

[Duane321]

Hey @liux3372 , I merged your branch because I think it's got everything it needs and I have no objections. Well done in fact - I can see a lot of work went into it.

Here are some comments:

• For warcraft3, is the DataProcessor necessary? Looks like we only use the DataSpliter (btw, it should be 'DataSplitter' - two t's) for that.
• It's interesting that warcraft3 has a AUC of 63% and boxing has an AUC of 74.6%. The calibration plots would suggest warcraft 3 should be better. What do you think? You reasonable sure these numbers are correct?
• Can you update the README.md with instructions on how someone else can run through these notebooks? I'd make sure to write them as you run through them yourself.
• Can you delete anything that you think is unused? E.g. maybe the Warcraft3 DataProcessor. But also outdated graphics or whatever else. I'm not sure if there is a lot of delete-able stuff - I'm just trying to cut down on what I have to look.

[liux3372]

reference Commit:
995bc7e

d7a7346ac032fe28c7bc78ce1e0e2f32f1e26e02
Hey DJ,
Thanks for your review and comments.

I have updated a README.md as a description of each file and folder.

I have also fixed the DataSplitter typos and update all the occurrences.

And I’ve deleted the unused code like Warcraft3 DataProcessor and unused plots.

Btw, I’ll leave the clean up work of true_skill_simulated.ipynb and trueskill_simulation.py to you.

Regarding the better calibration and lower AUC for warcraft 3 compared to boxing:

I have looked up the code and they should be correct. I think the reasons why this happens are

• Warcraft 3 has higher volatility on the game results and players skills, which makes it harder to predict test set of Warcraft 3 since players evolve rapidly and this game is in high uncertainty.
• Warcraft3 has multiple matches in each event between the two players and we did a random permutation of those matches, whereas boxing has a clear 1v1 match every time.
• Warcraft3 has over 18000 games in the test set while boxing has about 4000 games in the test set.
• Please note that tennis, even though it has clear 1v1 format and over 26000 games in the test set, has AUC roughly 0.70.

And if we compare the hyper parameters of these two:

Boxing: optimal_gamma = 0.0738 optimal_sigma = 0.6004 optimal_beta = 0.2843
Warcraft3: optimal_gamma = 0.1000 optimal_sigma = 2.5166 optimal_beta = 1.7098

Warcraft 3 has higher baseline uncertainty (a higher sigma) and more per-game performance variation (a higher beta) This makes sense for Warcraft 3 where individual game outcomes are more volatile, while boxing results tend to be more consistent with fighter skill levels.

So while the model can accurately predict the probabilities of outcomes in Warcraft 3 (good calibration), it has a harder time strictly ranking who will beat who (lower AUC) due to the inherent volatility and the way we handle multi-game matches.
I’ll have a short description on the post to explain this observation.

Let me know if you have any question, please.