Add tico model

[IgorSusmelj]

Changes

• Add TiCo model code for ImageNet benchmark
• Run experiments and add results to readme and benchmarks tab in the docs

Note that I'll have to rebase this PR on master as quite some time passed since I started this.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (67dc269) 85.49% compared to head (d97102b) 85.50%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1398   +/-   ##
=======================================
  Coverage   85.49%   85.50%           
=======================================
  Files         135      135           
  Lines        5655     5657    +2     
=======================================
+ Hits         4835     4837    +2     
  Misses        820      820           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[IgorSusmelj]

This will be a more tough one. I was running several experiments and the loss is going up and accuracy stays down. Somehow the training seems unstable. I varied some of the hyperparameters but didn't get it running.

[IgorSusmelj]

The training on ImageNet does not work as expected. We did several modifications to the loss but no matter what we do it ends up with having the loss saturing to the max value and the accuracy staying at 0%.

Epoch 3:  63%|██████▎   | 1583/2502 [14:05<08:10,  1.87it/s, v_num=0, train_loss=8.000, val_online_cls_loss=140.0, val_online_cls_top1=0.00106, val_online_cls_top5=0.00414]
...
Epoch 5:  10%|▉         | 238/2502 [02:11<20:47,  1.81it/s, v_num=0, train_loss=8.000, val_online_cls_loss=569.0, val_online_cls_top1=0.00094, val_online_cls_top5=0.00438]

Things we tried:

• Detaching the auxiliary matrix to prevent backprop through it (suggestion from Guarin):

  lightly/lightly/loss/tico_loss.py

  Line 104 in efc559c

  B = torch.mm(z_a.T, z_a) / z_a.shape[0]

• Toggling gather_distributed (suggestion from Guarin): but both, with and without we get the same results
• With the new default settings (this PR) the loss saturates at 8.0
• Variations of the hyperparameters (beta, rho of the loss and learning rate)
  

I'll do one more run with more in depth logs to better isolate which parts of the loss go out of control :)

[guarin]

@IgorSusmelj could you quickly summarize the changes you tried?

[IgorSusmelj]

The latest changes seem promising:

[guarin]

LGTM! Let's update the benchmarks once we figure out what is wrong with the model.

I'll also create follow-up issues to update BYOL and MoCo to use the correct backbone for evaluation.

[guarin]

Interesting that the linear accuracy is so low. In the paper they report linear accuracy that is similar to the other methods.

[IgorSusmelj]

100 epochs vs 1000 epochs could be the reason for that