From 0ee524bbc5ae0ca17ba5e381940c1a5988d059e5 Mon Sep 17 00:00:00 2001 From: philippmwirth Date: Wed, 22 Nov 2023 10:29:08 +0000 Subject: [PATCH] Re Guarin and Jeremy --- README.md | 48 ++++++---------------- docs/source/getting_started/benchmarks.rst | 7 ++-- 2 files changed, 16 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index 2f37d21bb..81b46a39d 100644 --- a/README.md +++ b/README.md @@ -268,6 +268,16 @@ tuned for maximum accuracy. For detailed results and more information about the ### ImageNet1k +[ImageNet1k benchmarks](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#imagenet1k) + +> **Note**: Evaluation settings are based on these papers: +> * Linear: [SimCLR](https://arxiv.org/abs/2002.05709) +> * Finetune: [SimCLR](https://arxiv.org/abs/2002.05709) +> * KNN: [InstDisc](https://arxiv.org/abs/1805.01978) +> +> See the [benchmarking scripts](./benchmarks/imagenet/resnet50/) for details. + + | Model | Backbone | Batch Size | Epochs | Linear Top1 | Finetune Top1 | kNN Top1 | Tensorboard | Checkpoint | |----------------|----------|------------|--------|-------------|---------------|----------|-------------|------------| | BarlowTwins | Res50 | 256 | 100 | 62.9 | 72.6 | 45.6 | [link](https://tensorboard.dev/experiment/NxyNRiQsQjWZ82I9b0PvKg/) | [link](https://lightly-ssl-checkpoints.s3.amazonaws.com/imagenet_resnet50_barlowtwins_2023-08-18_00-11-03/pretrain/version_0/checkpoints/epoch%3D99-step%3D500400.ckpt) | @@ -286,46 +296,12 @@ better results for smaller batch sizes. See Appendix B.1 in the [SimCLR paper](h ### Imagenette -| Model | Backbone | Batch Size | Epochs | kNN Top1 | -|-------------|----------|------------|--------|----------| -| BarlowTwins | Res18 | 256 | 800 | 0.852 | -| BYOL | Res18 | 256 | 800 | 0.887 | -| DCL | Res18 | 256 | 800 | 0.861 | -| DCLW | Res18 | 256 | 800 | 0.865 | -| DINO | Res18 | 256 | 800 | 0.888 | -| FastSiam | Res18 | 256 | 800 | 0.873 | -| MAE | ViT-S | 256 | 800 | 0.610 | -| MSN | ViT-S | 256 | 800 | 0.828 | -| Moco | Res18 | 256 | 800 | 0.874 | -| NNCLR | Res18 | 256 | 800 | 0.884 | -| PMSN | ViT-S | 256 | 800 | 0.822 | -| SimCLR | Res18 | 256 | 800 | 0.889 | -| SimMIM | ViT-B32 | 256 | 800 | 0.343 | -| SimSiam | Res18 | 256 | 800 | 0.872 | -| SwaV | Res18 | 256 | 800 | 0.902 | -| SwaVQueue | Res18 | 256 | 800 | 0.890 | -| SMoG | Res18 | 256 | 800 | 0.788 | -| TiCo | Res18 | 256 | 800 | 0.856 | -| VICReg | Res18 | 256 | 800 | 0.845 | -| VICRegL | Res18 | 256 | 800 | 0.778 | +[Imagenette benchmarks](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#imagenette) ### CIFAR-10 -| Model | Backbone | Batch Size | Epochs | KNN Top1 | -|-------------|----------|------------|--------|----------| -| BarlowTwins | Res18 | 512 | 800 | 0.859 | -| BYOL | Res18 | 512 | 800 | 0.910 | -| DCL | Res18 | 512 | 800 | 0.874 | -| DCLW | Res18 | 512 | 800 | 0.871 | -| DINO | Res18 | 512 | 800 | 0.848 | -| FastSiam | Res18 | 512 | 800 | 0.902 | -| Moco | Res18 | 512 | 800 | 0.899 | -| NNCLR | Res18 | 512 | 800 | 0.892 | -| SimCLR | Res18 | 512 | 800 | 0.879 | -| SimSiam | Res18 | 512 | 800 | 0.904 | -| SwaV | Res18 | 512 | 800 | 0.884 | -| SMoG | Res18 | 512 | 800 | 0.800 | +[CIFAR-10 benchmarks](https://docs.lightly.ai/self-supervised-learning/getting_started/benchmarks.html#cifar-10) ## Terminology diff --git a/docs/source/getting_started/benchmarks.rst b/docs/source/getting_started/benchmarks.rst index f9a2eabe6..4cc83a2ca 100644 --- a/docs/source/getting_started/benchmarks.rst +++ b/docs/source/getting_started/benchmarks.rst @@ -47,13 +47,13 @@ Imagenet100 - `Dataset `_ - :download:`Code ` -Imagenet100 is a subset of the popular ImageNet-1k dataset. It consists of 100 classes +Imagenet100 is a subset of the popular ImageNet1k dataset. It consists of 100 classes with 1300 training and 50 validation images per class. We train the self-supervised models from scratch on the training data. At the end of every epoch we embed all training images and use the features for a kNN classifier with k=20 on the test set. The reported kNN Top 1 is the max accuracy over all epochs the model reached. All experiments use the same ResNet-18 backbone and -the default ImageNet-1k training parameters from the respective papers. +the default ImageNet1k training parameters from the respective papers. The following experiments have been conducted on a system with single A6000 GPU. Training a model takes between 20 and 30 hours, including kNN evaluation. @@ -79,13 +79,14 @@ Imagenette - `Dataset `_ - :download:`Code ` +Imagenette is a subset of 10 easily classified classes from ImageNet. For our benchmarks we use the 160px version of the Imagenette dataset and resize the input images to 128 pixels during training. We train the self-supervised models from scratch on the training data. At the end of every epoch we embed all training images and use the features for a kNN classifier with k=20 on the test set. The reported kNN Top 1 is the max accuracy over all epochs the model reached. All experiments use the same ResNet-18 backbone and -the default ImageNet-1k training parameters from the respective papers. +the default ImageNet1k training parameters from the respective papers. The following experiments have been conducted on a system with single A6000 GPU. Training a model takes three to five hours, including kNN evaluation.