Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement W-MSE Loss and Transform #1459

Merged
merged 2 commits into from
Dec 19, 2023
Merged

Implement W-MSE Loss and Transform #1459

merged 2 commits into from
Dec 19, 2023

Conversation

johnsutor
Copy link
Contributor

This pull request implements the W-MSE Transform and W-MSE Loss as proposed in the paper https://arxiv.org/pdf/2007.06346.pdf. The code largely comes from the original repository https://github.com/htdt/self-supervised, preserving the default hyperparameters with some minor changes to make it compatible with lightly. This feature was recommended in this issue: #1424 (comment)

@johnsutor
Implement W-MSE Loss
acc904c 
Implement W-MSE Transform
@codecov Codecov
Copy link

codecov bot commented Dec 17, 2023 •
edited
Loading

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (66ad1b4) 85.24% compared to head (8b6e5b5) 85.52%.
Report is 7 commits behind head on master.

Files Patch % Lines
lightly/loss/wmse_loss.py 95.71% 3 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #1459      +/-   ##
==========================================
+ Coverage   85.24%   85.52%   +0.28%     
==========================================
  Files         130      135       +5     
  Lines        5517     5653     +136     
==========================================
+ Hits         4703     4835     +132     
- Misses        814      818       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@philippmwirth
Copy link
Contributor

Hi @johnsutor, thank you very much for your contribution. Someone from our team will have a look asap.

philippmwirth
philippmwirth suggested changes Dec 18, 2023
View reviewed changes
Copy link
Contributor

@philippmwirth philippmwirth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the changes, @johnsutor! The PR looks great so far. I have left a few comments on things I think we should change before merging. If you have any questions or require help updating the PR please let me know 🙂

@johnsutor
Add ValueError when using embedding_dim > w_size / 2 in W-MSE Loss
8b6e5b5 
Remove unecessary print statements in W-MSE Loss tests
Add additional documentation to forward pass of W-MSE loss
Add Gaussian Blur to W-MSE transform
philippmwirth
philippmwirth approved these changes Dec 19, 2023
View reviewed changes
Copy link
Contributor

@philippmwirth philippmwirth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good now! I'll quickly do a sanity check on Cifar10 and then we're good to merge. Thank you 🙂

@philippmwirth
Copy link
Contributor

The results look promising, after 3 epochs we get a kNN accuracy of over 50% with num_samples=4 and SimCLR projection head.

------------------------------------------------------------------------------------------
| Model         | Batch Size | Epochs |  KNN Test Accuracy |       Time | Peak GPU Usage |
------------------------------------------------------------------------------------------
| WMSE          |        512 |      3 |              0.519 |   12.9 Min |      9.4 GByte |
------------------------------------------------------------------------------------------

Will merge this now and leave the proper benchmarking for later. @guarin FYI.

@philippmwirth philippmwirth merged commit 492de87 into lightly-ai:master Dec 19, 2023
6 checks passed
@philippmwirth philippmwirth mentioned this pull request Dec 19, 2023
Merged
This was referenced Jan 2, 2024
Open
Open
guarin pushed a commit that referenced this pull request Jan 19, 2024
@johnsutor @guarin
Implement W-MSE Loss and Transform (#1459)
86d6497 
* Implement W-MSE Loss
* Implement W-MSE Transform
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@philippmwirth philippmwirth philippmwirth approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@johnsutor @philippmwirth