Initial commit

.gitignore

@@ -0,0 +1,4 @@
+*.npy
+__pycache__/
+checkpoints/
+data/

LICENSE

@@ -0,0 +1,24 @@
+MIT License
+
+Portions of this software are copyright of their respective authors and released
+under the MIT license:
+- Mammoth, Copyright  2020 Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, Simone Calderara
+- SimSiam, Copyright  2020 PatrickHua
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,76 @@
+# Representational Continuity </br> for Unsupervised Continual Learning
+This is the *Pytorch Implementation* for the paper Representational Continuity for Unsupervised Continual Learning
+
+**Authors**: [Divyam Madaan](https://dmadaan.com/), [Jaehong Yoon](https://jaehong31.github.io), [Yuanchun Li](http://yuanchun-li.github.io), [Yunxin Liu](https://yunxinliu.github.io), [Sung Ju Hwang](http://sungjuhwang.com/)
+
+## Abstract
+<img align="middle" width="700" src="https://github.com/divyam3897/UCL/blob/main/concept.png">
+
+Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent advances in continual learning are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on *unsupervised continual learning (UCL)*, where we learn the feature representations on an unlabelled sequence of tasks and show that the reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better to out-of-distribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations.
+Additionally, we propose Lifelong Unsupervised Mixup (Lump), a simple yet effective technique that leverages the interpolation between the current task and previous tasks' instances to alleviate catastrophic forgetting for unsupervised representations.
+
+__Contribution of this work__
+- We attempt to bridge the gap between continual learning and representation learning and tackle the two important problems of continual learning with unlabelled data and representation learning on a sequence of tasks.
+- Systematic quantitative analysis show that UCL achieves better performance over SCL with significantly lower catastrophic forgetting on Sequential CIFAR-10, CIFAR-100 and Tiny-ImageNet. Additionally, we evaluate on out of distribution tasks and few-shot continually learning demonstrating the expressive power of unsupervised representations. 
+- We provide visualization of the representations and loss landscapes that UCL learns discriminative, human perceptual patterns and achieves a flatter and smoother loss landscape. Furthermore, we propose Lifelong Unsupervised Mixup (Lump) for UCL, which effectively alleviates catastrophic forgetting and provides better qualitative interpretations. 
+
+
+## Prerequisites
+```
+$ pip install -r requirements.txt
+```
+
+## Run
+* __Split CIFAR-10__ experiment with SimSiam
+```
+$ python main.py --data_dir ../Data/ --log_dir ../logs/ -c configs/simsiam_c10.yaml --ckpt_dir ./checkpoints/cifar10_results/ --hide_progress
+```
+
+* __Split CIFAR-100__ experiment with SimSiam
+
+```
+$ python main.py --data_dir ../Data/ --log_dir ../logs/ -c configs/simsiam_c100.yaml --ckpt_dir ./checkpoints/cifar100_results/ --hide_progress
+```
+
+* __Split Tiny-ImageNet__ experiment with SimSiam
+
+```
+$ python main.py --data_dir ../Data/ --log_dir ../logs/ -c configs/simsiam_tinyimagenet.yaml --ckpt_dir ./checkpoints/tinyimagenet_results/ --hide_progress
+```
+
+* __Split CIFAR-10__ experiment with BarlowTwins
+```
+$ python main.py --data_dir ../Data/ --log_dir ../logs/ -c configs/barlow_c10.yaml --ckpt_dir ./checkpoints/cifar10_results/ --hide_progress
+```
+
+* __Split CIFAR-100__ experiment with BarlowTwins
+
+```
+$ python main.py --data_dir ../Data/ --log_dir ../logs/ -c configs/barlowm_c100.yaml --ckpt_dir ./checkpoints/cifar100_results/ --hide_progress
+```
+
+* __Split Tiny-ImageNet__ experiment with BarlowTwins
+
+```
+$ python main.py --data_dir ../Data/ --log_dir ../logs/ -c configs/barlowm_tinyimagenet.yaml --ckpt_dir ./checkpoints/tinyimagenet_results/ --hide_progress
+```
+
+## Contributing
+We'd love to accept your contributions to this project. Please feel free to open an issue, or submit a pull request as necessary. If you have implementations of this repository in other ML frameworks, please reach out so we may highlight them here.
+
+## Acknowledgment
+The code is build upon [aimagelab/mammoth](https://github.com/aimagelab/mammoth) and [PatrickHua/SimSiam](https://github.com/PatrickHua/SimSiam)
+
+## Citation
+If you found the provided code useful, please cite our work.
+
+```bibtex
+@inproceedings{
+  madaan2022rethinking,
+  title={Representational Continuity for Unsupervised Continual Learning},
+  author={Divyam Madaan and Jaehong Yoon and Yuanchun Li and Yunxin Liu and Sung Ju Hwang},
+  booktitle={International Conference on Learning Representations},
+  year={2022},
+  url={https://openreview.net/forum?id=9Hrka5PA7LW}
+}
+```

arguments.py

@@ -0,0 +1,112 @@
+import argparse
+import os
+import torch
+
+import numpy as np
+import torch
+import random
+
+import re 
+import yaml
+
+import shutil
+import warnings
+
+from datetime import datetime
+
+
+class Namespace(object):
+    def __init__(self, somedict):
+        for key, value in somedict.items():
+            assert isinstance(key, str) and re.match("[A-Za-z_-]", key)
+            if isinstance(value, dict):
+                self.__dict__[key] = Namespace(value)
+            else:
+                self.__dict__[key] = value
+
+    def __getattr__(self, attribute):
+
+        raise AttributeError(f"Can not find {attribute} in namespace. Please write {attribute} in your config file(xxx.yaml)!")
+
+
+def set_deterministic(seed):
+    # seed by default is None 
+    if seed is not None:
+        print(f"Deterministic with seed = {seed}")
+        random.seed(seed) 
+        np.random.seed(seed) 
+        torch.manual_seed(seed)
+        torch.cuda.manual_seed(seed)
+        torch.backends.cudnn.deterministic = True 
+        torch.backends.cudnn.benchmark = False 
+
+def get_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-c', '--config-file', required=True, type=str, help="xxx.yaml")
+    parser.add_argument('--debug', action='store_true')
+    parser.add_argument('--debug_subset_size', type=int, default=8)
+    parser.add_argument('--download', action='store_true', help="if can't find dataset, download from web")
+    parser.add_argument('--data_dir', type=str, default=os.getenv('DATA'))
+    parser.add_argument('--log_dir', type=str, default=os.getenv('LOG'))
+    parser.add_argument('--ckpt_dir', type=str, default=os.getenv('CHECKPOINT'))
+    parser.add_argument('--ckpt_dir_1', type=str, default=os.getenv('CHECKPOINT'))
+    parser.add_argument('--device', type=str, default='cuda'  if torch.cuda.is_available() else 'cpu')
+    parser.add_argument('--eval_from', type=str, default=None)
+    parser.add_argument('--hide_progress', action='store_true')
+    parser.add_argument('--cl_default', action='store_true')
+    parser.add_argument('--validation', action='store_true',
+                        help='Test on the validation set')
+    parser.add_argument('--ood_eval', action='store_true',
+                        help='Test on the OOD set')
+    args = parser.parse_args()
+
+
+    with open(args.config_file, 'r') as f:
+        for key, value in Namespace(yaml.load(f, Loader=yaml.FullLoader)).__dict__.items():
+            vars(args)[key] = value
+
+    if args.debug:
+        if args.train: 
+            args.train.batch_size = 2
+            args.train.num_epochs = 1
+            args.train.stop_at_epoch = 1
+        if args.eval: 
+            args.eval.batch_size = 2
+            args.eval.num_epochs = 1 # train only one epoch
+        args.dataset.num_workers = 0
+
+
+    assert not None in [args.log_dir, args.data_dir, args.ckpt_dir, args.name]
+
+    args.log_dir = os.path.join(args.log_dir, 'in-progress_'+datetime.now().strftime('%m%d%H%M%S_')+args.name)
+
+    os.makedirs(args.log_dir, exist_ok=False)
+    print(f'creating file {args.log_dir}')
+    os.makedirs(args.ckpt_dir, exist_ok=True)
+
+    shutil.copy2(args.config_file, args.log_dir)
+    set_deterministic(args.seed)
+
+
+    vars(args)['aug_kwargs'] = {
+        'name':args.model.name,
+        'image_size': args.dataset.image_size
+    }
+    vars(args)['dataset_kwargs'] = {
+        # 'name':args.model.name,
+        # 'image_size': args.dataset.image_size,
+        'dataset':args.dataset.name,
+        'data_dir': args.data_dir,
+        'download':args.download,
+        'debug_subset_size': args.debug_subset_size if args.debug else None,
+        # 'drop_last': True,
+        # 'pin_memory': True,
+        # 'num_workers': args.dataset.num_workers,
+    }
+    vars(args)['dataloader_kwargs'] = {
+        'drop_last': True,
+        'pin_memory': True,
+        'num_workers': args.dataset.num_workers,
+    }
+
+    return args

augmentations/__init__.py

@@ -0,0 +1,23 @@
+from .simsiam_aug import SimSiamTransform
+from .eval_aug import Transform_single
+
+
+def get_aug(name='simsiam', image_size=224, train=True, train_classifier=None):
+    if train==True:
+        augmentation = SimSiamTransform(image_size)
+    elif train==False:
+        if train_classifier is None:
+            raise Exception
+        augmentation = Transform_single(image_size, train=train_classifier)
+    else:
+        raise Exception
+
+    return augmentation
+
+
+
+
+
+
+
+

augmentations/eval_aug.py

@@ -0,0 +1,26 @@
+from torchvision import transforms
+from PIL import Image
+
+# imagenet_norm = [[0.485, 0.456, 0.406],[0.229, 0.224, 0.225]]
+imagenet_norm = [[0.4914, 0.4822, 0.4465],[0.2470, 0.2435, 0.2615]]
+
+class Transform_single():
+    def __init__(self, image_size, train, normalize=imagenet_norm):
+        if train == True:
+            self.transform = transforms.Compose([
+                transforms.RandomResizedCrop(image_size, scale=(0.08, 1.0), ratio=(3.0/4.0,4.0/3.0), interpolation=Image.BICUBIC),
+                # transforms.RandomCrop(image_size, padding=4),
+                transforms.RandomHorizontalFlip(),
+                transforms.ToTensor(),
+                transforms.Normalize(*normalize)
+            ])
+        else:
+            self.transform = transforms.Compose([
+                # transforms.Resize(int(image_size*(8/7)), interpolation=Image.BICUBIC), # 224 -> 256 
+                # transforms.CenterCrop(image_size),
+                transforms.ToTensor(),
+                transforms.Normalize(*normalize)
+            ])
+
+    def __call__(self, x):
+        return self.transform(x)