AdaQuo

Federated Learning Project, Group 13

Nguyen Thanh, Gamba Stefano, Carli Massimiliano, Tarantino Giovanbattista

We propose AdaQuo, a novel approach for sparse fine-tuning in Federated Learning designed to mitigate poor performances of highly non-IID setting. AdaQuo constructs a global sparsity mask by aggregating client-side binary masks. These masks are summed by the server as a form of voting, and a global mask is generated by selecting parameters that receive the highest number of votes. As training progresses, the server dynamically increases the vote threshold---based on observed average client drift---thereby adapting the level of sparsity over time. Experiments on CIFAR-100 demonstrate that AdaQuo outperforms decentralized sparse fine-tuned baselines in heterogeneous environments.

Note

We use Flower for our federated learning implementation, you can find the original flower documentation here, and a summary of the main Flower tools we used here.

How to Install

Using make

1. Ensure you have make installed on your system. If not, you can find installation guides for your operating system online (e.g., for Linux: sudo apt install build-essential, for macOS: brew install make, for Windows: choco install make).

2. Run the following command to set up the project environment:

   make install

   This will:

   • Create a new virtual enviroment (venv)
   • Install the required Python dependencies listed in requirements.txt.
   • Set up any additional configurations needed for the project.

Using venv

1. Ensure you have Python installed on your system. This project was developed using Python 3.10, which is the suggested version.

2. Create a virtual environment and install the required dependencies:

   python -m venv venv
   source venv/bin/activate  # On Windows use `venv\Scripts\activate`
   pip install -r requirements.txt

   This will:

   • Create a new virtual environment named venv.
   • Install the required Python dependencies listed in requirements.txt.
   • Set up the environment for the project.

Using conda

1. Ensure you have conda installed on your system.

2. Create a new conda environment and install the required dependencies:

   conda create --name fl-g13 python=3.10
   conda activate fl-g13
   pip install -r requirements.txt

   This will:

   • Create a new conda environment named fl-g13 with Python 3.10.
   • Install the required Python dependencies listed in requirements.txt.
   • Set up the environment for the project.

Project Organization

All Flower simulations are run exclusively through the Jupyter notebooks located in the notebooks/ directory. The Python source code modules can be found within fl_g13/. For the federated learning implementation please refer to fl_g13/fl_pytorch and read FLOWER.md to understand how Flower works

├── LICENSE            <- Open-source license if one is chosen
├── Makefile           <- Makefile with convenience commands for environment setup and notebook export
├── README.md          <- The top-level README for developers using this project.
├── FLOWER.md          <- A guide describing the concepts of Flower necessary to understand our project.
├── data
│   ├── external       <- Data from third party sources.
│   ├── interim        <- Intermediate data that has been transformed.
│   ├── processed      <- The final, canonical data sets for modeling.
│   └── raw            <- The original, immutable data dump.
│
├── models             <- Checkpoints produced during training
│
├── notebooks          <- Jupyter notebooks.
│
├── pyproject.toml     <- Project configuration file with package metadata for 
│                         fl_g13
│
├── requirements.txt   <- The requirements packages for reproducing experiments
│
└── fl_g13   <- Source code of this project.
    │
    ├── __init__.py             <- Makes fl_g13 a Python module
    ├── config.py               <- Store useful variables and configuration
    ├── dataset.py              <- Scripts to download and manage data
    ├── architectures/          <- Classes for models architectures
    │   └── BaseDino.py
    ├── editing/                <- Code for model editing
    │   ├── fisher.py
    │   ├── masking.py
    │   └── sparseSGDM.py
    ├── fl_pytorch/             <- Code for Flower federated learning apps
    │   ├── client_app.py
    │   ├── client.py
    │   ├── datasets.py
    │   ├── DynamicQuorumClient.py
    │   ├── DynamicQuorumStrategy.py
    │   ├── model.py
    │   ├── server_app.py
    │   ├── strategy.py
    │   ├── task.py
    │   ├── utils.py
    │   └── editing/            <- Code for federated mask computation
    │       └── centralized_mask.py
    └── modeling/               <- Code for train, test, save and load models
        ├── eval.py
        ├── load.py
        ├── train.py
        └── utils.py

---

How to contribute

Commit Message and Branch Naming Rules

1. Commit Message Format

   • Use the following format for commit messages:

     <type>: <short description>
     

   • Types:
     • feat: A new feature
     • fix: A bug fix
     • docs: Documentation changes
     • style: Code style changes (formatting, missing semicolons, etc.)
     • refactor: Code refactoring without adding features or fixing bugs
   • Example:

     feat: add data preprocessing pipeline
     fix: resolve issue with model training script
     

2. Branch Naming Convention

   • Use the following format for branch names:

     <type>-<short-description>-<initials>
     

   • Types:
     • feat: For new features
     • fix: For bug fixes
     • docs: For documentation updates
     • refactor: For refactoring tasks
   • Example:

     feat-add-preprocessing-pipeline-pjb
     fix-model-training-bug-mc
     

Jupyter Notebook Usage

1. Notebook Organization

   • Notebooks must be stored in the notebooks/ directory.

   • Naming convention: PHASE.NOTEBOOK-INITIALS-DESCRIPTION.ipynb

     Example: 0.01-pjb-data-source-1.ipynb

     • PHASE codes:
       • 0 – Data exploration
       • 1 – Data cleaning & feature engineering
       • 2 – Visualization
       • 3 – Modeling
       • 4 – Publication
     • INITIALS – Your initials; helps identify the author and avoid conflicts.
     • DESCRIPTION – Short, clear description of the notebook's purpose.

Code Reusability & Refactoring Regulation

1. Refactor Shared Code into Modules

   • Store reusable code in the fl_g13 package.
   • Add the following cell at the top of each notebook:

   %load_ext autoreload
   %autoreload 2

Code Review & Version Control Regulation

1. Use nbautoexport Tool

   • Install with:

   nbautoexport install
   nbautoexport configure notebooks

   • Then, anytime you want to export a notebook to a Python script, run:

   nbautoexport export notebooks/<notebook_name>.ipynb

   • Equivalently, you can also run:

   make export

   for convenience — this will export all notebooks in the notebooks/ folder automatically.

   • 💡 Pro Tip: Add the following line at the end of each notebook to automatically export it every time you run it:

   !nbautoexport export notebooks/<notebook_name>.ipynb

   or just do

   !make export

2. Ensure Reviewability

   • Commit both .ipynb files and their exported .py versions to version control.

(PyCharm only) Use a Git Hook or File Watcher

You can set up PyCharm to run nbconvert (or even nbautoexport export) every time a file is saved or committed.

PyCharm File Watcher

1. Go to Settings > Tools > File Watchers
2. Add a new watcher with the following configuration:

• File Type: Jupyter Notebook (*.ipynb)

• Scope: Current project

• Program: Your Python interpreter path (e.g., python)

• Arguments:

  -m nbautoexport export $FileDir$

• Working Directory: $FileDir$