[RFC] Differential Privacy Support in FL-bench

[KarhouTam]

RFC: Differential Privacy Support in FL-bench

Summary

This RFC is to collect suggestions, requirements, and comments from the community regarding the integration of differential privacy (DP) features into FL-bench. Differential privacy is a key technique for enhancing privacy guarantees in federated learning (FL) systems. We aim to design, implement, and maintain DP-related utilities and algorithms in FL-bench, and your feedback will help shape the direction and priorities of this work.

Motivation

• Privacy is a critical concern in FL, especially for sensitive data.
• Differential privacy is a widely recognized standard for privacy-preserving machine learning.
• There is growing interest in benchmarking and comparing DP-FL algorithms.

Scope

• DP mechanisms for client updates (e.g., noise addition, clipping)
• Integration with existing FL methods (FedAvg, FedProx, etc.)
• Configurable privacy budgets and accounting
• Utility functions and APIs for DP
• Evaluation and benchmarking tools for DP-FL
• Documentation and best practices

Request for Comments

• What DP mechanisms or algorithms are most important to support?
• What are your use cases or requirements for DP in FL-bench?
• Are there existing libraries or standards we should integrate with?
• What metrics or benchmarks are useful for DP-FL?

How to Comment

Please reply to this issue with your suggestions, requirements, or questions. You may also:

• Link to relevant papers, repos, or standards
• Propose API or config designs
• Share example use cases

Related Issues/PRs

• Add ICML2025 FL Paper  #190

[ASUKaiwenFang]

FL-bench Differential Privacy Implementation Discussion

Thank you for raising the important topic of implementing differential privacy (DP) in FL-bench. This is indeed a crucial feature for strengthening privacy guarantees in federated learning. I’d like to address the key points systematically:

• Central vs. Local DP
  Based on the Flower documentation and recent research trends, it would be beneficial to support both central and local DP approaches.

• Per-Round Privacy Analysis
  Should we add a privacy accountant module to track per-round privacy expenditure and report cumulative guarantees?

• Opacus Integration Strategy

  • Option A: Full Opacus integration — comprehensive but may not align perfectly with FL-bench’s architecture and FL-specific requirements.

  • Option B: Partial integration (e.g., GradSample module, privacy accountant).

  • Option C: Fully custom implementation.

• Initial Algorithm
  I suggest starting with DP-FedAvg as a testbed. This will help us evaluate how smoothly DP can be incorporated into the current system (I’ve done some preliminary tests, and it seems there shouldn’t be major issues).

I’d appreciate your feedback on these questions, as well as any additional considerations we should keep in mind when implementing DP in FL-bench.

[KarhouTam]

Thanks for the comment first. Very helpful and inspiring. @ASUKaiwenFang

Should we add a privacy accountant module to track per-round privacy expenditure and report cumulative guarantees?

Sounds fair. But I remember that the privacy accountant is not compatible with some optimizers, right? Correct me if I'm wrong. Because I've tried integrate the privacy library from pytorch (opacus) to FL-bench before and found some compatibility problems. The compatibility is the key thing FL-bench cares.

Opacus Integration Strategy

Like I said above, the compatibility is important and If you are familiar with opacus, then you can make more suggestions and lead the integration, cuz I'm not. 😂

I suggest starting with DP-FedAvg as a testbed. This will help us evaluate how smoothly DP can be incorporated into the current system (I’ve done some preliminary tests, and it seems there shouldn’t be major issues).

Agree. FedAvg is the most basic baseline and DP-FedAvg is the most suitable first algorithm that FL-bench should include. Feel free to open PR when is ready. Since @6lyc already opened a PR (#190) and planned to contribute the DP-FedAvg. I think is better have an alignment.

[ASUKaiwenFang]

Thanks for the reply! A couple of quick thoughts:

1. I also tried Opacus before, and I did run into some issues when wrapping certain components. I haven’t yet tried using the PrivacyEngine for a full integration though—so far I’ve only experimented with manually computing per-sample gradients and implementing a simple privacy accountant. I’m not sure what setup @6lyc is using, but one concern I still have is that Opacus mainly supports local DP, while central DP is not really covered. Does this mean we may need another tool or implement something ourselves for central DP?

2. At the moment, I’ve extended FedAvg class to implement both central and local DP variants, with some manual DP configurations. Once I finish polishing this, I think it could serve as a demo to discuss feasibility and potential design choices for FL-bench. Maybe when convenient, it could be helpful to have a short discussion or meeting to talk about DP integration strategies and align on design choices.

[KarhouTam]

Does this mean we may need another tool or implement something ourselves for central DP?

Probably. I am not that familar with the DP and its tools. But I think we can accomplish this feature step by step, if local DP and central DP is not tangled. We can try integrate opacus first and test it. Then the central DP. Wdyt?

Maybe when convenient, it could be helpful to have a short discussion or meeting to talk about DP integration strategies and align on design choices.

Seems good. Create a channel on Discord or slack? Wdyt? @ASUKaiwenFang @6lyc Or simply github issue or discussion is also fine for me.

[6lyc]

I'm open to any form of discussion and would be delighted to have the opportunity to collaborate with everyone to improve this excellent open-source FL library.

I suggest we could start by implementing LDP-FedAvg and CDP-FedAvg. For the implementation of CDP-FedAvg, we could later refer to flower's approach (https://github.com/adap/flower/blob/main/framework/py/flwr/client/mod/centraldp_mods.py). Subsequently, we can consider integrating more advanced DP FL algorithms, such as PPSGD (ICML 2022), FedDPA (NeurIPS 2023), and FedCEO (ICML 2025)...

On my side, I'll start by familiarizing myself with the FL-bench codebase structure and can try to submit a PR for LDP-FedAvg (based opacus) as an initial attempt.

[KarhouTam]

Happy to see that. I've made a slack channel already. Feel free to talk and exchange idea. @6lyc @ASUKaiwenFang

[ASUKaiwenFang]

I’m not sure why I can’t join the Slack workspace. I received an email saying, “It looks like there isn’t an account on FL-bench tied to this email address.” Or could you please try inviting me through the Discord channel instead?

[KarhouTam]

Alright. Here is the discord invite link below:

https://discord.gg/pAYy9QSx

cc: @6lyc @ASUKaiwenFang