Robust Federated Reinforcement Learning for Reliable Multi-Agent Decision Making

In many real-world applications, multiple intelligent agents or devices need to learn how to make decisions over time based on experience. For example, self-driving cars navigating different cities must learn locally from their environments while also improving through shared experience. This is where reinforcement learning (RL) comes into play [1], allowing systems to learn optimal actions through trial and error. But as we move toward systems involving many agents operating in parallel and in different locations, we need a way for these agents to learn together without sharing their raw data due to privacy concerns, limited bandwidth and system constraints. Federated learning (FL) is the key to addressing this challenge, which enables collaborative model training across distributed clients while keeping data local and private [2].

Federated reinforcement learning (FRL) builds on the idea of applying FL principles to RL problems. In FRL, multiple agents independently interact with their local environments and share only learned knowledge (not raw experiences) to collaboratively improve a shared policy [3], [4]. This approach preserves privacy, reduces communication overhead, and improves generalisation across diverse tasks and environments. It has vast potential in domains where distributed decision-making and privacy are essential, such as autonomous driving, smart factories, collaborative robotics, warehouse automation, and personalized healthcare.

However, a critical threat to the success of FRL is the presence of adversarial attacks, where malicious participants intentionally disrupt training by submitting false information. These adversaries may poison rewards, send misleading policy updates, inject delayed or corrupted data, or manipulate environment trajectories. Unlike conventional FL [2], [5], FRL faces additional challenges due to temporal dependencies, continuous action spaces, and the exploration-exploitation tradeoff.

This project aims to develop a secure and scalable anti-adversarial FRL framework that can detect malicious behaviours and defend against their effects while still enabling effective and efficient learning. The key objectives are as follows.

• Propose a systematic taxonomy of adversarial attacks in FRL and then design statistical methods to detect malicious agents.

• Develop defense methods through robust actor-critic aggregation with local trust scoring and anomaly-aware filtering.

• Demonstrate the effectiveness of the proposed methods by theoretical analyses and experimental evaluations.

References:

[1] Shakya, A. K., Pillai, G., & Chakrabarty, S. (2023). “Reinforcement learning algorithms: A brief survey”, Expert Syst. Appli., vol. 231, pp. 1-32.

[2] McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & y Arcas, B. A. (2017). “Communication-efficient learning of deep networks from decentralized data”, Int. Conf. Artif. Intell. & Stat., Florida, USA.

[3] Jiang, W., Wang, J., Zhang, X., Bao, W., Tan, C. & Fan, F. X. (2025). “FedHPD: Heterogeneous federated reinforcement learning via policy distillation”, Int. Conf. Auton. Agents and Multiagent Syst., Michigan, USA.

[4] Zhu, Y., & Gong, X. (2025). “Single-loop federated actor-critic across heterogeneous environments”, AAAI Conf. Artif. Intell., Pennsylvania, USA.

[5] Lyu, L., Yu, H., Ma, X., Chen, C., Sun, L., & Zhao, J. (2024). “Privacy and robustness in federated learning: Attacks and defences,” IEEE Trans. Neural Netw. and Learning Syst., vol. 35, no. 7, pp. 8726-8746.