Train LLM agents collaboratively across decentralized clients, without sharing local data.
- [Sep 2026] Is Decentralized LLM Agent RL Robust to Heterogeneity? An Asymmetric Tale is accepted to NeurIPS 2026, by Canyu Chen*, Kangyu Zhu*, Zhaorun Chen, Zhanhui Zhou, Shizhe Diao, Yiping Lu, Tian Li, Manling Li+, and Dawn Song+ (*Equal contribution, +Equal Advising, Homepage: https://fed-agent.github.io, PDF).
- [Sep 2026] Single-H100 recipe. Every accelerated cell has a one-GPU twin under
fedagent/config/paper_accelerated_1gpu/(gen_paper_configs.py --accel --n-gpus 1): same batch geometry, seeds and eval cadence, only the memory placement changes. Measured at Qwen2.5-1.5B over the full 70-round budget: one H100 is only about Γ2 slower per round than four (WebShop PPO 21 vs 10.6 min; on one card WebShop GRPO 17 min, ALFWorld GRPO 40 min, ALFWorld PPO 55 min per round), so a cell costs roughly half the GPU-hours, and four cells run concurrently on one 4ΓH100 node (CUDA_VISIBLE_DEVICES=<k>per driver;run_fedmaps its GPU pins through it since 2026-09-10). Recipe, per-round times, memory and caveats:fedagent/docs/gpu_recipes.md. - [Jul 2026] Reproduce the paper Γ2.5βΓ3.5 faster. Every one of the 176 paper configs
now has a ready-made accelerated version under
fedagent/config/paper_accelerated/; to use it, just swappaper/βpaper_accelerated/in any run command. Same experiment, same results: the speedup comes from removing fixed overheads (per-round process restarts, engine cold starts, env-service reboots), not from changing the training, and every lever was verified to leave the final model unchanged (differences β€ 9.3e-5, below GPU run-to-run noise). Which GPU count to use:fedagent/docs/gpu_recipes.md; how each lever works and all measurements:fedagent/docs/acceleration.md. - [Jul 2026] FedAgent is now fully migrated to stock verl 0.8
and no longer depends on verl-agent: the trainer imports verl as a library (no fork) and the
federation logic lives in the thin
fedagent/overlay. The paper's original implementation, built on verl-agent (verl 0.3.1), is preserved unchanged on thepaper-reproduce-verl-agentbranch; what changed and the fidelity record are infedagent/docs/migration.md. - [Jun 2026] Initial release of the FedAgent library, federated PPO/GRPO trainer, two-level heterogeneity suite, and full WebShop + ALFWorld reproduction.
- [Jun 2026] Paper online: Is Decentralized LLM Agent RL Robust to Heterogeneity? An Asymmetric Tale, by Canyu Chen*, Kangyu Zhu*, Zhaorun Chen, Zhanhui Zhou, Shizhe Diao, Yiping Lu, Tian Li, Manling Li+, and Dawn Song+ (*Equal contribution, +Equal Advising, Homepage: https://fed-agent.github.io, PDF). This work is honored to receive the π Best Paper Award in the AAAI 2026 Workshop on Trust and Control in Agentic AI and π Outstanding Paper Award in the AAAI 2026 Workshop on Personalization in the Era of Large Foundation Models.
FedAgent is a library for federated RL training of LLM agents. It implements a federated training loop with FedAvg aggregation (plus optional client-side FedProx), a two-level heterogeneity suite (task vs environment partitioning), and federated PPO/GRPO trainers, built as a thin overlay on stock verl 0.8 (no trainer fork: verl is imported as a library and driven through its public extension points). The paper's full experiment matrix is configured (194 cells: the paper's 176 plus the ALFWorld env-het extension; the uniform GRPO and PPO arms are GPU-verified end-to-end over the full 70-round budget on 4ΓH100 and on a single H100, while the decentralized ablations and the larger backbones are code-audited pending GPU runs β see the verification status), and the framework is extensible with your own datasets, environments, and algorithms.
FedAgent is the reference implementation for the paper, which formalizes agent
heterogeneity at two structurally distinct levels (task vs environment) and derives an
asymmetric robustness result: federated training is robust to task-level heterogeneity
but worst-case non-robust to environment-level heterogeneity. See
fedagent/docs/heterogeneity.md for the full construction.
The maintained code lives in
fedagent/; this README and thefedagent/docs/suite document it. The original verl-agent-0.3.1 implementation is preserved on thepaper-reproduce-verl-agentbranch; what changed and why is infedagent/docs/migration.md.
- Federated GRPO and PPO on stock verl 0.8: GRPO is the default (group size G=8
via
rollout.n=8, no critic); PPO (adv_estimator=gae) additionally federates the value model alongside the actor each round. The KL reference policy is an explicit role (ref_anchor): pinned to the base model for the whole run by default, or rolling with each round's aggregate (round). - Two-level heterogeneity suite: task-level (Preference / Coverage / Hardness) and environment-level on both benchmarks (WebShop: Catalog-Split + 4 transition variants: BM25 field-subset, BM25 reweight, lookalike, rank-wrapper; ALFWorld: Scene-Disjoint + obs / dyn / goal kernel variants), the first systematic decomposition for agent FL.
- FedAvg aggregation over FSDP-sharded checkpoints, plus optional client-side
FedProx (a proximal term added to local training, injected non-fork via the repo-root
sitecustomize.py, not a server rule). - Baselines built in:
federated(default),centralized(one client on pooled data), andlocal(one pinned client, no federation), selectable from the same config. - Fully configurable protocol: clients
N, clients/roundM, local epochsE, roundsT, tasks/client|Xα΅’|, with a ready-made 194-config paper matrix (the paper's 176 cells + the ALFWorld env-het extension). - Verified acceleration, results unchanged: optional speed levers (keep one trainer+vLLM
process alive across rounds instead of restarting it per client, evaluate on the
already-loaded engine, keep env services warm, run ALFWorld's env service as parallel
replicas) make paper runs Γ2.5βΓ3.5 faster; each lever was checked to leave the final
model unchanged (β€ 9.3e-5, below GPU run-to-run noise). All 194 cells come
pre-accelerated in
config/paper_accelerated/, and as single-H100 twins inconfig/paper_accelerated_1gpu/; seefedagent/docs/acceleration.md. - Any HuggingFace backbone (paper: Qwen2.5-1.5B/3B/7B-Instruct, Llama-3.2-3B-Instruct); WebShop and ALFWorld benchmarks out of the box, each behind a per-client HTTP env service so their conflicting dependencies stay isolated from the trainer.
- FSDP sharding (single-GPU to 4-GPU; at 1.5B a full paper cell runs on one H100 at about
half the GPU-hours of the 4-GPU recipe, see
fedagent/docs/gpu_recipes.md), W&B-free (metrics go to JSON / console).
Within a round, clients train sequentially by default (one subprocess per client, then
FedAvg) or concurrently with parallel_clients: P (each client on its own GPU slice,
numerically identical: FedAvg is order-free and data seeds are client-indexed); the loop is
verl-agnostic and resumable. Extension points for new datasets,
environments, heterogeneity strategies, and aggregation rules are documented in
fedagent/docs/extending.md; the capabilityβconfigβsource map
is in fedagent/docs/features.md.
fedagent/ β the maintained verl-0.8 overlay (start here)
βββ fed/ federated round loop (run_fed.py) + JSON metrics logger
βββ envs/ BaseTextEnv contract + registry; tiny_guess + per-env packages:
β βββ {webshop,alfworld}/ βββ <env>_env.py (client) + service/ (HTTP backend) + engine/ (vendored sim)
βββ agent_loops/ GymTextAgentLoop, multi-turn rollout (verl AgentLoopBase)
βββ hetero/ two-level heterogeneity constructions (task + environment)
βββ data/ AgenticDataset (verl custom_cls) + per-client partitioning
βββ config/ Hydra base, agent registry, env specs, + the 194-cell paper matrix (and its accelerated 4-GPU / single-H100 twins)
βββ docs/ full documentation suite (architecture β¦ migration)
βββ fedprox.py client-side FedProx proximal term
βββ ref_anchor.py explicit KL-reference role: pinned to the base model (default) or rolling per round
βββ main_ppo_fed.py per-client entry: stock verl run_ppo + FedAgent hooks
sitecustomize.py repo-root startup hooks (FedProx, KL-reference pin, persistent trainer, critic loss), auto-imported on PYTHONPATH
data/ shipped data assets: env_heterogeneity (holdout/lookalike) + hardness reference labels
tools/ operator toolbox: paper-config generator, hardness-label generator, eval/log/plot helpers (+ setup/: flash-attn build, verl patch)
LICENSE Β· NOTICE Β· CITATION.cff
Per-subpackage READMEs live alongside the code (e.g. fedagent/fed/,
fedagent/envs/, fedagent/hetero/); the design and
the fileβrole map are in fedagent/docs/architecture.md.
FedAgent uses three conda envs because the trainer, WebShop, and ALFWorld have mutually conflicting dependencies; they communicate over HTTP, so each stays isolated:
| Env | Role | Key deps |
|---|---|---|
fedagent-verl08 |
the trainer / federated runner | Python 3.12, verl 0.8, vLLM, FSDP |
verl-agent-webshop |
the WebShop env service | Python 3.10, gym 0.24, pyserini (JDK/Lucene), torch 2.6 |
verl-agent-alfworld |
the ALFWorld env service | Python 3.10, alfworld, textworld, gymnasium |
Full step-by-step setup (env creation, the JDK for WebShop, ALFWorld game files via
alfworld-download) is in fedagent/docs/installation.md.
W&B logging is removed: no tracking account or key needed.
The default WebShop configs run out of the box: the small WebShop catalog files are
already shipped inside the vendored env package
(fedagent/envs/webshop/engine/webshop/data/), and the env-level heterogeneity holdout/lookalike
sets are tracked under data/env_heterogeneity/. Two things are
fetched/generated separately:
- ALFWorld game files: one-time
alfworld-download -fβ~/.cache/alfworld/(seefedagent/docs/installation.md). - Hardness arm trajectories: the Hardness heterogeneity configs require per-backbone
task-difficulty labels at
data/hardness/*.json; generate them before any hardness run withpython tools/gen_hardness_trajectories.py(seefedagent/docs/reproducing.md).
Backbones are HuggingFace model ids (default Qwen/Qwen2.5-1.5B-Instruct) and
auto-download on first run to ~/.cache/huggingface (set HF_HOME to relocate). The main
table's Llama-3.2-3B-Instruct is gated (accept its license + huggingface-cli login);
on offline / air-gapped clusters, pre-fetch on a login node and pass --model-path <local snapshot>. See fedagent/docs/installation.md.
Run a FedAgent experiment directly with the federated runner, from the repo root inside
the fedagent-verl08 env (WebShop/ALFWorld runs also need their service env available):
# 0) In-process smoke, verifies the federated loop end-to-end, no remote service
python -m fedagent.fed.run_fed --config fedagent/config/examples/tinyguess_2cl_2rd.yaml
# 1) WebShop, homogeneous, GRPO
python -m fedagent.fed.run_fed --config fedagent/config/examples/webshop/homog_long.yaml
# 2) A paper cell (WebShop, Qwen2.5-1.5B, main, GRPO) from the 194-cell matrix
python -m fedagent.fed.run_fed \
--config fedagent/config/paper/uniform/Qwen2.5-1.5B-Instruct/main/grpo/fed_webshop_grpo_total-100_cl-per-rd-2_rd-70_ep-per-cl-3_min-goals-per-cl-100_p-uniform.yaml
# 3) The same cell ACCELERATED: identical science, a fraction of the wall-clock
# (swap paper/ -> paper_accelerated/ for any cell; see fedagent/docs/gpu_recipes.md)
python -m fedagent.fed.run_fed \
--config fedagent/config/paper_accelerated/uniform/Qwen2.5-1.5B-Instruct/main/grpo/fed_webshop_grpo_total-100_cl-per-rd-2_rd-70_ep-per-cl-3_min-goals-per-cl-100_p-uniform.yaml
# 4) The same cell on ONE H100: about x2 the per-round time of 4 GPUs at half the GPU-hours.
# Four such drivers co-host a 4-GPU node (CUDA_VISIBLE_DEVICES=0..3, distinct --output-dir).
CUDA_VISIBLE_DEVICES=0 python -m fedagent.fed.run_fed \
--config fedagent/config/paper_accelerated_1gpu/uniform/Qwen2.5-1.5B-Instruct/main/grpo/fed_webshop_grpo_total-100_cl-per-rd-2_rd-70_ep-per-cl-3_min-goals-per-cl-100_p-uniform.yamlCLI flags override the YAML: --rounds N Β· --clients N Β· --n-gpus 4 Β· --base-seed S
Β· --fedprox-mu 0.1 Β· --local-client-id K Β· --critic-path DIR Β· --fresh. Every config key is documented in
fedagent/fed/README.md; hardware/run modes are in
fedagent/docs/running.md.
The paper's experiments are the 194-cell matrix under
fedagent/config/paper/, mirroring the original tree 1:1
(uniform/ main table across 4 backbones Γ WebShop + ALFWorld; env_heterogeneity/ on both
benchmarks, task_heterogeneity/, decentralized/ on Qwen2.5-1.5B). Each cell is one command:
python -m fedagent.fed.run_fed --config fedagent/config/paper/<family>/<...>.yamlEvery cell also has an accelerated twin at the same relative path under
fedagent/config/paper_accelerated/; swap
paper/ β paper_accelerated/ for the same science at roughly Γ2.5βΓ3.5 less wall-clock, or
paper/ β paper_accelerated_1gpu/ to run it on one H100 (about Γ2 the 4-GPU per-round time
at half the GPU-hours; fedagent/docs/gpu_recipes.md).
Per-table recipes, seeds, and compute estimates (~1,800 H100 GPU-hours total) are in
fedagent/docs/reproducing.md, covering the main table
(Local / Centralized / FedAgent Γ four backbones Γ WebShop + ALFWorld), the task- and
environment-level heterogeneity studies, and the decentralized ablations.
| Doc | Contents |
|---|---|
fedagent/docs/architecture.md |
The overlay design: the round loop + hooks on stock verl 0.8, and the fileβrole map. |
fedagent/docs/installation.md |
The three-conda-env setup (trainer + WebShop + ALFWorld), JDK / game-file notes. |
fedagent/docs/running.md |
Running run_fed.py: modes, GPUs, baselines, FedProx, eval, worked examples. |
fedagent/docs/gpu_recipes.md |
Best-practice recipes per GPU count (1 / 2 / 4) and the accelerated paper matrix. |
fedagent/docs/reproducing.md |
Per-experiment reproduction recipes, the 194-cell matrix, measured compute (4 GPUs vs 1), seeds. |
fedagent/docs/heterogeneity.md |
The two-level taxonomy and how to construct/select each arm. |
fedagent/docs/configuration.md |
Config-file decoder and the federated-runner key reference. |
fedagent/docs/features.md |
Each capability β its config key β its source file. |
fedagent/docs/extending.md |
Extension points: new dataset/env, heterogeneity strategy, RL algorithm, aggregation rule. |
fedagent/docs/migration.md |
What changed from the verl-agent-0.3.1 fork to stock verl 0.8, and the equivalence checks. |
fedagent/docs/acceleration.md |
How to accelerate & why: the final recipe, why each lever works, the equivalence bar. |
fedagent/docs/revision.md Β· bugfixes.md |
The ledger of deliberate default/protocol changes (what older runs inherit, e.g. the 2026-09-16 ref_anchor flip), and the dated defect log. |
If you use FedAgent in your research, please cite:
@inproceedings{chen2026fedagent,
title = {Is Decentralized {LLM} Agent {RL} Robust to Heterogeneity? An Asymmetric Tale},
author = {Chen, Canyu and Zhu, Kangyu and Chen, Zhaorun and Zhou, Zhanhui and Diao, Shizhe and Lu, Yiping and Li, Tian and Li, Manling and Song, Dawn},
booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
year = {2026},
url = {https://openreview.net/forum?id=SsF2kYJfYC}
}This project is released under the Apache License 2.0, see LICENSE.
FedAgent is a thin overlay on stock verl 0.8 and reuses the verl-agent environment packages. We gratefully acknowledge:
- veRL: Β© ByteDance / the veRL authors (Apache-2.0): the base RL training framework that FedAgent imports as a library. https://github.com/volcengine/verl
- verl-agent / GiGPO: Feng et al., Group-in-Group Policy Optimization for LLM Agent Reinforcement Learning (arXiv:2505.10978): the agent-RL environment integrations FedAgent builds on. https://github.com/langfengQ/verl-agent
- WebShop: Yao et al., Princeton NLP (MIT License): the e-commerce agent benchmark.
- ALFWorld: Shridhar et al., Microsoft Research (MIT License): the embodied household agent benchmark.
Full per-component attributions and license texts are aggregated in NOTICE.