Authors Amine Dirhoussi, Quentin Gallouédec, Kashif Rasul and Sergio Paniego describe how AsyncGRPOTrainer (TRL v1.14, PR #7017) can train a LoRA adapter and synchronize only that adapter to vLLM. Because a rank-1 adapter is a few megabytes compared with a ~3 GB full model, the adapter can be exchanged via a Storage Bucket mounted in each HF Job rather than over NCCL.
Architecture and deployment
The deployment separates the trainer and vLLM replicas into distinct Hugging Face Jobs. The trainer writes adapters to a shared path (for example /.vllm_lora/trl-policy-v{N}) on a Storage Bucket mounted via hf-mount, then calls vLLM’s /v1/load_lora_adapter endpoint with that path. vLLM loads adapter files from disk, so adapter-only sync requires only a shared filesystem rather than direct inter-node communication.
The setup used by the authors comprises one trainer Job, two vLLM Jobs (each on one GPU), a Storage Bucket mounted read-only in the servers, and a small proxy running on the trainer Job. vLLM was pinned to v0.27.1 and configured with VLLM_ALLOW_RUNTIME_LORA_UPDATING=1 and VLLM_SERVER_DEV_MODE=1. The authors note that max_staleness controls how many adapter versions must be resident; for max_staleness=4, the replicas require –max-loras 6.
Proxy, routing and KV cache affinity
The proxy serves two purposes: it adds the required Authorization header for exposed Job ports and it implements routing and broadcast logic. Broadcasts ensure state-changing requests (adapter loads, pause/resume) reach every replica so a policy name has the same meaning everywhere. Routing steers completion requests to the replica most likely to hold the prompt’s KV prefix, reducing redundant prefill work.
Routing is based on chained 16-token block hashes seeded with the adapter name. The router tracks which replica has served which block hashes and directs each of the G rollout requests for a prompt (G=8 in the experiment) to the replica holding the cached prefix when possible. This preserves affinity for shared prefixes while avoiding stale cache collisions across adapter versions.
Evaluation dataset and results
The authors validated the setup on sail/Sanity-Test-R1D-1.5B with Qwen/Qwen2.5-Math-1.5B using LoRA rank 1 (alpha 2), learning rate 4e-5, 8 samples per prompt, 128 completions per step, up to 3,000 generated tokens and 4,096-token context. They report that five runs of the same recipe reduced time for 500 steps from 3 h 27 min to 53 min. The post also notes that TRL falls back to merged-weight sync for incompatible configurations and that the adapter-only path is enabled when a lora_config is detected during initialization.
Concrete artifacts and versions mentioned by the authors: AsyncGRPOTrainer LoRA support in TRL v1.14 (PR #7017) and vLLM behavior as exposed in v0.27.1.
Original source: Hugging Face Blog