A context window is not a training budget. During inference, a prompt is evaluated forward, converted into attention state, and extended token by token. Reinforcement-learning training adds multiple responses, scoring, backward activations, gradient buffers, and an optimizer step. A sequence that fits once can fail when its shared computation is retained across rollout branches. The central question is not “How many tokens fit?” but “Which bytes coexist, for how long, and how many times?”
A concrete case appears in Macaron-V1, LongStraw reports grouped scoring and response backward for Qwen3.6-27B at 2.1M positions with group sizes of 2 and 8 on eight H20 GPUs, together with a separate 4.46M-position stress test. For GLM-5.2, the latest validation reports an exact-2M, G=2 response-only GRPO update across 32 H20 GPUs. These results establish method-level execution under a defined training contract; they do not demonstrate full-parameter training, general scaling, or task quality at two million tokens.
Why inference capacity does not imply training capacity
Let a prompt contain P tokens and produce B candidate responses with lengths R_1 through R_B. At inference, the prompt can be evaluated once and represented by model-specific retained state, such as attention pages, compressed latent state, or recurrent state. Each response extends that state, and temporary generation data can be released as branches finish. The runtime does not need a differentiable graph spanning every branch.
Training state is broader. Conventional RL must retain activations for backward, score multiple responses, and accumulate gradients over the shared prompt. If an implementation constructs a full prompt-dependent graph for every rollout, its large activation term behaves roughly like:
B × prompt graph + live response graphs
This is a scaling model, not a byte-accurate formula. Actual memory also includes parameters, optimizer state, gradient buffers, communication workspaces, and allocator headroom. The formula nevertheless identifies the damaging multiplier: computation shared by all candidates is represented as if it were unique to each one.
An inference KV cache therefore proves only that retained forward state fits. It says nothing about backward graphs, rollout concurrency, gradient accumulation, or how long those objects overlap.
Account for memory by lifetime
A useful capacity model groups state according to when it must exist:
- Run-persistent state: parameters, optimizer state, gradient buffers, and distributed-runtime allocations.
- Prompt-group state: the evaluated prompt, valid while compatible responses use the same model version.
- Replay-temporary state: the live graph and branch-specific data for one response or replay microbatch.
- Update-boundary state: accumulated gradients and synchronization metadata that survive replays but end at the optimizer step.
This turns peak memory into a scheduling problem. The objective is not merely to compress objects, but to avoid unnecessary overlap and prevent prompt-sized state from acquiring a rollout-sized replication factor.
The update boundary belongs to that schedule. Unless an algorithm explicitly supports mixed policy versions, changing parameters between replayed branches would make the shared prompt state and later response graphs inconsistent. Accumulating compatible branch gradients before one distributed update creates a clean lifetime barrier. The old resident state can then be invalidated, and the next group can begin from a new model snapshot.
Make the prompt resident and the learning graph selective
Resident state changes the prompt from repeated branch computation into a retained intermediate representation. In an idealized design, peak memory becomes closer to:
fixed training state + resident prompt state + K × live response graph + communication headroom
Here, K is the number of responses replayed concurrently, not necessarily the number sampled. If branches are replayed sequentially or in small microbatches, K can remain much smaller than B. The prompt is still expensive, but it is paid once per compatible response group rather than once per rollout.
Response replay supplies the complementary mechanism. Sampling and learning need not use identical memory schedules. A system can perform old-policy and reference scoring without retaining a training graph, then replay each policy response that contributes to the loss under autograd. After backward, each graph can be released while its gradient contribution remains accumulated.
This exchange is not free. Reconstructing response computation costs time, and excessive serialization can leave GPUs underused. The trade is additional or rescheduled compute for lower activation residency. It is useful when memory otherwise prevents execution or when a modest replay microbatch still uses the hardware effectively.
The release identifies distributed model-native execution as LongStraw’s third central idea. The broader design implication is that state ownership should align with model partitioning. Repeatedly exporting large prompt representations through a coordinator could replace a memory bottleneck with communication and serialization overhead. The resident tensors should remain near their consumers while compact control information moves between workers.
Preserve gradient semantics, not just forward outputs
Prefix reuse at inference and resident state during training are not interchangeable. A cached forward output may be enough to continue generation yet insufficient to compute the gradient that end-to-end training would assign to prompt processing. Detaching resident state without compensating work creates a truncated-gradient objective.
LongStraw makes this boundary explicit through a response-only training contract. The shared prompt is captured without autograd, while each policy response is replayed under autograd and contributes to the accumulated update. This preserves the direct response gradient but does not reproduce every prompt-side gradient path of a conventional full-sequence run.
In the project’s 32K differential validation, the conventional and response-only paths reached a gradient cosine of approximately 0.99993, with relative L2 differences of about 1.16% to 1.17%. This is strong evidence for the tested response-only contract, although it does not by itself establish task quality at untrained two-million-token positions.
Freezing a base model does not automatically remove this concern because trainable adapters or other modules may participate in prompt evaluation. Full-model training makes the requirement direct. “Resident” describes an object’s lifetime and reuse; it does not make the object semantically constant or exempt from gradient accounting.
State identity must also be explicit. A resident prefix can depend on model and adapter versions, tokenization, attention masks, positional treatment, routing choices, and other execution settings. Matching tokens alone do not establish compatibility. Each resident object needs a versioned identity and a defined invalidation point. Reuse after an update is unsafe unless the relevant representation is known to be unchanged.
Distributed replay adds accounting hazards. Retries must not double-count gradients, and worker failures must not silently drop contributing branches. Gradient normalization must use the intended response set. No worker should advance to a new model version while another still replays against the old one.
Validation should be differential. On a small workload, hold sampled responses and loss inputs fixed, then compare the lifetime-optimized path with a conventional run. Gradients and parameter deltas should agree within a declared numerical tolerance. Tests should vary replay order, microbatch size, worker count, failure recovery, and state invalidation.
Know when the geometry stops helping
Resident-prefix execution pays off when the prompt is much longer than each response, several candidates share the exact prompt, only some branches receive gradient updates, and resident state leaves capacity for useful replay batches.
The advantage contracts when responses dominate sequence length, branching factor is low, nearly every response must be replayed, or the resident prefix consumes most available memory. Replay may become compute-bound. Placement, version tracking, synchronization, and recovery also have costs.
Agent trajectories introduce another boundary: context is shareable only to the exact point of divergence. Once branches receive different tool results, observations, or environment state, their suffixes are no longer one reusable prefix. Treating the entire trajectory tree as shared risks incorrect reuse; retaining every internal node may create a state-management problem as large as the graph it replaced.
A fixed-budget design checklist
Before judging a long-context RL system, ask:
- What is the peak-memory ledger for parameters, optimizer state, resident prompt state, live response graphs, gradients, communication buffers, and safety margin?
- How many responses share each resident prefix, and how many actually receive gradient updates?
- What gradient semantics cross the resident-state boundary, and how are they checked against an end-to-end baseline?
- Which model version owns each resident object, and exactly when is that object invalidated?
- How do retries, worker failures, and duplicate replay requests affect gradient counting and normalization?
- At what prompt length, response length, branching factor, and replay batch size does recomputation outweigh memory savings?
- Is performance measured as completed, correct policy updates under a fixed GPU budget, or merely as the largest executable context?
Long-context RL becomes tractable when shared prompt state is persistent, learning graphs are branch-selective and short-lived, and the optimizer step closes a versioned accumulation window. This does not make the prompt free. It removes the assumption that every rollout needs its own full-sequence training graph. The decisive capability is not accepting more tokens, but controlling the lifetime, replication, and ownership of the state those tokens create.