Long-Term Optimization for Large-Scale Generative Retrieval with Off-Policy REINFORCE

Generative retrievers are usually trained with supervised next-item prediction, which does not directly optimize long-term user satisfaction. We cast recommendation as a session-level sequential decision-making problem and train retrievers with off-policy REINFORCE on pre-collected data, using a multi-step approximation of importance weights enabled by the autoregressive formulation. A learned user-feedback model supports doubly robust off-policy evaluation and a test-time scaling procedure that simulates future responses; on the public Yambda-5B dataset this improves cumulative session reward over next-item baselines while largely preserving retrieval quality.