TinyRLVR
x

u/adnanqidwai/projects/tinyrlvr.md

TinyRLVR

TinyRLVR is a symbolic Countdown/24-game environment for studying verifiable reasoning and test-time compute control. It generates solvable arithmetic tasks, checks candidate expressions with exact rational arithmetic, and learns when extra inference-time attempts are worth the cost.

What It Builds

  • Solvable 24-game task generator backed by an exact brute-force solver.
  • Verifier that rejects invalid syntax, repeated numbers, floats, unsupported operators, and wrong expressions.
  • Tree policies trained with verifier rewards, process imitation, and GRPO-style group-relative updates.
  • Q-learning controller that decides when to sample, verify, accept, or abstain under a budget.

Why It Matters

  • Studies RLVR mechanics in a small setting where every answer can be checked exactly.
  • Separates raw pass@k capability from the cost of selecting a good answer.
  • Reports matched multi-seed comparisons instead of relying on one lucky run.
  • RLVR
  • verifiable rewards
  • test-time compute
  • Q-learning
  • GRPO