u/adnanqidwai/projects/tinyrlvr.md
TinyRLVR
TinyRLVR is a symbolic Countdown/24-game environment for studying verifiable reasoning and test-time compute control. It generates solvable arithmetic tasks, checks candidate expressions with exact rational arithmetic, and learns when extra inference-time attempts are worth the cost.
What It Builds
- Solvable 24-game task generator backed by an exact brute-force solver.
- Verifier that rejects invalid syntax, repeated numbers, floats, unsupported operators, and wrong expressions.
- Tree policies trained with verifier rewards, process imitation, and GRPO-style group-relative updates.
- Q-learning controller that decides when to sample, verify, accept, or abstain under a budget.
Why It Matters
- Studies RLVR mechanics in a small setting where every answer can be checked exactly.
- Separates raw pass@k capability from the cost of selecting a good answer.
- Reports matched multi-seed comparisons instead of relying on one lucky run.
- RLVR
- verifiable rewards
- test-time compute
- Q-learning
- GRPO