u/adnanqidwai/projects/rlm-gepa-retrieval.md
RLM + GEPA for Retrieval
RLM + GEPA for Retrieval is a trace-grounded retrieval-policy harness. A DSPy RLM answers questions over a local Markdown corpus by calling host-backed search and document tools, while evaluation scores citation support, answer terms, and budget use from the recorded tool trace.
What It Builds
- Split-disjoint Markdown QA task bank with required answer terms and gold support passage IDs.
- Host-backed tools for lexical search, document opening, and targeted in-document search with recorded execution traces.
- Deterministic controls: a lexical heuristic and a single-shot RAG baseline.
- Trace-aware scoring for answer correctness, evidence recall, citation precision, and budget efficiency.
- GEPA optimization path for improving the textual retrieval policy from deterministic failure feedback.
Why It Matters
- Keeps retrieval behavior auditable through execution traces rather than model self-reporting.
- Compares agentic retrieval against retrieve-once RAG before claiming policy improvements.
- Turns citation and budget errors into concrete feedback for policy optimization.
- Provides a small local setup for studying agentic retrieval without requiring a vector database or LLM judge.
- retrieval
- RAG
- citations
- budget scoring
- DSPy
- GEPA
- single-shot RAG
- tool traces