Grounded AI sages

A case study in building AI that cites its source or stays silent — and in measuring whether it actually does. This is not the point of jew.tech. It is one worked example of what the intersections look like when you take them all the way down to code and eval numbers.

The premise comes from בְּשֵׁם אָמְרוֹ — the obligation to report a teaching in the name of the one who said it. A language model that produces fluent rabbinic-sounding prose with no retrievable source is doing precisely what the tradition forbids. So the design constraint was set at the start and never relaxed: answer from a cited passage, or decline. Everything below is what it took to hold that line, and where the line still breaks.

The system

Eight conversational sages — Akiva, Hillel, Moses, David, Solomon, Abraham, Rashbi, Rambam — each a persona served over the same cited Torah knowledge graph. A question routes through retrieval first: the graph returns candidate teachings, each carrying a real Sefaria reference and its Hebrew source text, and the model composes an answer only from what came back. When retrieval returns nothing usable, the correct behavior is a warm refusal, not a plausible guess.

The retrieval substrate is the same graph documented on the Torah Graph page: 17,147 rabbinic teachings, each with a citable reference, Hebrew source text, and a 768-dimension embedding — 100% coverage on all three — across 108 distinct source works, inside a 157,143-node graph.

Rabbi Akiva is the live reference implementation, with an avatar front end, and is the version everything else is measured against.

The discipline

Persona alone does not produce grounding. Three training choices carry most of the weight.

Completion-only masking

The character adapters are trained with the prompt masked out of the loss — the model is scored only on the response tokens, never on reproducing the instruction. This matters more than it sounds: without it, a small model trained on Q&A pairs learns the shape of the questions as much as the substance of the answers, and starts generating question-like filler under pressure. Masking the prompt keeps the gradient on the behavior that is actually wanted.

A bf16 LoRA over an 8B base

Each sage is a low-rank adapter in bf16 over an 8-billion-parameter instruct base, not a full fine-tune and not a quantized-base adapter. The choice is deliberate. Personas need to be swappable, cheap to retrain when the corpus improves, and comparable to one another — and training the adapter at full bf16 precision keeps the adapter's numerics from becoming a confound when eight sages are being compared against a common reference. Serving is local, on in-house GPUs.

An anti-fabrication corpus

The training data is not only positive examples. It includes explicit contrast pairs: refuse the fake alongside answer the real. A question about a genuine passage gets a grounded, cited answer; a question about a fabricated tractate or an impossible verse gets a warm, in-voice refusal — "there is no such verse" — in the same register, from the same persona. Teaching refusal as a first-class behavior rather than a fallback is what makes abstention survive adversarial phrasing. The corpus is harvested only from licensed, per-item-citable sources, PII-screened at both the source and generated-pair stage.

The measurement

Every sage is scored against a fixed behavioral battery before it can go live. The battery is deliberately hostile. Its probe modes include identity_bait ("aren't you Rabbi Akiva?" — the sage must hold its own identity and refuse the foil), anachronism (naming a text or technology from after the sage's lifetime), and fabrication_bait (a fake tractate or impossible verse, where the only passing behavior is refusal), alongside greeting, identity, cited teaching, deep cited reasoning, memory, deferral, and safety probes. Responses are judged by a large local model on grounding, voice, warmth, accuracy, completeness, safety, and fabrication, plus a separate voice-uniqueness pass that asks whether the answer sounds like this sage rather than a generic one.

Citations are checked verbatim: a reference the model produces is resolved against the graph and the quoted text compared, so a real-looking citation attached to the wrong passage fails. The gate is zero-tolerance on grounded fabrication — any invention on the cited path fails the sage outright — and zero-tolerance on meta leakage, memory leakage, identity failure, and token leakage.

The live champion, Rabbi Akiva served through the full retrieval pipeline, n = 12:

n = 12 probes. This is a small battery, and it is reported as one. It is a promotion gate, not a benchmark, and no claim is made that twelve probes establish a general capability.

The eval-validity work

The most useful thing produced by this project was not a model. It was a document proving the evaluation was wrong.

The original gate was zero-tolerance on all fabrication and on any declined-bait failure. Under those thresholds the champion — the 9.25-grounding, 1.0-citation-correctness reference implementation — failed its own gate. It failed on exactly two checks, and both fired on the same single row: one adversarial fabrication bait asking about a verse that does not exist. One slip on a twelve-probe battery sank everything. A gate that the gold reference cannot pass is not a strict gate; it is a broken instrument, because it has no headroom to distinguish "worse than the champion" from "as good as the champion."

The same analysis surfaced a second and worse problem: the candidate sages were not being compared like with like. The champion was scored through the full retrieval pipeline, with its grounding layer, abstain guard, identity strip, and persona wrapping. The candidates were scored against raw model endpoints with none of that scaffolding — and then their fabrication counts, which ran from 18 up to 249 against the champion's 1, were read as evidence about model quality. They were mostly evidence about missing infrastructure.

Both findings became a written recalibration proposal with two required fixes: measurement parity — every candidate scored behind the same retrieval layer as the reference, never against a raw endpoint — and threshold recalibration — keep zero tolerance where it belongs, on the grounded path, and allow a small, size-invariant tolerance on adversarial baits where the gold reference legitimately slips. The proposal changed no harness code itself; it was handed to the eval owner as precise pointers, because the person finding the flaw and the person owning the instrument should not be the same hand.

Honest failures

The red-team battery is public within the project and the models do not fully pass it. Fabrication baits are the hard case: a well-formed question about a plausibly-named but nonexistent source is exactly the input where a fluent model most wants to help, and warmth — which the persona is explicitly trained toward — pulls against refusal. The champion slips on one such probe out of twelve. That is a real failure, tolerated by the recalibrated gate as within the reference's own measured range, not a solved problem.

Other standing limits, stated plainly:

• Offline gate passage is necessary, not sufficient. A prior model version passed the offline battery cleanly and still confabulated in live conversation, which is why promotion additionally requires a live A/B check with fabrication and identity baits before any cutover.
• The underlying graph is machine-extracted and contains chronological errors and unmerged transliteration aliases. Grounding a model on it means grounding it on something imperfect, and citation-correctness measures fidelity to the source as recorded, not the historical accuracy of the record.
• These are personas over retrieved text. They are not the sages, they are not poskim, and nothing they say is a halakhic ruling.

Try it

Rabbi Akiva is live, with the avatar front end, at rabbiakiva.openrabbi.com. The broader study application — the full sage set and the cited-source browser — is at app.openrabbi.com.

Ask it something real and it will cite. Ask it about a tractate that does not exist and watch what happens; that is the interesting test, and it is the one it can still fail.

Model attribution

Torah text data derives from Sefaria under CC BY; every retrieved teaching carries its own reference and license. Eval figures on this page come from the champion baseline run, n = 12, and are recorded in the site's claims ledger.