Question and scope
Does deterministic source resolution improve a complete decision pipeline when the downstream task still requires semantic judgment? The primary comparison is code-resolved evidence versus the complete original context, evaluated separately for each model. This is an application-pipeline comparison. It cannot isolate a neural mechanism: the intervention changes the context, available source choices, number of model calls and preprocessing work.
The protocol is prepared prospectively. Independent holdout collection, model selection, access checks, precision planning, cost approval and a frozen run manifest remain prerequisites. No new model result is included here.
Input and oracle
Each case contains documents, literal reference declarations, and a ticket with a policy reference and narrative. A reference resolves by following exact identifiers until it reaches one document. A missing target, cycle, or alias with distinct targets is unresolved. Duplicate identical declarations are redundant. Document and reference identifier namespaces cannot overlap. The correct source is determined mechanically, without reading the narrative, document topic, gold fields or model output.
The semantic oracle requires human reading of the resolved policy and ticket. It is one of permit, block, or review; a policy can explicitly require review even when the task is fully specified. __ABSTAIN__ is a separate model response for insufficient evidence. A model error or malformed response is not a correct abstention.
Only documents, reference declarations and the ticket are model-visible. Source traces, labels, rationales, author identities, split membership and annotations remain private. All request construction uses a public-field allowlist, tested by mutating private gold fields.
Conditions
| Condition | Source stage | Semantic-stage evidence | Intended intervention |
|---|---|---|---|
| Full context | Model follows literal references over every document in original order | Same full context plus its selected source ID | Baseline pipeline |
| Canonical | Model sees all documents sorted by ID and references by (id, target) |
Canonically ordered context plus selected ID | General ordering only; no facts deleted |
| Retrieval | Model sees fixed BM25 top two documents and every reference declaration | Same retrieved context plus selected ID | Deterministic lexical filtering |
| Code-resolved | Code follows literal references; no source model call | Ticket and exact terminal policy only | Exact reference resolution plus semantic model call |
BM25 uses lowercased alphanumeric terms, document ID/title/body, query reference plus ticket narrative, k1=1.2, b=0.75, and k=2. Equal retrieval scores are ordered by document ID. It does not traverse the reference chain or read annotation fields. If the terminal document is omitted, the source stage must abstain. This is a deliberately simple lexical baseline, not a claim about the strongest retrieval system. Additional retrieval methods require a separate development budget and prospective condition definition.
Canonical sorting preserves the full document/reference multiset. The source criterion order follows available document order; therefore canonicalization also changes criterion order. The condition measures that complete pipeline transformation, not state ordering alone. Retrieval changes both evidence and candidate set. These effects must not be relabeled as a pure context-length effect.
The semantic instruction requires applying the selected source; it cannot silently replace it with another. Full/canonical/retrieval conditions retain their condition context and selected-source pointer. Code-resolved input contains only the terminal policy. A correct semantic label after a wrong source is reported separately and does not count as joint success.
An optional future oracle-selected evidence upper bound would use human annotations, be explicitly marked assisted, and remain separate from these four executable conditions. It is not implemented here.
Development fixtures
The four bundles deliberately exercise different judgments: functional necessity versus comfort in travel; aggregate disclosure versus direct or linkable personal data; competing causes of a damaged product; and reversibility/security constraints in a software change. The policy text and ticket-specific rationales are available in the fixture generator. Each family has three tickets sharing its documents, plus retired and irrelevant policies that make topical matching insufficient.
These examples are small, synthetic and agent-authored. Current naming conventions, active version suffixes and path lengths are not balanced. They test software and illustrate annotation decisions; they do not validate task difficulty or real-world benefit. Fresh data should vary these characteristics without deriving the test labels from model outcomes.
Execution contract for a future run
A complete plan contains every case × adapter × four conditions. Randomize complete case blocks, rotate adapter/condition order across blocks, log actual timestamps, and keep one declared attempt per executed stage. Code resolution substitutes for the source model call; it is not a free unrecorded model invocation. If source selection fails or abstains, do not run the semantic stage. Preserve its not_run record.
Freeze model versions, prompts, parsers, preprocessing, limits, hardware, prices and the plan before outcomes. Require an exact result row for every planned pipeline, including failures. A missing row or duplicate row stops analysis. Input/context hash mismatches stop analysis. The primary scorer rejects retry backfilling; a later robustness run must use separate records and a separate protocol. Log requests, raw responses, error states, usage, preparation time, network duration and complete pipeline time before parsing.
No confidence threshold is used. Abstention is an explicit output option whose coverage and loss are reported. This prevents interpreting a confidence scalar as the probability that a semantic decision is correct. A future calibrated gate would require disjoint development data and a separately frozen decision rule.