A structured decision API gives software a convenient interface to model judgments. An application supplies context, defines the possible answers, and receives a typed result. That interface removes some integration problems. It leaves another question open: how stable is the decision when the same information is represented differently?
We investigated this question using Jev jev-1.13.0, the model served by TypeSafe during the selected experiments on 18 September 2026 (UTC). We constructed graph problems with exact answers, changed their representations, repeated identical requests, and inspected failures. The experiments reveal substantial sensitivity to record arrangement and consistent identifier changes. They also show why an apparent improvement can be difficult to interpret: a representation may expose a shortcut that bypasses the intended task.
This article presents three completed studies. Their combined sample comprises 496 independent graph worlds and 6,560 attempted requests, of which 6,559 returned usable responses. Requests within one world share an underlying problem. They are not independent examples. The technical paper specifies the sampling units, analyses, and limitations; the evidence package contains the requests, responses, and offline reproduction materials.
A changed relationship, an unchanged root answer
Consider a forest represented by statements such as “Node A has immediate parent B.” Each node belongs to one of four trees. Finding its root requires following parent links until reaching a node with no parent.
In one saved example, we changed four parent references so that corresponding branches moved between trees. The edited graph remained valid. Its tree sizes and structural symmetry were preserved. The question about a particular node remained identical:
Which root contains node n_yuqwszit?
Before the edit, the correct option was a. After the edit, it was c. All obsolete parent facts had been replaced. However, the records initially retained their old ordering. A separate question about the edited local relationship received the correct new parent, while the root question still selected a.
Reordering those same edited records by their current root changed the selected root to c.
The full experiment gives this example context. Across 16 fresh graph worlds, every direct-parent observation was correct: 512 in baseline states and 768 in edited states. Root identification behaved differently. Refreshing the grouping improved affected-root accuracy by 72.08 percentage points, with a 95% forest-bootstrap interval of 65.42–78.33, across the 15 forests complete for this comparison. One failed request prevented the sixteenth forest from contributing to that paired endpoint. The failure remains in the dataset.
This result has a practical interpretation: correct responses to local checks do not establish that a model will incorporate those facts correctly into a global decision. It does not identify an internal graph representation or a memory mechanism. A simple association between record blocks and nearby roots remains a sufficient explanation for success in the regrouped condition.
Regrouping also requires exact knowledge of the graph. For a production task that only asks for a root, ordinary code should compute the answer directly. The model experiment is useful as a diagnostic, not as a recommendation to replace a graph algorithm.
Renaming is an intervention, not just a cosmetic change
A second study changed the names assigned to nodes while preserving the graph and consistently updating all references. If a problem depends only on relationships, a bijective renaming should preserve its answer after the labels are mapped back.
An isolated disagreement would be weak evidence. The service can return different answers to identical request bodies. We therefore submitted two exact copies of every representation and compared cross-name disagreement with matched within-body disagreement.
The study used 96 fresh forests, five name assignments, two layouts, and two copies: 1,920 requests. For root questions in shuffled layouts, disagreement between differently named versions averaged 67.38%. The matched identical-request reference was 12.24%. The excess was 55.14 percentage points, with a 95% forest-bootstrap interval of 49.93–60.35.
All four root/ancestor-by-layout comparisons specified in the local study protocol showed positive excess disagreement. The grouped-root effect was much smaller and concentrated in a few forests: 10 positive, 2 negative, and 84 zero effects. These protocols were frozen locally before the recorded calls; they were not externally preregistered.
The distinction between disagreement and accuracy is essential. A different answer can repair an error or introduce one. In this study, the intervals for signed accuracy changes included zero for all four endpoints. We did not discover a naming convention that reliably improves accuracy.
The original intervention also changed more than abstract identity. It changed names in the question and candidate descriptions, and it could change how often individual identifiers appeared in the state. Equal identifier inventories and string lengths do not imply equal contextual tokenization. Unlike the locality intervention, these renamings also changed billed input size: the within-forest range across views was 7–80 tokens. These are plausible contributors, not minor details to remove from the discussion.
The next prepared renaming study addresses these alternatives. It holds question and candidate identifiers fixed and restricts internal substitutions to names with the same role frequencies. The protocol and software have been prepared; no result from that prospective study is included here.
Relevant facts depend on both location and spacing
A third study placed the relevant parent path within a larger forest description. We independently varied whether the path records were compact or dispersed and whether they appeared early or near the middle of the context. The underlying facts, questions, and billed token counts remained equal within each comparison.
Across 384 fresh forests, root accuracy was:
| Path representation | Correct answers | Accuracy |
|---|---|---|
| Compact, early | 309 / 384 | 80.47% |
| Compact, middle | 157 / 384 | 40.89% |
| Dispersed, early | 117 / 384 | 30.47% |
| Dispersed, middle | 122 / 384 | 31.77% |
Compactness helped much more in the early position. The prespecified interaction—the compactness benefit early minus the compactness benefit in the middle—was 40.89 percentage points, with a 95% forest-bootstrap interval of 34.11–47.66.
The root declaration remained at the front, so this experiment does not separate absolute path position from proximity to that declaration. A completed follow-up moved the unchanged declaration between the front and end in 48 new forests and 576 requests. It found no observed improvement for late compact paths when the declaration moved to the end. Its primary interaction was inconclusive: −16.67 percentage points, with a 95% interval of −37.50 to +4.17. The full control does not support the simple predicted improvement, while leaving other proximity explanations unresolved.
This does not establish a universal rule to place every important sentence first. The intervention used exact graph traversal to identify the relevant path, and other tasks need different relationships. It demonstrates a specific failure boundary: equal facts and equal billed size can still produce very different results when the arrangement of the relevant relationships changes.
Why successful representations need their own controls
An easy interpretation of these findings would be to choose the layout with the highest accuracy and recommend it everywhere. That would skip a central part of the evidence.
Root grouping makes membership easier to infer from a block. An explicit root index supplies a computed answer. A path selected using the known graph supplies relevant evidence that a deployed system may not yet know how to retrieve. These interventions can be valuable, but their benefits differ from those of a lossless syntax change.
Earlier studies in the research program also contained candidate sets with unintended structural cues. Some ancestor answers could be identified from candidate depth without relating the candidate to the queried node. Later experiments strengthened the candidates, but grouping and record-location shortcuts remained possible. The technical paper records these limitations because they determine what the measurements mean.
A practical evaluation should distinguish four operations:
- Re-encoding facts: for example, writing the same edges as sentences or a table.
- Reordering records: changing their sequence while retaining every fact.
- Exposing computed structure: grouping records by a relationship already established in code.
- Adding or selecting derived evidence: supplying an index, resolved source, or known relevant path.
The operations may produce similar accuracy improvements while requiring very different capabilities from the surrounding application. Reporting them under one label such as “better formatting” conceals that difference.
Confidence needs workload-specific evaluation
Typed outputs and numerical confidence do not remove the need to test the final decision policy. In a separate path study, a fixed confidence threshold of at least 0.9 accepted 261 root answers, of which 45 were incorrect. Those answers were repeated views of underlying forests, not 261 independent problems.
The accepted cases are a selected subset of the workload. A useful evaluation therefore reports both coverage—the fraction of decisions accepted—and error among accepted decisions. A threshold validated in one representation should not be assumed to transfer to another.
TypeSafe documents confidence as a statistic derived from the returned probability distribution. It is a separate field, not an independent correctness estimate. We distinguish the selected-option probability, this distribution summary, and observed correctness. The confidence example is supporting evidence from the broader campaign and is not included in the three-study request total above.
A practical division of work
The current evidence supports a measured engineering approach. Resolve exact identifiers and relationships in code when the necessary structure is available. Supply the resulting evidence with its provenance. Evaluate the semantic judgment that remains, including ambiguous cases and realistic changes in the surrounding document.
For example, an application may need to identify the policy that governs a support ticket and then interpret the ticket under that policy. Resolving an explicit policy reference is a different problem from interpreting whether the narrative satisfies a condition. Testing only the final answer can conceal which stage failed.
Our earlier application studies demonstrated improvements in exact ownership lookup and substantial reductions in context size. They did not establish a semantic accuracy improvement on the harder narrative corpus. The prepared application evaluation therefore scores source selection and semantic interpretation separately and jointly. It includes development fixtures and annotation procedures; independent held-out documents and baseline model measurements remain to be collected.
Relation to prior research
Representation sensitivity is an established research topic. Talk Like a Graph studies graph encodings. Can Graph Descriptive Order Affect Solving Graph Problems with LLMs? studies ordering effects across tasks and models. Lost-in-Distance examines the separation between relevant graph facts. Recent GRAIN/GRIT work explicitly tests naming and task-formulation shifts and investigates deterministic graph extraction. These studies are listed with primary-source links in the references.
Our contribution is a controlled case study of a structured-decision service: repeat comparisons, counterfactual changes, exact answer checks, and explicit analysis of shortcuts. The evidence does not establish that Jev is unique in these sensitivities. Comparisons with conventional models are part of the prepared follow-up program.
TypeSafe's published Jev 1.13 limitations already identify difficulties with indirection, irrelevant context, and structural invariants, and recommend exact computation in code. Our experiments quantify controlled instances of these acknowledged limitations. They do not establish a defect in the advertised type guarantee or disprove performance on other workloads.
What the next tests can change
The follow-up work is designed to resolve alternatives, not merely to increase the number of requests. A constrained renaming effect that remains large would weaken explanations based only on changed query names or marginal name frequencies. A much smaller effect would explain part of the original result and narrow the claim.
A larger layout-conflict study will independently place truthful root declarations to favor the current root, the old root, or a third root. That intervention can test whether answers follow an incorrect layout cue without requiring any claim about internal memory. Its interpretation will still need to account for root-to-target distance and the information supplied by grouping.
Finally, independently authored application cases can establish whether a proposed preprocessing method improves useful decisions, only reduces cost, or fails to transfer. Those are different outcomes, and the final publication should reflect whichever occurs.
The completed experiments provide a reproducible starting point: semantic equivalence alone does not ensure stable model decisions, and high accuracy under an assisted representation does not establish how the answer was obtained. A reliable application must evaluate the representation, the evidence-selection procedure, and the final decision together.