Semantic invariance and evidence layout

Abstract

We study representation sensitivity in Jev jev-1.13.0, a hosted model that returns structured decisions. Three completed graph studies comprise 496 independent forest worlds and 6,560 attempted requests, with one retained failed response. Consistent identifier substitutions produce excess label disagreement beyond matched exact-request variation; for shuffled root queries, the excess is 55.14 percentage points across 96 forests. A counterfactual study finds correct local parent answers alongside root answers associated with obsolete record grouping; refreshing the grouping improves affected-root accuracy by 72.08 points across 15 complete forests. A 384-forest study finds a 40.89-point interaction between relevant-path compactness and context position. We distinguish unchanged-fact interventions from preprocessing that exposes computed structure. The latter admits simple membership cues, and some historical candidate protocols admit additional shortcuts. These limitations constrain claims about reasoning and architecture. The accompanying materials include selected evidence and offline analysis materials, together with prepared tests that fix query identifiers, control name frequencies, manipulate conflicting layout cues, and evaluate practical source resolution with semantic decisions. The prospective tests have not produced model results.

1. Introduction

Structured decision interfaces let an application specify a finite answer space and consume a model's result directly in code. Type correctness is an interface property. It does not establish invariance to alternative representations of the same problem, correct propagation of factual changes, or calibration on a deployed workload.

We examine these distinctions through graph problems whose answers can be computed exactly. The graph is a diagnostic object rather than a proposed replacement for an exact graph algorithm. It permits explicit control over facts, names, order, relevant paths, candidate alternatives, and counterfactual edits. It also permits independent parsing of the actual submitted text to check the intended answer.

The central questions are: (1) whether meaning-preserving naming changes alter decisions beyond ordinary repeat variability; (2) whether a correct local response after an edit is sufficient for a correct global response; and (3) whether the accessibility of relevant facts depends jointly on their spacing and position. We describe observable service behavior. The experiments cannot uniquely identify neural architecture, internal serialization, training method, or a latent reasoning procedure.

This paper is a focused report drawn from a larger exploratory research program. The three selected studies were completed before this publication framing was chosen. Each study retained its own previously specified comparisons and analysis. The paper does not claim that these three studies were selected prospectively as one globally corrected experiment.

2. Related work

Graph representation effects have prior precedent. Fatemi, Halcrow, and Perozzi's Talk Like a Graph evaluates graph encodings for language models. Ge et al. examine graph description order across tasks and models. Firooz et al. analyze distance between relevant facts in graph contexts. Liu et al.'s Lost in the Middle establishes related position effects in long-context use. The recent GRAIN/GRIT preprint studies shifts in node identifiers and task formulation.

These papers prevent a broad novelty claim for order, distance, or naming sensitivity. GRAIN's GRIT evaluation crosses four naming schemes with two task formulations and investigates deterministic graph extraction. Our narrower contribution is a reproducible audit of this structured interface that combines exact-request controls, counterfactual factual edits, literal answer verification, and shortcut analysis. We do not claim that representation invariance, deterministic preprocessing, or the distribution-distance estimator below is new. Primary-source links and bibliographic status appear in the references.

TypeSafe's Jev 1.13 documentation already describes limitations involving indirection, irrelevant context, numerical precision, and structural invariants, and recommends exact computation in code. The graph workloads quantify instances of acknowledged limitations. They do not test a claim that type-correct outputs must be semantically correct.

Probability interpretation is a separate topic. Lovering et al.'s numeric-context study overlaps closely with exploratory numerical anomalies elsewhere in our campaign. Those anomalies are not a principal contribution of this paper and are not used as evidence of an unusual architecture.

3. Interface, tasks, and interventions

3.1 Structured decisions

Each submitted request specifies the model, a state containing context, and one or more questions. The selected experiments use Choice questions with explicit alternatives. The response includes a selected alternative, an associated probability map, and a separate confidence field. Analyses retain the recorded model identifier, question text, alternative ordering, and response fields.

The documentation defines confidence as a statistic derived from the returned distribution. It is not an independent correctness estimate. Selected-option probability, distribution concentration, and empirical answer correctness remain separate quantities in this analysis.

The selected label is the primary outcome for the main effects. Numerical coordinates and confidence are not substituted for observed correctness. Reported probability vectors can have small sum residuals compatible with limited displayed precision. The historical analyses retain the allowed rounding tolerance and do not silently renormalize the maps or fill absent values with zero.

3.2 Exact graph tasks

A forest is a collection of rooted trees. Each non-root node has one immediate parent. Following the parent relation repeatedly reaches a unique root. Root questions ask for that terminal root. Ancestor questions ask for a node reached after a specified number of parent links. Direct-parent questions require a single local relationship.

Identifiers are opaque strings. Candidate labels identify explicit alternatives. An exact parser reconstructs the submitted parent relation; ordinary traversal checks the answer. Repeated questions, layouts, names, or requests from the same forest remain within that forest's statistical unit.

3.3 Four intervention classes

The distinction between reordering and computed preprocessing depends on how the order is chosen. Root grouping preserves the fact set but requires computing membership. Oracle path placement requires knowing which edges answer the question. A gain under these conditions is an effect of the complete preprocessing procedure, not evidence that the model independently recovered its inputs.

4. Evidence and experimental design

4.1 Cohorts

All requests use jev-1.13.0. The selected experiments were conducted on 18 September 2026 (UTC). One HTTP 504 in the rewiring cohort remains failed. The original queue resumed only untouched suffix jobs; the failed request was not replaced with a successful retry. The paired grouping endpoint therefore contains 15 complete worlds, while the other rewiring primary endpoint contains all 16.

The hypotheses and analysis rules were specified and frozen in local study manifests before the recorded first calls. These records are not an external preregistration or independent timestamp certification. The evidence download includes the protocols and freeze manifests, and the subsequent publication audit is explicitly post hoc.

4.2 Naming confirmation

Each forest contains 256 non-root nodes and four roots. Five consistent name assignments comprise the original assignment and four bijections. Two layouts—shuffled and grouped—are crossed with two exact-body copies. Each request contains six questions, but the four prespecified primary endpoints are root and fourth-ancestor tasks under each layout.

For one original request body and one renamed body, let the selected labels from two copies be A₁, A₂ and B₁, B₂, expressed in their corresponding abstract alternatives. Define cross-name disagreement C as the mean of the four indicators I(Aᵢ ≠ Bⱼ). Define matched repeat disagreement R as one half of I(A₁ ≠ A₂) + I(B₁ ≠ B₂). The endpoint is D = C − R, averaged over the four bijections within each forest.

All pairings from one forest are dependent and are averaged before inference. The subtraction estimates excess disagreement under this repeat design; it is not a direct measurement of a neural component. Both exact-body copies, all name assignments, and validated response structures are required for an endpoint's complete forest. All 96 forests satisfy this rule.

Under mutually independent serving copies with stationary selected-label distributions p and q, E[D] = ½ Σₖ(pₖ − qₖ)². These are distributions of selected labels across repeated requests, not the probability maps returned within an individual response. Algebraically, D is one half of the unbiased squared maximum mean discrepancy statistic using the equality kernel k(x,y) = I(x = y), with two observations per representation (Gretton et al., 2012). Finite-sample D can be negative. This identity clarifies the estimand; it does not establish independence or stationarity of the hosted service, or exact validity of a sign-flip test.

The intervention preserves the abstract problem and identifier inventory but does not fix the literal query names or each identifier's weighted occurrence count. Billed input-token ranges across the views within each forest vary from 7 to 80 tokens. Equal string lengths therefore do not establish equal tokenization. These limitations motivate the prepared follow-up.

4.3 Counterfactual rewiring

Each of 16 symmetric forests has four roots and 512 non-root nodes. Four corresponding parent values are cyclically reassigned across roots. The edits preserve acyclicity and root-basin sizes. Four target nodes and edit distances of 1, 4, 8, and 16 parent links are tested.

The stale-grouped view retains the baseline record order after replacing the parent values. It contains updated facts, not false root labels or obsolete edges. The refreshed-grouped view reorders those same edited facts using current membership. Shuffled views and unaffected-root questions provide additional controls. Exact-body serving copies are reported separately and excluded from primary inference.

The two prespecified contrasts are affected-root accuracy at distance 16 minus distance 1 under shuffled order, and refreshed minus stale grouping accuracy averaged across edit distances. The latter first averages targets and distances within each eligible forest. Baseline references reused across distances are not new observations.

Edit distance also changes the edited node, moved subtree, parent degree, and record geometry. The first contrast is therefore a controlled remote-edit comparison at fixed target depth, not an isolated measurement of propagation radius.

4.4 Locality confirmation

The 384 fresh forests each contain 512 parent records. Their queried root paths have eight edges. Four views cross compact versus dispersed path records and early versus middle placement. The relevant path is selected using the known graph, and unrelated records fill the remaining positions. Each world retains the same facts, question set, and billed token count across views.

The root endpoint's interaction is:

The historical two-test family includes root and fourth-ancestor interactions. Root candidates do not have the candidate-depth shortcut discussed below; the ancestor interpretation is narrower. This paper leads with root results and preserves the complete historical analysis in the download.

The root declaration stays at the front. Absolute path position and distance to that declaration are therefore not isolated. A separate, completed header-position control is reported in Section 6.4 rather than omitted from interpretation of the positive result.

5. Statistical analysis

The selected studies use paired, within-forest contrasts. Their historical procedures retain 10,000 forest-bootstrap resamples for intervals and 10,000 Monte Carlo sign flips, with a plus-one correction, for two-sided tests. The sign-flip interpretation relies on a symmetry or exchangeability assumption. In particular, a naming disagreement contrast need not have a symmetric null distribution. Its historical p-values are approximate and are not exact randomization p-values. Enumerating every sign pattern removes simulation error but does not remove that assumption.

Holm correction is applied within each study's fixed family: four naming endpoints, two rewiring endpoints, and two locality interactions. The 95% intervals are descriptive and are not simultaneous intervals across the entire adaptive research program. Absolute locality accuracy intervals use Wilson bounds. A cell with no observed errors is a finite-sample observation, not a guarantee of future correctness.

Attempts, unusable responses, duplicates, and frozen-plan mismatches are audited before analysis. Missing results are not assigned favorable labels. The rewiring grouping estimate uses its historical completeness rule; a separate post hoc missing-label sensitivity exists in the archive and is not substituted for the prespecified interval.

A post hoc independent audit reparsed every literal graph and question, verified 23,840 expected answers across the three main studies, and reconstructed every main effect without importing historical analyzers. It used 50,000 bootstrap draws, exact sign-flip enumeration, chronological-block sensitivity analyses, and a separate naming label-permutation sensitivity under a stronger serving-exchangeability assumption. These analyses support the principal effects but do not correct selection across the wider research program. See the audit supplement for results and reproducible commands.

The export preserves literal request and response JSON, relevant metadata, and failure status while omitting operational headers and machine-specific latency fields. Exported rows have their own hashes. The original source-log hashes and an explicit field inventory document the transformation. The methods and reproduction page specifies how to verify and recompute the selected analyses.

6. Results

6.1 Naming effects exceed exact-repeat disagreement

Each row contains 96 forests. The three very small p-values shown at 0.000400 reflect finite Monte Carlo resolution after correction, not exact probabilities reported to arbitrary precision. Unrounded Monte Carlo estimates are retained in the machine-readable output. The grouped-root contrast is sparse: 10 forests have positive effects, 2 negative, and 84 zero. Exact sign enumeration gives a raw p-value of 0.005859 for that contrast; it remains a smaller, less stable result than shuffled-root disagreement. The historical table is preserved rather than replaced by a selected sensitivity analysis.

All four signed accuracy-change intervals include zero. This result establishes instability under the intervention, not a dependable accuracy improvement from a specific name assignment. Direct-parent lookup remains correct across all 1,920 requests in this cohort.

6.2 Local update correctness and global answer correctness diverge

All 1,280 direct-parent observations in the rewiring study are correct: 512 baseline and 768 edited observations. Affected-root questions can nevertheless select the old root under stale grouping. At edit distance 4, stale grouping selects the old root in 56 of 64 complete target observations; at distance 8, it does so in 53 of 60.

Refreshing grouping improves affected-root accuracy by 72.08 percentage points [65.42, 78.33] across the 15 complete forests; Holm-adjusted p = 0.000200. The shuffled distance-16-minus-distance-1 contrast is 7.81 points [−4.69, 20.31] across 16 forests; adjusted p = 0.34487. The latter does not establish distance invariance.

Assigning each possible returned label to the one failed request gives full-16-forest point estimates between 71.09 and 71.48 points. No possible label for that request reverses this contrast within the frozen cohort. This post hoc sensitivity is a bound on point estimates, not a replacement confidence interval or a model of why the request failed.

The illustrative example in the article uses separate root and local-parent requests. Its selected option changes from correct a before editing, to incorrect a after editing in stale order, to correct c after regrouping. The three root calls have identical question and criterion bytes and report 8,819 input tokens. The example was selected after results were inspected and supplies no additional inferential test.

6.3 Path compactness interacts with position

Compact early, compact middle, dispersed early, and dispersed middle root accuracy is respectively 309/384, 157/384, 117/384, and 122/384. The compactness-by-position interaction is 40.89 percentage points [34.11, 47.66], with Holm-adjusted p = 0.000200.

Equal billed token counts exclude one simple account based solely on reported input size. They do not establish equal internal token positions or a particular attention mechanism. Relevant-path selection supplies exact graph knowledge that must be acknowledged when translating the result into an application.

6.4 Moving the root declaration does not resolve the positional explanation

A completed follow-up crosses early, middle, and late paths; compact and dispersed spacing; and a front or end root declaration in 48 new forests. All 576 requests succeeded. Each forest retains the same facts, questions, state length, and billed input size across its 12 views. Direct-parent answers are correct in all 576 requests.

For compact paths, root accuracy is 36/48 under either header position when the path is early. When the path is late, it is 23/48 with the front declaration and 15/48 with the end declaration. The locally specified late-minus-early difference in the effect of moving the declaration to the end is −16.67 points [−37.50, +4.17], with Holm-adjusted p = 0.389. Its two-test family also includes fourth ancestor, whose interaction is −2.08 points [−20.83, +14.58], adjusted p = 1.

The follow-up does not show the predicted improvement for late paths from a nearer root declaration. The interval is inconclusive; it neither proves no header effect nor establishes that moving the declaration harms performance. The question already names all four candidate roots, making the declaration redundant information. Other query-to-evidence distances and positional heuristics remain possible. This supporting cohort is outside the three-study total and has its own methods and evidence.

7. Shortcut audits and limits of interpretation

Some early ancestor and lowest-common-ancestor protocols made the gold candidate unique in depth. A query-blind procedure using candidate depth could therefore succeed without computing the requested relationship to the target. The root candidates in the selected locality study all have depth zero and balanced root-basin structure, so that specific objection does not explain its root contrast.

Stronger later candidates remove some structural asymmetries, but root grouping still admits block-membership cues. A refreshed layout can be useful and still fail to demonstrate edge-by-edge traversal. The rewiring experiment shows that factual updates and global decisions can dissociate under the tested views; it does not reveal whether the service uses block association, contextual retrieval, graph approximation, or another mechanism.

Name substitutions change contextual strings. The original design does not isolate identifier frequency, query names, or tokenization. Exact repeats provide a necessary reference but only a limited estimate of serving variability. Neither one later replay nor two copies per body determines a general stochastic law.

The synthetic forests are narrow diagnostic distributions. Their controlled geometry and exact truth are strengths for intervention analysis and limitations for deployment claims. The selected studies do not cover arbitrary cyclic graphs, uncertain edges, open-ended answer spaces, natural document variation, or specialist domains. Repeated use of a model identifier also does not establish that every hidden service component remained unchanged.

8. Practical implications and supporting evidence

If an application already has an exact graph, compute root and parent relations in code. If a semantic model needs evidence from that graph, evaluate both the evidence-selection stage and the semantic decision. The model should not receive credit for rediscovering a relationship explicitly supplied by an index or an oracle-selected path.

Supporting application studies in the broader campaign improve exact ownership lookup and reduce context size through deterministic retrieval. The harder semantic corpus does not demonstrate an accuracy improvement or a formal noninferiority result. The publication therefore makes no general claim that retrieval preserves or improves semantic accuracy.

A separate confidence-gate study accepts 45 incorrect roots among 261 accepted answers at a fixed threshold of 0.9. These are repeated forest views, and the same policy behaves differently on another cohort. This supports workload-specific evaluation of coverage and error rather than a universal threshold recommendation. The supporting result is outside the three-study evidence inventory and is summarized in the supporting material.

9. Prospective studies

The prepared study materials describe a new stage of research. They contain no live results. The renaming study fixes query anchors and matches identifier role frequencies. The layout study introduces old-root, current-root, third-root, and neutral cues while preserving literal factual truth. The application package prepares source-resolution and semantic-decision comparisons, development fixtures, and a process for constructing an independent held-out corpus.

Final comparative claims require frozen adapters, version-pinned comparator models, comparable development effort, and fresh model measurements. Baseline model access and an execution budget are not established by preparing offline requests. Software fixtures used to test analyzers are explicitly synthetic and cannot enter empirical estimates.

10. Conclusion

Jev's structured decisions are sensitive to meaning-preserving naming changes and to how relevant graph facts are arranged. Correct local responses can coexist with incorrect global responses after factual edits. Some representation improvements expose relationships that ordinary code has already computed, so their success does not establish an internal reasoning procedure.

The evidence supports explicit evaluation of representation invariance, factual-update consistency, and preprocessing assistance. The prepared follow-ups are intended to determine which explanations survive stronger controls and whether the resulting guidance improves decisions on independent application cases.

Data and reproducibility statement

The evidence page provides the selected historical requests and responses, complete planned cohorts, frozen analysis source dependencies, compact publication tables, and file hashes. The package is designed for offline use and contains no API credential. Historical evidence is pinned to repository commit 149932ddd4e09fcdfcb7833c6dea7955fa5c5985. The publication record states authorship, internal review, rights, and correspondence details. This is a technical report, not an externally peer-reviewed publication.