Methods and reproduction

What can be reproduced

The evidence download contains the naming, rewiring, and locality studies presented in the article and paper. It includes every planned request and all 6,560 attempted responses from those studies. One response failed and remains marked as failed. It also includes four separately identified supporting cohorts: the header-position control, confidence path study, initial application pilot, and harder policy pilot. Across these seven packaged cohorts there are 9,488 attempts, including two failures; these accounting totals are not a pooled sample or test. The package reproduces the saved-data analyses; it does not reproduce the hosted model's future behavior.

The independent sampling units are 96 naming forests, 16 rewiring forests, and 384 locality forests. Multiple questions, layouts, renamings, or copies from a forest do not enlarge its independent sample. The paired rewiring grouping comparison retains 15 complete forests because one required request failed.

Download and verify

Download the evidence archive and its file manifest. The archive size and SHA-256 are recorded in the download summary.

Extract the archive, then create a local analysis environment:

tar -xzf representation-evidence.tar.gz
cd representation-evidence
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python reproduce.py --verify-only

The historical analyses used Python 3.9.6 and NumPy 2.0.2. The included requirement pins NumPy. Dependency installation requires a package download; the verifier and analyses themselves require no network access and read no API credential.

The verifier compares every packaged file with its recorded hash. The manifest also identifies the original source plans, logs, and analysis hashes. Hash verification checks file identity; it does not independently establish that the experimental design supports every possible interpretation.

Recompute the selected analyses

Use a new output directory:

.venv/bin/python reproduce.py --output reproduced

The command runs the historical analyzers on all seven exported plans and response logs and compares the relevant result sections with the recorded outputs. These sections include the paired per-world values, accuracy cells, uncertainty intervals, primary tests, and repeat summaries where applicable. Application latency endpoints are excluded because operational timing fields are intentionally outside this export. The confidence analysis evaluates the frozen gate without refitting it. The command writes verification.json after all comparisons succeed. Existing output directories are rejected to prevent accidental replacement of a prior reproduction.

The export changes source paths and removes operational metadata, so a complete analysis-file hash need not equal the original file hash. The verifier compares the relevant numerical structures directly. The preparation verification record records the result of this check on the publication preparation machine.

How the export preserves evidence

The package uses an explicit field selection rather than copying an entire working directory. Each exported row retains its study name, literal request, response when available, expected-answer metadata, status, request-start timestamps, validation state, and relevant evidence flags. Request and response object values and insertion order are checked against the source rows during export.

Operational HTTP headers, machine-specific connection fields, and client timing measurements are omitted. This package therefore does not support reanalysis of the separate latency experiments. Frozen request plans, protocol text, and the required historical Python source files are copied byte-for-byte. Machine-specific paths in expected analysis output, provenance JSON, and review reports are normalized without changing numerical values. The manifest describes these transformations and records both source and export hashes.

The naming and rewiring protocol hashes match their locally frozen manifests. The locality/header manifests embed their prospective comparisons but do not hash-bind the explanatory Markdown protocol, whose current version includes a disclosed post-freeze shortcut note. The evidence does not constitute external preregistration. The independent audit reports recorded freeze/dispatch chronology and supplies separate literal-answer and statistical checks.

The export does not impute a response for the failed rewiring call, replace an unsuccessful request with a retry, or omit an unfavorable completed condition. The selected studies are a focused subset of the larger exploratory campaign, chosen for the publication narrative after completion.

Interpreting the statistics

Paired effects

Each intervention is compared within the same forest before effects are aggregated across forests. For naming, all relevant name pairings and serving copies are averaged within a forest. For rewiring, the primary grouping effect averages targets and edit distances within each complete forest. For locality, each forest contributes one compactness-by-position interaction.

Treating the underlying API answers as independent would alter the uncertainty and often the estimand. Request count is an accounting quantity; forest count is the primary sampling quantity for these tests.

Intervals and test families

The historical analyses use 10,000 forest-bootstrap resamples and 10,000 Monte Carlo sign flips for their paired endpoints. A plus-one correction prevents a simulated p-value from being reported as zero. Holm correction applies within each study's prespecified test family. There is no claim of one correction over every exploration in the broader campaign.

The sign-flip tests require a symmetry or exchangeability approximation that is not guaranteed for the naming statistic. Enumerating all sign assignments removes Monte Carlo error but does not make this an exact randomization test of equal serving distributions. The post hoc review reports additional permutations under a stronger serving-exchangeability assumption. Neither procedure verifies unobserved service independence.

The intervals shown for individual locality accuracy cells are Wilson intervals. The interaction uses paired forest differences and a bootstrap interval. These summaries answer different questions and should not be interchanged. The displayed intervals are descriptive, not simultaneous confidence bands for all results.

Missing data and repeats

The primary analyses retain the original completeness rules. A failed endpoint can remove a forest from one comparison while leaving it eligible for another. The manifest and exported status fields make this visible. Exact request copies estimate serving variation under the tested schedule; they are not new semantic examples.

Probability coordinates

The primary findings concern selected labels and exact correctness. Probability coordinates and the separate confidence field are retained for analysis but have different interpretations. Small printed probability-sum residuals are handled using the historical rounding tolerance. The data are not silently renormalized, and absent values are not replaced by zero.

Inspect the worked example

The download's example/ directory contains five historical request/response pairs: original root, edited root in stale order, edited root after regrouping, original parent, and edited parent. A provenance file records the underlying graph facts and selected records.

The example was chosen after the completed cohort was inspected. It illustrates the failure mode but provides no new statistical evidence. The calls were separate requests; the display order is explanatory, not a conversational sequence. The selected option's probability and the separate confidence field remain separate.

Live replication is a different experiment

Running the same request against the service later would create a new observation under new service conditions. It requires a current API contract, an explicit execution budget, model/version records, and a new run identifier. The download contains no automatic live runner. The prospective study materials prepare new experiments without treating software fixtures as model results.