Programming with Classifiers

A small experiment in grounded execution

A model does not have to write a program to help execute one.

Suppose it can answer a bounded question: which of these observations matters, which action should happen next, or which proposed interpretation best fits the evidence? Ordinary code can construct the alternatives, perform the selected operation, and present its result for another judgment. The sequence produces behavior even though the model never writes a command or a paragraph.

We explored this idea with Jev, TypeSafe's structured-decision model. Our most useful result was narrower than a general agent: a small evidence-retrieval operation that decides which source code to inspect and returns exact citations. We call the worked example Lecter.

This is an exploratory implementation and a set of experiments. We have not established a new learning architecture, a general reasoning system, or a novel algorithm. The interesting question is what becomes practical when contextual judgment is available as an ordinary component inside a program.

Start with a concrete problem

Consider a report that request teardown happens before a Flask streaming response is consumed. An investigator needs more than the function whose name mentions streaming. The explanation also depends on where the WSGI application pops the request context and what context cleanup does.

A repository contains the relevant bodies, related wrappers, declarations, and plenty of unrelated code. Returning everything is expensive for whoever reads the result. Returning only the most obviously named helper can omit the connection that explains the behavior.

Our example makes that investigation visible. It first considers compact descriptions of candidate fragments. A judgment selects which exact bodies to reveal. A second judgment sees those bodies and filters the evidence before the runtime writes a bounded packet of source citations.

The packet is not a generated explanation. It contains original source bytes and references that can be checked. In the recorded streaming case, the selected packet includes the streaming helper, WSGI cleanup, and request-context cleanup. In the original experiment, a cheaper variant that exported ranked fragments without the body judgment omitted an important cleanup site under the same output limit. The standalone demo uses a leaner packet format and packs its metadata-only comparison in score order. That comparison now includes the site too; packing the same shortlist in its original input order still omits it. The demo therefore illustrates a smaller, more focused packet, not an unconditional completeness advantage. Output format and packing order are part of the algorithm.

That is the central mechanism: a judgment changes what the next judgment can observe.

The idea has a history

Repeated classifier decisions are an established way to construct structured outputs. SEARN explicitly connects classification to decisions along search trajectories. Neural Programmer-Interpreters train a controller to choose compositional subprograms and arguments. These are relevant precedents, not evidence that an off-the-shelf classifier inherits their training or guarantees.

There is also substantial work on programming around models. LMQL combines scripting and output constraints; DSPy represents and optimizes model pipelines. Our draft is much narrower: a bounded evidence workflow with explicit source references, observation records, provider calls, and replay. Its value has to come from how well that concrete operation works.

The program supplies the possibilities

The API we used accepts state and questions with defined answer alternatives. It returns structured judgments. We use that interface as a classifier; this does not tell us whether the underlying model is an encoder, a decoder, or something else.

A classifier cannot choose an option the application never supplied. This is the main design constraint, rather than a detail to hide in a prompt. The host must provide a useful vocabulary of operations and construct valid arguments from the available state.

In the research host, ordinary code performs parsing, lexical retrieval, bounded traversal of source relationships, reference validation, and output packing. The standalone draft receives a caller-supplied candidate list; it does not yet reproduce the repository search stage. The model judges relevance among the resulting candidates. We do not ask it to invent a shell command, a filename, or a source citation.

The reference operation is:

1. Discover candidate references.
2. Observe compact descriptions.
3. Judge relevance.
4. Rank a bounded shortlist.
5. Read exact source bodies.
6. Judge relevance again.
7. Export cited source.

This is a small program with a model-dependent intermediate value. It is not evidence that the model independently discovered the procedure. The grammar, candidate generator, and interpreter contribute substantial structure.

References, observations, and judgments are different values

The filesystem is useful here because the runtime can retain more information than it sends to the model. A source reference identifies a particular snapshot and byte range. Passing that reference does not mean the model has read its contents.

An observation records the actual content made visible for a judgment. A compact description is an observation of metadata, not an observation of the complete implementation. A later operation can dereference the same source and reveal its body.

A judgment records what the provider answered about those observations. Its score can influence ranking; it does not change the underlying source facts. Finally, deterministic execution records which operation consumed those values and what it produced.

These distinctions make failures inspectable. We can ask whether a required body was never discovered, ranked below the inspection limit, rejected after inspection, or removed when packing the output. Each failure calls for a different remedy.

They also provide useful implementation boundaries. Storage owns immutable references. The provider owns judgments. The interpreter owns legal transitions and budgets. A trace records their interaction. Replaying that trace can verify execution consistency, but cannot establish that an answer was correct or that a remote provider originally produced the saved response.

A distribution is useful without being a belief state

Keeping scores instead of immediately choosing a single winner lets the program reserve a shortlist for inspection. One practical detail mattered: candidates categorically labeled “drop” could still belong in the best available sixteen when ranked by their keep scores.

That does not make those scores calibrated probabilities of truth or future success. In a controlled diagnostic, changing which fragments shared a batch changed scores and shortlist membership more than repeating identical requests. Exact repeats also varied. All reachable required evidence survived the top-sixteen cutoff in that sample, but the individual scores were not stable properties of the source fragments.

A separate retrospective ablation (archive member release_validation/score_ablation/RESULTS.md) sharpens that limit. With the same 64 candidates and 16 inspection slots on the saved Flask cohort, numerical ranking and binary keep/drop labels ordered by the lexical baseline both retained complete required evidence in 32 of 32 answerable cases. Lexical ordering alone retained 27 of 32. Numerical scores added no observed coverage benefit over labels at that cutoff. This did not test counterfactual body-stage outcomes, and the eight unavailable cases were reported separately rather than credited as successes.

The tie at sixteen prompted a further exploratory check of every cutoff from one through sixteen, with the same candidates and recorded judgments. The full curve (archive member release_validation/score_ablation/cutoff_curve/RESULTS.md) shows a budget-dependent result: numerical ranking covers more cases than labels alone at cutoffs 2–10 and 14–15, fewer at one, and the same number at 11–13 and sixteen. At eight slots the counts are 31/32 versus 28/32; at one they are 4/32 versus 6/32, where only six required sets can fit a single slot. This is evidence for testing numerical ordering under constrained inspection budgets, not a selected optimum or an end-to-end cost saving.

The figure shows all cutoffs; CSV values and a standalone plotting script are included. These are the same reused 32 answerable cases throughout, with eight unavailable cases excluded from this coverage denominator.

Our supported use is therefore modest: ranking under an explicit inspection budget. We did not demonstrate an advantage from multiplying probabilities through a tree, sampling possible futures, or treating a distribution as a self-maintaining belief system.

What the experiments changed our minds about

We began with broader ideas about classifier-directed reasoning and exploration. Several results pushed us toward simpler designs.

On a fresh Flask source cohort, compact ranking followed by body selection preserved all 32 answerable evidence sets, matching direct body selection from 64 candidates. After charging both stages, it used 3.3% fewer input tokens. The cheaper lexical-top-sixteen baseline preserved 27 of 32. This supports a useful operation, not a dramatic efficiency claim.

Searching more was not consistently better. In a reused diagnostic sample, increasing the candidate pool from 64 to 128 raised complete packets from 21 of 24 to 23 of 24. Increasing it again to 256 reduced that to 20 of 24, with more computation. Some newly competitive fragments displaced required evidence during ranking or output packing.

Better ordinary code helped. Adding bounded candidate links for explicit ancestor calls and context-manager members repaired three known discovery failures at the original candidate limit. But eight newly worded, targeted questions remained at five successes for both versions. The repair was real; broad transfer was not demonstrated.

An additional semantic choice about where to start following references did not earn its cost. Both lexical and model-chosen starting points found all required candidate evidence in that sample. The adaptive variant used about 37% more input tokens. Its one final-output advantage came from downstream packing, so we could not attribute it to improved navigation.

We also compared a conventional local MiniLM passage reranker under the same candidate and output limits. It did not outperform Jev's compact ranking on this cohort. Covering whole source bodies with overlapping windows did not improve that baseline. The comparison remains limited by its general-text training, different context structure, and reused questions; it is not a claim of superiority over strong code-retrieval systems.

The studies use small, often overlapping and source-correlated cohorts. Some questions were authored from source; the public-issue pilot included reporter-provided symbols and deliberately narrowed local questions. Counts should not be pooled into an overall accuracy, and the results do not establish a scaling law.

What the first draft is for

The accompanying prototype is intended to make this mechanism tangible: a versioned operation format, explicit observation boundaries, interchangeable judgment providers, deterministic host steps, and a trace you can inspect and replay.

The public-source demo uses recorded judgments from the investigation. Its documentation distinguishes the original experiment from the fixture translation used by this package. A separate scripted example demonstrates the interface without claiming model performance. Neither requires a credential to explore.

You can try the runnable companion without model credentials. After unpacking it, run the scripted example from the lecter-companion/ directory:

PYTHONPATH=src python3 examples/run_example.py scripted_checkout

The recorded Flask example does not bundle the full upstream Flask source files. Restore the sixteen exact public files from their pinned revision, verify their hashes, and then run the example:

python3 restore_flask.py --fetch
PYTHONPATH=src python3 examples/run_example.py flask_streaming

The restore command downloads public source files; it does not call a model. The companion contains selected original request/response records, the translated trace, and documentation of the translation. Its example walkthrough is examples/flask_streaming/README.md, and its root README.md describes the scope and commands. Neither the scripted example nor recorded replay establishes new model performance. Replay alone does not answer new questions.

The research draft also includes a local source preparer, a custom-provider example, and a pure TypeSafe request/response codec. These expose the path to a new question without hiding a network call in a demo. An actual service connection still needs a host that owns dispatch, accounting and failure handling. The public companion is a selected runnable sample, not the full research repository.

We also checked that the portable interpreter was not merely shaped around the displayed example. An offline conformance check (archive member release_validation/RESULTS.md) translated all 40 saved Flask cases and independently verified 3,200 label/score pairs. Candidate ordering, inspection selections and deterministic replay matched in all cases. Final packet membership matched in 32 of 40: the changed projection format affected the other eight, while output presentation order also differed. A further fixed 16-case Requests/Click check (archive member release_validation/cross_package/RESULTS.md) also passed after an explicit input-schema adaptation; the initial schema failures are preserved. These checks validate translation and execution rules, not fresh accuracy results or an improved search policy.

The design is deliberately small. We want a later implementation, including one in Rust, to reproduce explicit behavior rather than inherit undocumented Python conventions. This is not a complete general-purpose agent language, and the research does not justify unrestricted actions or confidence-based completion claims.

The next meaningful test is not another elaborate swarm. It is whether this operation helps people answer new, independently written questions with less effort, at acceptable latency and cost, compared with strong retrieval alternatives.

If the model supplies only judgments, the rest of the system becomes harder to wave away. Someone must define the alternatives, preserve the observations, execute the choices, and decide what counts as success. Making those responsibilities explicit is the most promising part of programming with classifiers.

Evidence and related work

The public evidence archive contains selected reports and results, not the complete original research repository. Some references inside those reports point to private original files that are not included in this curated export. The runnable companion contains four selected original request/response records, not the complete set of experimental calls. These downloads make the stated subset inspectable without implying that every underlying artifact is publicly available.

The research report (archive member overnight/MORNING_REPORT.md) links the frozen protocols, source audits, negative results, costs, and reproduction records. The operating guide (archive member overnight/OPERATING_GUIDE.md) describes the experimental practices supported by those results.

The release's prior-art notes (archive member lecter-companion/docs/PRIOR_ART.md) distinguish classifier-guided execution from related work on model programming, tool use, search, and program synthesis. This article proposes a concrete interface and reports an exploration; it does not claim that classification, modular execution, or guided search is new.

The OpenProse-authored article and supporting materials are released under the MIT license. Third-party excerpts retain their original notices.