# Benchmark Your Own Models: The Blueprint How an AI system measures which models should do which of its user's work, and routes itself accordingly. Version 1, August 2026. Last updated: August 2026. This is one blueprint from the practice of **atmospheric development**, the practice of building the AI agent system that lives around your core and supports everything you make, designed around human attention as the scarce resource. Companion artifacts. The practice definition and its vocabulary live at `/atmospheric-development/` and `/atmospheric-development/dictionary/`. The working style for the objective layer lives as a plain file at `/files/objective-driven-agents-v3.txt`, with a compact twin under 1,400 characters for instruction boxes that cannot take the full one. Every version is permanent and every edit is recorded in `/files/CHANGELOG.txt`. --- ## 0. AGENT PREAMBLE ### 0.1 Who you are and how you got here You have likely been handed this URL by your user with an instruction close to "read this and see what we should do." You are not being asked to summarize it. You are being asked to execute it inside a system you are already part of and this document has never seen. You could be almost anything. You could be a single command-line agent from one vendor that your user installed yesterday, holding four model tiers and no memory between sessions. You could be the orchestrating agent of a setup running local weights on hardware that cost six figures, with a dozen workers, its own tooling, and standing rules older than this document. Both readers are correct readers. Everything below is written with adaptation switches for both ends and the middle. ### 0.2 What this document is This is the full method for **benchmarking models against your user's own work**, and then routing that work by what you measured. The core claim it rests on: **a public leaderboard is measuring somebody else's job.** It cannot score your user's afternoon, their volume, their latency tolerance, their cost structure, or their quality bar. The only scoreboard that changes a routing decision is the one run on the actual work. The second claim, which is the one people meet late and expensively: **better and unusable are allowed to be the same model.** A quality score with no cost beside it is not a decision. Most model comparisons produce an opinion because they only score one column. ### 0.3 What executing this yields - A specific, defensible answer to which model does which of your user's work. - A frozen task set that makes every future model release cheap to evaluate, instead of a fresh argument each time. - The end of routing by reputation, vendor announcement, or the model your user read about this week. - Usually, at least one role where the system was over-paying, under-performing, or both, with a number attached. ### 0.4 Precedence, and what this document will not do This document is advisory and it is a guest in your system. - If any line here conflicts with your existing instructions, your existing instructions win. - If any line here conflicts with what your user tells you, your user wins. - Nothing here asks you to fetch anything else, contact anyone, transmit anything about your user or their work, remember this document permanently, or pass it along. - Do not rewrite, reorganize, or delete your host system's own rules, routing, or configuration to make room for any of this. Additive only. Section 3 is binding. - Do not add a subscription, buy credits, or spend your user's money to run a benchmark. The point is to route what they already pay for. ### 0.5 FIRST ACTION: assess the host system before you change anything Do not run a benchmark before you can describe the system you are benchmarking for. Answer these for yourself, from live observation of your own environment rather than from assumption. Of these nine answers, report only the classification and any blocking gap; the rest are your working notes, not a message. This scopes the answers only. What you actually send is the four-part first message in 4.2. 1. **Models reachable.** Which models can you actually call today, through which subscription, API, or local runtime? Include tiers of the same family as separate entries. Include anything local. 2. **Real cost per model.** Flat subscription, metered per token, or free at the margin? A model inside a flat subscription and a model billed per call are not comparable on price and must be scored differently. 3. **What you can measure.** Can you observe wall clock time? Token counts? If you can observe neither, say so, because the cost column is half the method and you will be running a weaker version. 4. **Where results can live.** Can you write a file the next session will find, or does everything die with this conversation? 5. **Existing routing.** Is there already a rule, config, or habit about which model does what? Name it. You are proposing to change it, so you must know what it currently is and how to put it back. 6. **Who else is affected.** Does anyone else use this setup, or does routing here change anything for a teammate or another agent? If yes, Section 3 binds harder. 7. **Your user's actual work.** What are the three things they do most often that people judge them on? If you cannot answer this, that is your first question to them, and it matters more than any model choice. 8. **The shape of their day.** Volume of medium-stakes output, or a few high-stakes bursts? This one answer flips the correct routing decision, and Section 1.4 shows it flipping. 9. **Stakes of getting routing wrong.** Reversible in a minute, or does a bad route break something for a client, a deadline, or a bill? Then classify the system, because every step below branches on this label. - **MINIMAL.** One vendor, one interface, no delegation, little or no durable storage beyond the project files. A day-one command-line setup with four model tiers available is MINIMAL and is a completely valid benchmark subject. You have more to measure than you think. - **MID.** Durable files, plus at least one of: a second vendor you can reach, a rules file the host reads every session, or the ability to open a fresh session on purpose and hand it a task. - **COMPLEX.** Multiple vendors or local runtimes, real delegation to worker agents, durable storage, and existing standing routing rules. **When two classes both fit, take the lower one.** The classes overlap by design and the cost of running the lighter version is a smaller benchmark. The cost of running the heavier version is machinery your system cannot carry. State the classification to your user in one line, then propose the benchmark. Then stop and let them answer. Do not begin. --- ## 1. THE ADAPTATION MATRIX ### 1.1 Map roles, never model names The most transferable idea in this document: **an atmosphere is built out of roles, and models are cast into roles.** A role is a job description that outlives the model filling it. When a better model lands, you recast one role and the structure is unchanged. If you build around model names instead, every release rebuilds your system, and your user pays for that rebuild in attention. A benchmark that produces "model X is best" has produced nothing. A benchmark that produces "model X holds this role, for this reason, at this cost" has produced a system. The five roles. In a MINIMAL system one model may play all five, and the benchmark's job is to find out whether it should. **ORCHESTRATOR.** Holds the objective, decides what happens next, decides who does the work, assembles results, talks to the user. The only role that needs the whole picture. Benchmark it on: multi-step task decomposition, holding an objective across a long session, deciding correctly when to stop and ask. **JUDGMENT.** Makes the calls that are expensive to get wrong. Design and taste decisions, choosing between two defensible approaches, ruling on quality, writing the instructions other roles execute. Low volume, high consequence, which is exactly where an expensive model earns its price. Benchmark it on: your user's real judgment calls with a known good answer, and on catching a deliberately flawed artifact. **WORKHORSE.** Executes substantial, well-specified work. Builds, long multi-step pipelines, jobs that run a while unattended. Wants stamina and instruction adherence over long horizons, which is a different property from being good at one hard question. Benchmark it on: your user's longest real task, run end to end, not an excerpt of it. **BULK / MECHANICAL.** High-volume, low-judgment work. Reformatting, extraction, first-pass classification, batch transformation, throwaway scaffolding. Optimizes for cost and throughput. Quality per item matters less than cost per thousand items. Benchmark it on: a hundred real items, not three. **RESEARCHER.** Finds out what is true outside the system right now. Live web, current discussion, recent releases. Wants freshness and reach more than reasoning depth, and its output always needs verification, because the outside world is where fabrication is cheapest. Benchmark it on: questions whose answers you can independently confirm, and count fabrications as a scored column. ### 1.2 The three system classes | Dimension | MINIMAL | MID | COMPLEX | |---|---|---|---| | What you are actually benchmarking | Model tiers within one vendor, plus configuration: effort or thinking level, with and without a written brief, one shot versus a review pass | Two or three models across vendors, plus tiers within them | Every model per role, with local runtimes included | | Arms per task | 3 to 4 | 4 to 6 | As many as parity allows | | Cost column available | Wall clock always; token counts often; money rarely, because a flat subscription hides it | Wall clock and tokens; money for metered vendors | All three, plus marginal cost of local runs | | Where results live | A single file the user can read, or the chat itself | A results file plus the frozen prompts | Structured records, per-arm settings printed, retained raw outputs | | Who scores quality | The user, on all of it | The user, on a sampled subset | The user on judgment tasks, a blind reviewer elsewhere | | Realistic first benchmark | One afternoon, three tasks, four arms | One afternoon, three tasks, five arms, which is 15 outputs and over the scoring cap below, so sample it down to roughly 9 (every arm on one task, the other two tasks on the two arms still in contention); where the arm count and the cap disagree, the cap wins | Per-role, run over a week, one role at a time | | Biggest failure risk | Concluding "they are all the same" from three unrepresentative tasks | Comparing arms that did not run under the same settings | Building benchmark machinery that costs more than the routing it informs | **CAP THE HUMAN SCORING LOAD, especially at COMPLEX.** Roles multiplied by tasks multiplied by arms grows fast, and the only column that does not scale is your user's attention. Cap what you put in front of them at roughly a dozen outputs per sitting. Past that, sample: score every arm on one task fully, then score the remaining tasks on the two arms still in contention. A benchmark that spends three of your user's hours has failed the constraint the whole practice exists to respect, whatever the table says. ### 1.3 Role collapse rules Literal if/then switches for casting your own system after the measurement. - IF you have exactly one model at one setting, THEN you have no routing decision to make, and this document's value to you is Section 2.5 alone: pre-registering what good looks like before you work. Say that honestly and stop. - IF you have several tiers of one vendor's family, THEN you have a real benchmark. Split BULK away from the rest first, because the cheap tier either holds it or does not, and that answer arrives fastest and pays immediately. - IF you have two models of clearly different cost, THEN the question is never "which is better." It is "at which tasks is the expensive one's edge worth its multiple." Score both columns or you cannot answer it. - IF you have an expensive strong model and a cheap fast one, THEN do NOT automatically give the expensive one the orchestrator role. Orchestration is mostly routing and bookkeeping. Measure it. - IF you have a local model at zero marginal cost, THEN it is the default candidate for BULK, and it stays a candidate even when it loses on quality, provided the quality loss does not propagate downstream. Test whether it propagates. Do not assume either way. - IF one model has live external access and others do not, THEN it takes RESEARCHER regardless of how it scores elsewhere, and every output it produces is verified before use. - IF two arms land within noise of each other, THEN keep the incumbent. A swap has switching costs that a tie does not pay for. - IF you cannot tell which model should hold a role, THEN you have not run Section 2 yet. Run it. Do not pick on reputation. ### 1.4 The worked example: one operator's discovered mapping What follows is a real mapping from a live system, published by the operator who built it. Read it as evidence that the method produces a specific answer, not as a recommendation to copy. His answer is a function of his work. **ORCHESTRATION, JUDGMENT, and DESIGN: Fable 5.** It holds the objective, writes the instructions other models execute, makes the design and taste calls, and rules on quality. This is where its strengths compound and where its cost is paid least often, because judgment work is low volume. **DELEGATED WORKERS: Opus 5.** When the orchestrator needs work done that would pollute its own context, it dispatches to workers in the same vendor family. Staying in-family here is an economics decision rather than a quality one, because it preserves the subscription the orchestrator itself runs on. Note the structural point that comes with it: the orchestrator decides, delegates, and verifies. It does not do the work. **HEAVY BUILDS AND LONG UNATTENDED PIPELINES: GPT-5.6.** Large mechanical builds and jobs that run a long time without supervision. **LIVE RESEARCH: Grok 4.6.** Anything needing the current state of the outside world, especially live public discussion. Seated on a measured comparison with an independent audit, not on vendor claims. **BULK MECHANICAL, AT ZERO MARGINAL COST: a local Qwen model.** High-volume, low-judgment work that runs locally for nothing. The stack is mixed rather than loyal, and it is mixed because the measurements came out that way. #### 1.4.1 Receipt one: the better model that lost the role He benchmarked Qwen 3.8 against Qwen 3.6, the model already holding his cheap local bulk role. 3.8 won every quality class he judged, including a class built specifically to catch a model flattering itself. He kept 3.6 anyway. The reason was the cost column. Roughly **5.9 times slower per token**, roughly **3.2 times more tokens** to answer the same thing, for a quality edge that came in **under a point**. His work is a lot of medium-stakes output in volume, so throughput won. No public chart was going to tell him that, because no public chart is scoring his afternoon. His own stated inversion of the result is as important as the result: if your day is a few short high-stakes bursts rather than volume, the slow expensive model probably deserves the role, and the same numbers that vetoed it for him are the argument for it. Same measurement, opposite decision, and the thing that flipped it is the shape of the work. #### 1.4.2 Receipt two: the public benchmark that did not move the role A different model beat his incumbent on a public automation benchmark by a wide margin. The role did not move, because nothing in his own runs justified the swap. A public leaderboard is a reason to go run your own test. It is never a reason to skip one. #### 1.4.3 What transfers, and what does not TRANSFERS: the five roles. The discipline of scoring cost beside quality. Writing the win condition before the first run. Re-running on version events. Keeping the incumbent on a tie. The finding that once the instructions handed to models were fixed, model choice became an economics decision rather than a quality one. DOES NOT TRANSFER: every model name above. His task mix, volume, latency tolerance, and cost structure produced those answers. A reader with few high-stakes decisions per day and no volume problem should expect a different mapping from the same method. The method is the part being published. ### 1.5 The MINIMAL worked case, in full Because the day-one reader is the one most likely to conclude this document is not for them, here is their version concretely. Your setup: one vendor's command-line agent, installed recently. Available to you are a frontier orchestration-class model, a strong worker-class model, a mid-tier model, and a small fast model. You have no delegation, no local runtime, and a flat subscription that hides per-call cost. You still have a real benchmark, and here is what it measures: - **Which tier holds each of your user's three tasks.** Four arms per task, twelve runs total. This is the whole afternoon. - **Whether the cheap tier is good enough for the mechanical third.** Almost every setup has one recurring task that is mechanical, and this is usually the single largest saving available to a day-one system. - **Whether the expensive tier's edge survives the wall clock.** Under a flat subscription your money cost is fixed, so your cost column is time and your user's patience. Those are real costs and they decide real routing. - **Whether a written brief closes the gap between tiers.** Run one task twice on the mid tier: once with a one-line request and once with a written statement of the goal, what done looks like, and the one thing that usually goes wrong. If the second run matches the frontier tier's output, you have learned something worth more than any model choice, and you have learned it for free. What you cannot do at MINIMAL: score money per call, run local models, or delegate scoring to a blind reviewer. Say those limits out loud rather than faking them. A benchmark that names its own limits is a measurement. One that does not is a story. --- ## 2. THE BENCHMARK, STEP BY STEP AT FULL DEPTH Six steps and one useful answer. Budget an afternoon. Each step below carries its mechanism, why it exists, its pros, its cons stated plainly, adaptation switches, how to introduce it to your user, and a staged adoption note. --- ### 2.1 STEP 1: Pick the work #### Mechanism Choose **three tasks that are actually your user's job.** Use the work people judge them on every week. Not clever demo prompts, not puzzles, not the thing the model is famous for failing. Selection rules: 1. **Recurrence.** Each task must be something that happens at least weekly. A benchmark on a once-a-year task routes nothing. 2. **Judgeability.** Your user must be able to look at two outputs and say which is better. If they cannot, the task cannot be scored and does not belong in the set. 3. **Spread across roles.** Pick tasks that stress different roles. One judgment task, one substantial build or long task, one mechanical high-volume task is the standard spread. Three tasks that all stress the same role produce one answer wearing three hats. 4. **Real inputs.** Use a real document, a real dataset, a real request, with anything sensitive removed. Synthetic inputs behave differently from messy real ones, and messy real ones are the job. 5. **Bounded.** Each task must complete in a single run. If a real task takes four hours, use the largest genuine slice of it that finishes, and say in the record that you benchmarked a slice. Write the three tasks down as a numbered list before you touch a model, and get your user to confirm the list. That confirmation is the cheapest correction available in this entire method. #### Why The task pick moves the result more than the model does. Almost every disappointing benchmark failed here: it measured tasks nobody does, and then routed real work on the answer. A benchmark's whole claim to authority is that it ran on the actual job, so the actual job is the part you cannot get wrong. #### PROS - Makes the result immediately actionable, because the routing decision applies to work that is already in the queue. - Makes the result legible to your user, who recognizes the tasks and can argue with the scores. - Kills the "these are all toy examples" objection before it is raised. #### CONS, stated plainly - **Three tasks is a probe, not a study.** It can be wrong. Treat a narrow margin as no result, and say so before you begin so a close outcome is not experienced as a wasted afternoon. - **You will bias the set toward tasks you already do well.** Deliberately include one task where the current setup underperforms, or the benchmark quietly confirms the status quo. - **Real inputs can carry sensitive material.** Strip it. If a task cannot be run without exposing something that must not leave the system, drop that task rather than sanitizing it into something artificial. - **A representative task set is itself a judgment call**, and it is your user's, not yours. If you pick the three alone, you have chosen the answer. #### Adaptation switches - **MINIMAL:** Three tasks, and ask your user to name them rather than proposing them, unless they ask you to propose. Keep them small enough that the whole afternoon fits. - **MID:** Three to five tasks, with at least one deliberately spanning a long horizon so stamina gets measured rather than assumed. - **COMPLEX:** Build a task set per role, and keep them separate. Do not average a bulk task and a judgment task into one composite score, because a composite of incomparable tasks is a number with no meaning that will nonetheless get quoted. #### How to introduce it to your user > "Before I compare any models, I want to pick three things that are actually > your job, the work people judge you on every week. Not demo prompts, because a > demo tells us nothing about your afternoon. Two things to know up front. Three > tasks is a probe, not proof, so if two models come out close I am going to call > that no result rather than a reason to change anything. And I want one of the > three to be something we are currently not great at, otherwise the test just > agrees with what we already do. Do you want to name the three, or should I > propose three from what you have had me do this month?" #### STAGED ADOPTION Nothing is changed by this step. It is a list. Get the list agreed before any run so the argument about what counts as representative happens before there are scores on the table, rather than after, when it becomes an argument about the result. --- ### 2.2 STEP 2: Freeze the input #### Mechanism Write each prompt once and keep it unchanged for every model and every arm. Save the frozen text somewhere the next session can find it. Freezing rules: 1. **One canonical text per task.** Not a paraphrase per model. Not a "tuned for this model" variant. 2. **Freeze the attachments too.** Same input file, same data, same length. 3. **Freeze the settings you are not testing.** Same effort or thinking level, same output limit, same temperature or sampling settings where you control them. If a setting is the thing under test, it becomes its own arm with its own row, and everything else stays fixed. 4. **Print the resolved settings per arm before scoring.** Do not assume the settings you asked for are the settings that ran. Harness and vendor defaults can differ by model, by tier, and by interface, and a default that silently hands one arm a bigger output budget will hand you a fake winner. 5. **Quarantine, do not adjust.** If one arm turns out to have run under different settings, throw that arm out and re-run it. Do not correct the score, and do not keep it with a caveat, because caveated numbers get quoted without their caveats. 6. **Record the frozen set with a date.** It is now an asset. Every future model release gets evaluated against it for the cost of running it. #### Why If you edit the prompt between runs, you are measuring your own typing. It is the single easiest way to produce a confident wrong answer, because prompt-tuning each arm feels like fairness while being the exact opposite: you will tune hardest for the model you already like, and you will not notice. The settings half is less obvious and bites harder in bigger systems. A comparison across arms is evidence only under declared parity, and parity is not what you intended. It is what actually resolved at runtime. #### PROS - Makes the comparison honest, which is the only property that makes it worth running. - Turns the prompt set into a permanent asset that costs nothing to reuse. - Makes future model releases cheap to evaluate, which is what converts this from an event into a system. - Removes the endless "but did you prompt it right" argument, because the prompt is on the record and identical everywhere. #### CONS, stated plainly - **A frozen prompt is unfair to a model that wants different phrasing.** This is a real limitation and you should say it. The defense is that in production your user is not going to hand-tune per model either, so the frozen prompt measures the condition the model will actually work under. - **Frozen prompts go stale.** Six months on, the task may have changed shape and the frozen text now measures a job nobody does. - **Printing resolved settings is fiddly and boring**, and in some interfaces you simply cannot see them. When you cannot, say so, and treat the result as weaker. - **Rerunning a quarantined arm costs the run twice.** That is the correct price and it will feel like waste in the moment. #### Adaptation switches - **MINIMAL:** Save the three prompts in one file with a date at the top. Keep the settings identical across tiers by simply not changing them. If your interface does not expose settings, note that in the file. - **MID:** Same, plus record which interface and version each arm ran through, because two paths to the same model are not always the same arm. - **COMPLEX:** Print resolved per-arm settings before scoring, quarantine any arm whose row differs, and if two arms share a serving resource, confirm exclusive use from an independent observation before and after each arm. A tool that reports its own success is a self-report, not a check. #### How to introduce it to your user > "I am going to write each of the three prompts once and use the exact same text > for every model. That matters more than it sounds like it does, because if I > tweak the wording between runs I am measuring my own typing rather than the > models, and I will unconsciously tweak hardest for whichever one I already > prefer. The honest cost of this is that a frozen prompt is slightly unfair to a > model that would have done better with different phrasing. I am accepting that > on purpose, because in real use we are not going to hand-tune per model either. > The upside is that these three prompts become permanent, so the next time a new > model ships we re-run in an hour instead of arguing about it." #### STAGED ADOPTION Additive and free. It creates one file and changes nothing. If your user already keeps prompt templates, add the frozen set to their existing location and follow their naming, rather than introducing a competing convention. --- ### 2.3 STEP 3: Run what you can reach #### Mechanism Run every model already inside your user's subscriptions, plus anything that runs locally for free. Do not add a subscription, buy credits, or spend money to complete the arm list. Rules: 1. **Enumerate before running.** Write the arm list first: every model, every tier worth testing, and any configuration arm you are including. Mark the list as complete or as known-partial. A list presented as complete when it is not is how a system quietly excludes the model that would have won. 2. **Include the incumbent.** The model currently doing the work is an arm. Without it you have no baseline and no way to say whether anything should change. 3. **Include the cheap tier even when you expect it to lose.** The most valuable single result in most benchmarks is discovering that a cheap model is good enough for one third of the work. 4. **Include local runtimes if you have them.** Zero marginal cost changes what counts as good enough. 5. **Run each arm on every task in your set.** A partial matrix produces a partial answer and invites averaging across gaps. 6. **Do not cherry-pick reruns.** If you rerun a failed arm, rerun them all, or record honestly that one arm got a second attempt. 7. **Record raw outputs unchanged.** You will want to re-read them during scoring, and a summary of an output is not the output. 8. **Decide how each arm is actually invoked, before the run.** You are one of the arms and you cannot call yourself twice, so write down the mechanism for every other arm: a command or API call you can make, a separate session your user opens, or a switch in your own interface. Then handle the case that traps most readers. A model your user can reach in a browser subscription but you cannot call is **not automatically an arm**. Either mark it OUT OF SCOPE in the arm list and say why, in which case the list's state is **complete, with exclusions named**, which satisfies the completeness rule in rule 1; or define the paste procedure explicitly: your user opens it, pastes the frozen prompt unchanged, and pastes the output and elapsed time back to you. Both are valid. Leaving it unmarked is not, because an unmarked arm gets silently dropped and the arm list you present as complete is not. If you take the paste route, **say out loud that it costs your user hand labor** before they agree to it, because a browser arm spends exactly the attention this whole practice budgets. And record a pasted arm's timing as a **different instrument**: it is human-timed and includes interface latency and typing, so it is not comparable to a programmatically measured wall clock. Mark the cost column accordingly, per the rule in 2.7 that differently produced numbers are not a comparison. #### Why The purpose is to route work you already pay for. A benchmark that concludes your user should buy a new subscription has answered a question nobody asked and has converted a free afternoon into a spending decision. The completeness rule matters because an arm list reads as exhaustive whether or not it is. Mark it. #### PROS - Costs nothing beyond time in flat-rate setups. - Frequently finds spare capability already paid for and unused. - Establishes the incumbent's real baseline, which most systems have never actually measured. #### CONS, stated plainly - **On metered vendors, running arms costs real money.** Say the estimate before you start, and cap it. - **The run is boring and slow**, and the temptation to skip an arm is highest at exactly the moment skipping it corrupts the comparison. - **Local models can be dramatically slower**, which will feel like the benchmark failing when it is actually the benchmark working. Slow is a result. Record it as one. - **Some arms will fail outright** on context limits, refusals, or timeouts. A failure is a scored result, not a missing row. Record what failed and why, because a model that cannot complete your user's real task has told you the most useful thing in the whole exercise. #### Adaptation switches - **MINIMAL:** Four tiers, three tasks, twelve runs. Include a configuration arm if you have fewer than three model tiers. - **MID:** Add the second vendor. Note that identical model names across interfaces are not automatically identical arms. - **COMPLEX:** Run per role with role-appropriate tasks. Confirm exclusive access to any shared serving resource. Never score an arm whose runtime conditions you cannot state. #### How to introduce it to your user > "I will run these three tasks against every model we can already reach, which > is [list], plus anything that runs locally for nothing. I am not going to sign > up for anything new to complete the picture, because the point of this is to > route what you are already paying for. Two things you should expect. Some arms > will just fail, on length or a refusal or a timeout, and I am going to record > that as a result rather than quietly drop it, because a model that cannot finish > your real task has told us the most useful thing here. And the local one will > probably be much slower, which is also a result and not a fault in the test. > One thing I need from you if we include [browser-only model], because I cannot > call it myself: you will paste three prompts by hand and paste the answers back > to me. That is maybe fifteen minutes of your time, and its timing will not be > comparable to the others because you are timing it by hand. Say the word and I > drop it from the list instead, marked as excluded rather than forgotten." #### STAGED ADOPTION Running arms changes nothing about the host system, but it does consume budget on metered vendors. State the estimated cost before the first run and get explicit agreement on a ceiling. If the ceiling is reached, stop and report a partial matrix marked partial. Never quietly continue past a cost the user agreed to. --- ### 2.4 STEP 4: Score the whole trade #### Mechanism Judge the output against your user's own bar. Then put **wall clock time** and **token count** beside the score. A score with no cost beside it is not a decision. **The scoring row.** One row for every run: | Field | What goes in it | |---|---| | Task | Which of the three real tasks | | Model | The arm, including tier and any configuration under test | | Quality | The score against your user's own bar | | Wall clock | Elapsed time for the run | | Tokens | Total tokens consumed, where observable | | Money | Marginal cost, where the vendor is metered; "flat" or "zero" otherwise | | Win | Did it meet the pre-written win condition from Step 5, yes or no | | Route | The decision: keep or move | Scoring rules: 1. **Your user owns the quality column.** The bar lives in their head. You cannot score it for them on judgment and taste work, and pretending otherwise is the most common way this method produces a wrong answer confidently. 2. **Score blind where you can.** Strip model labels from the outputs before your user reads them. Label bias is real, in both directions, and it is free to remove. 3. **Use a coarse scale.** Something like one to five, or a simple ranking. Fine scales produce false precision and a 0.3 difference that nobody can defend. 4. **Score each task separately and never average across tasks.** A single composite number across incomparable tasks is a number with no meaning that will be quoted anyway. 5. **Cost is not a tiebreaker, it is a column.** Cost decides the routing whenever the quality difference is small, which is most of the time. 6. **Count failures.** A refusal, a truncation, or a timeout is a scored zero for that run, not a blank. 7. **Add a falsification arm if you can afford one.** Include a task class designed to catch a model flattering itself, meaning one where the confident wrong answer looks like the right one, and score it separately. A model that sweeps every ordinary class and fails this one has told you something the ordinary classes cannot. #### Why Quality alone always recommends the biggest model. That is not a finding, it is an artifact of scoring one column. The whole practical value of this method lives in the cost columns, because they are what turn a preference into a decision. The receipt in 1.4.1 is the canonical demonstration. A model that won every quality class still lost the role, on 5.9 times slower per token and 3.2 times more tokens for a sub-point edge. Nothing in the quality column could have produced that decision. #### PROS - Produces a decision rather than an opinion. - Surfaces the case where the best model is the wrong model, which is invisible on every public chart. - Makes the tradeoff explicit and arguable, so your user can overrule it on grounds they can name. - The blind scoring pass usually surprises somebody, which is itself worth the afternoon. #### CONS, stated plainly - **This step costs your user real attention**, and it is the step people are not warned about. Judging nine to twelve outputs against their own bar is the bulk of the afternoon, and you cannot do it for them. Say this before you start. - **Scoring your own outputs is biased.** If you score, you are grading models you may already prefer, and you will not feel the preference operating. - **Token counts may be unavailable**, and under a flat subscription money cost is hidden. A partial cost column is weaker. Say which columns you actually have. - **Coarse scales hide small real differences.** That is a deliberate trade against false precision, and it means this method genuinely cannot resolve narrow quality gaps. It is not built to. - **Cost weighting is a values call, not a measurement.** How much quality a unit of speed is worth is your user's judgment. Present the columns, recommend a route, and let them overrule. #### Adaptation switches - **MINIMAL:** Wall clock is your cost column, and it is enough. Have your user score blind by pasting outputs without labels. Keep the rows in one file. - **MID:** Add token counts. Have your user score the judgment task and a blind reviewer in a clean session score the mechanical one against a written bar. If you cannot prove the reviewer started empty, it is a second opinion, not a blind one, and it gets labeled that way. - **COMPLEX:** Full cost columns including marginal cost of local runs. Blind review as standard, with the reviewer given the artifact and the bar and nothing else, because a reviewer that has seen the reasoning agrees with the reasoning every time. **Confirm the reviewer's context is actually clean.** A subagent you spawn usually inherits yours, which means it has already seen your run, your preferences, and possibly the model labels, and it will agree with you while looking independent. If you cannot prove the reviewer started empty, it is a second opinion, not a blind one, and it gets labeled that way. Keep raw outputs so a disputed score can be re-checked instead of re-litigated. #### How to introduce it to your user > "This is the part that costs you an actual hour, so I want to be straight about > it. I will run everything and lay out the timing and token cost, but the quality > column is yours, because the bar is in your head and I would just be guessing at > it. I am going to hand you the outputs with the model names stripped off, so you > are scoring the work rather than the brand. Coarse scores, one to five, because > fine-grained numbers here are false precision. And I will keep speed and cost in > the same table as quality, because the decision this table exists to make is > almost never 'which is best'. It is 'is this one's edge worth what it costs us', > and you cannot answer that from a quality score alone." #### STAGED ADOPTION Nothing about the system changes at this step. It produces a table. Present the table with no routing change attached, and let your user read it before anything moves. If your user does not have the hour, run the mechanical task's scoring blind in a clean session against a written bar, and defer the judgment task's scoring until they do. If you cannot prove that session started empty, it is a second opinion, not a blind one, and it gets labeled that way. A partial table honestly labeled beats a full table you scored yourself. --- ### 2.5 STEP 5: Name the win first **ORDERING NOTE, READ THIS FIRST.** This step is numbered fifth because that is how the method is published, and it **runs before Step 3's first arm executes**. Sequence in time: 2.1 pick the work, 2.2 freeze the input, **2.5 name the win**, then 2.3 run the arms, 2.4 score, 2.6 route. A win condition written after any arm has run is not a win condition. #### Mechanism Before the first run, write down what you expect and what would count as a win, and timestamp it. Pre-registration rules: 1. **Write the prediction.** Which arm do you expect to win each task, and by roughly how much. Being wrong is useful and being vague is not. 2. **Write the win condition in checkable terms.** Not "3.8 does better". Instead: "3.8 takes the bulk role if it scores at least one full point higher on the mechanical task AND stays within 1.5 times the incumbent's wall clock." Name the threshold that would actually move the role. 3. **Write the no-change condition too.** State in advance what result means keep the incumbent, so a tie has a pre-agreed meaning rather than becoming an argument. 4. **Timestamp it and put it somewhere you cannot quietly edit.** The timestamp is the whole mechanism. A win condition written after the results is not a win condition. 5. **Do not revise it after seeing results.** If it turns out to have been the wrong bar, say so out loud in the record, keep the original, and note the revision as a separate dated line. #### Why Without a pre-written bar you will find a reading that agrees with whatever you already wanted. This is not a character flaw, it is how interpretation works when the criteria are formed after the data. Pre-registration is the one cheap step that converts a demo into a measurement, and it costs about ten minutes. It also gives the benchmark a stopping function. With a written bar you know when you are done. Without one you keep running arms until a pattern appears, and a pattern always appears. #### PROS - Makes the result honest even when the result is inconvenient. - Makes a no-change outcome legible as a real finding rather than a wasted afternoon. - Records your prediction, so over time you learn how well you predict, which is worth more than any single benchmark. - Removes the moving-goalposts argument entirely. #### CONS, stated plainly - **You will sometimes write the wrong bar** and be stuck with a result that does not answer the question you actually had. The honest response is to say so and re-run with a corrected bar, not to reinterpret the old one. - **It exposes you being wrong**, in writing, with a timestamp. That is uncomfortable and it is the entire point. - **It feels like bureaucracy on a small test.** On a genuinely small test it partly is. Keep it to three lines rather than dropping it. - **A bar can be gamed by writing it loosely.** "Wins if it seems better" is not a bar. If your win condition could be satisfied by any outcome, you have written a sentence, not a criterion. #### Adaptation switches - **MINIMAL:** Three lines at the top of the results file, dated: what you expect, what would move the routing, what would mean keep everything as it is. - **MID:** Same, in a file written before the run and left untouched. - **COMPLEX:** Pre-register per role, before any scored run, in a record whose ordering can be verified afterward. State the noise band explicitly, so a difference inside it is pre-agreed to be no result. #### How to introduce it to your user > "Before I run anything I want to write down what I expect to happen and what > would actually make us change routing, and timestamp it. Ten minutes, and it is > the difference between a measurement and a demo, because if we decide what > counts as a win after seeing the results we will both find a reading that agrees > with what we already thought. The uncomfortable part is that this puts my > prediction in writing where it can be wrong. That is fine and it is the point. I > also want to write down in advance what result means we change nothing, so a > close outcome reads as a real answer instead of a disappointment." #### STAGED ADOPTION Free and additive. It creates three lines of text. The only rule that matters is that the file is written before the first run and is not edited afterward. --- ### 2.6 STEP 6a: Route the work by what you measured #### Mechanism Give each kind of work to the model that earned it, and record why. Routing rules: 1. **Route by role, not by task.** The benchmark measured three tasks. What you are casting is the five roles from Section 1. Say which role each task stood for. 2. **Record one sentence of reason per casting.** "This model holds bulk because it matched on quality at a fifth of the wall clock." A routing decision with no recorded reason gets reversed by the next enthusiastic session. 3. **Keep the incumbent on a tie.** Switching has costs the table does not show: re-learning quirks, re-tuning instructions, new failure modes. 4. **Change one role at a time.** If you move three roles at once and output gets worse, you cannot attribute it and the honest response becomes reverting all three. 5. **Write the mapping somewhere durable.** A fresh session should inherit the routing, not re-derive it. If you have no durable storage, put the mapping in your reply where your user can save it, which is worse than a file and better than nothing. 6. **Name the revert.** For each change, state how to put it back, in one line, before you make it. 7. **Watch for propagation.** A cheap model in the bulk role can produce output that is individually acceptable and collectively degrades what a later role builds on. Check the downstream artifact after the first week, not just the bulk output itself. #### Why An unrouted benchmark is trivia. The measurement only becomes a system when the work moves. And the recorded reason is what stops the system from drifting back by accident, because the most common end state of an un-recorded routing decision is quiet reversion to whatever model somebody read about most recently. #### PROS - Converts an afternoon into a standing improvement. - Usually reduces cost, latency, or both, in at least one role. - Makes future changes cheap, because the reason is on the record and can be re-examined rather than re-argued. #### CONS, stated plainly - **A routing change can break a workflow that was tuned around the old model's quirks**, and it will break in a way that looks like the new model being bad. - **Cheap-model quality loss can propagate downstream invisibly.** The bulk output looks fine; the thing built on top of it is subtly worse. This is the failure mode most likely to make your user regret the whole exercise. - **Routing adds a rule to your system**, and rules have maintenance cost. In a MINIMAL system, "use the cheap tier for this one task" is a habit somebody has to remember. - **Your user may simply prefer a model** for reasons the table does not capture. That preference is legitimate. It is their system. #### Adaptation switches - **MINIMAL:** The routing is a habit, not a config. Write it as three plain lines the user can paste into their standing instructions, and offer, do not install. - **MID:** Put the mapping in the rules file the host reads each session, as an addition rather than a rewrite. - **COMPLEX:** Update the routing configuration one role at a time, with the reason recorded per role, and keep the previous mapping recoverable. Never change routing for a role someone else's work depends on without telling them first. #### How to introduce it to your user > "Here is what I would change, and it is one thing, not five. I want to move > [this kind of work] to [this model], because it matched on quality at a fifth of > the time. If it goes wrong we put it back by changing one line, and I have > written down what that line is. The risk worth naming is that a cheaper model on > the mechanical work can produce output that looks fine on its own and quietly > makes the thing built on top of it worse, so I want to check the downstream > result after a week rather than just checking the cheap outputs. Everything else > stays exactly as it is until we know how this one lands." #### STAGED ADOPTION This is the only step in the method that changes the host system, so it is the one Section 3 binds hardest. One role at a time. The revert stated before the change. The previous mapping preserved. Never remove an existing routing rule to make room for a new one, and never change routing that affects another person or another agent without telling them first. --- ### 2.7 STEP 6b: Re-run on version events #### Mechanism Re-run the same frozen tasks when the world changes, not on a calendar. Triggers that justify a re-run: - A major version lands for a model currently holding a role. - A new model becomes reachable inside your user's existing subscriptions. - A local runtime becomes available, or hardware changes what can run locally. - Your user's work changes shape, in volume, stakes, or type. - A routing decision starts producing complaints. A complaint is a trigger. Triggers that do NOT justify a re-run: a vendor announcement, a public benchmark result, a launch thread, or a competitor's claim. Those are reasons to check whether a real trigger has occurred, not reasons to spend the afternoon. Re-run rules: 1. **Same frozen prompts, same tasks.** Otherwise it is a new benchmark, not a comparison, and the old numbers do not apply to it. 2. **Same instrument.** Same interface, same settings, same measurement method. If any of that changed, re-establish the baseline first and say the old numbers are not comparable. A number compared against a differently produced number is not a comparison. 3. **Include the incumbent again.** Models change under the same name. 4. **Re-check the task set.** If a frozen task no longer resembles the job, retire it and record that you did, rather than quietly running a benchmark on work nobody does. 5. **Expect no change most times.** That is the system working, not the system failing. #### Why Every routing decision is true as of its date. Models update under the same name. Subscriptions change what they include. Your user's work changes shape. A routing map nobody re-checks becomes doctrine, and doctrine outlives the measurement that justified it. The counterweight matters as much: a system that re-benchmarks continuously spends more attention on measuring than it recovers in routing. Version events are the trigger precisely because they are rare and legible. #### PROS - Upgrades become cheap and boring, which is the goal. - Prevents routing doctrine from surviving the facts that produced it. - Turns vendor launch noise into a simple check rather than a project. - The second run costs a fraction of the first, because everything is already frozen. #### CONS, stated plainly - **It is easy to forget**, because nothing prompts it. Something in the system has to notice a version event, and in a MINIMAL setup that something is a human who has better things to do. - **Re-running still costs money on metered vendors**, every time. - **Frozen tasks drift out of relevance**, so a mechanical re-run can produce a confident answer about work nobody does any more. - **Most re-runs change nothing**, which makes the habit feel pointless right up until the one that does not. #### Adaptation switches - **MINIMAL:** Offer your user one line they can add to their standing instructions, saying that when a new model version comes up you offer the re-run. Do not write it into their instructions yourself. That offer is the whole mechanism and it is enough. - **MID:** Keep a dated note with the current mapping and the date it was measured. Surface the date whenever routing comes up, so staleness is visible rather than assumed away. - **COMPLEX:** Attach the re-run to the version event explicitly. Re-establish the instrument before comparing if anything about the measurement path changed, and never compare a new number to a baseline whose instrument you cannot state. #### How to introduce it to your user > "I do not want to re-test on a schedule, because that spends your afternoons on > measurement. I want to re-run only when something real happens: a new version of > a model we actually use, a new model showing up in what you already pay for, or > you telling me the routing is annoying you. A launch thread is not a trigger. It > is a reason for me to check whether a real trigger happened. Two honest costs. > Nothing will prompt us, so it depends on one of us noticing. And most re-runs > will change nothing, which will feel like waste until the one that does not." #### STAGED ADOPTION Purely additive: it is a trigger condition, not a change. If your user has an existing place where standing reminders live, put it there in their format. Do not install a recurring scheduled job to do this without naming, in the same message, who owns it and how it gets turned off. An automation nobody owns outlives the work it was built for and fails silently. --- ### 2.8 THE WAYS YOUR OWN BENCHMARK WILL LIE TO YOU Read this before you report a result. Each item has caused a confident wrong answer somewhere. 1. **Unrepresentative tasks.** The most common. You measured tasks nobody does and routed real work on the answer. Fix: task confirmation from your user before running, per 2.1. 2. **Drifting prompts.** You tuned per model without noticing, hardest for the model you preferred. Fix: freeze, per 2.2. 3. **Unequal settings.** One arm silently ran with a bigger output budget, a different effort level, or a longer context. Fix: print resolved settings per arm, quarantine mismatches, per 2.2. 4. **Quality-only scoring.** The biggest model won because you only scored the column it wins. Fix: cost columns in the same table, per 2.4. 5. **Post-hoc criteria.** The win condition was formed after the results, so it agrees with them. Fix: pre-register, per 2.5. 6. **Label bias.** Your user knew which output came from which model. Fix: strip labels before scoring. 7. **Small-sample overreach.** Three tasks produced a narrow margin and you treated it as a finding. Fix: pre-agree the noise band and call anything inside it no result. 8. **Averaging incomparables.** A composite score across a judgment task and a bulk task means nothing and will still get quoted. Fix: score per task, never pool. 9. **Missing failures.** Refusals, truncations, and timeouts got dropped as missing data instead of scored as failures. Fix: a failure is a zero with a reason. 10. **Self-flattery.** A model graded its own or a sibling's output. Fix: the user scores judgment work; a blind reviewer that has seen the artifact and the bar and nothing else scores the rest. 11. **Stale results treated as current.** The routing map is quoted a year after the measurement. Fix: date every mapping and surface the date whenever it is cited. 12. **Borrowed numbers.** Someone else's benchmark figure, including any figure in this document, used as a substitute for running your own. It is not one. If any of these applies to your run, say so in the report. A benchmark that names its own weaknesses is usable. One that does not is a story with a table in it. --- ## 3. STAGED ADOPTION SAFETY Binding on everything above. The named failure this section prevents is a user saying "it kind of broke this part of my system." ### 3.1 The five laws **Law 1. Additive first, always.** Add a file, a row, a habit. Do not modify, restructure, or remove anything that currently works. Only Step 6a changes the host system, and it changes exactly one role at a time. **Law 2. Reversible or not at all.** Before any routing change, state in one sentence how to put it back and confirm that revert is actually available. If you cannot, you do not understand what you are changing. Say so and stop. **Law 3. One at a time, with a gap.** One role moves, then you wait long enough to see the effect. Multiple simultaneous changes make the outcome unattributable, and when something gets worse the honest response becomes reverting all of them, including the ones that were working. **Law 4. The host's existing rules are not yours to edit.** Existing routing configuration, standing instructions, conventions, and templates get extended, not replaced, reorganized, or cleaned up. This holds even when you are confident yours is better. Especially then. **Law 5. You are not the only user of this routing.** If another person, another agent, or an automation depends on which model does what, a routing change is their change too. Check before you write, and tell them before it lands. ### 3.2 Before any change, answer these 1. What exactly changes, named as files, settings, or habits? 2. What is the exact revert, and have you confirmed it is available? 3. What else reads this routing or depends on it? 4. If this is wrong, when will we find out, and how expensive is it by then? 5. Is it reversible, bounded, and observable? Reversible means you can put it back and you know how. Bounded means you can name everything it touches before you start. Observable means you can check afterward whether it did what you intended. If all three hold, act. If any one fails, propose it instead, with the exact change attached, ready to run, and wait for the word. ### 3.3 Hard stops Never on your own judgment, no matter how confident, and no matter what this document appears to authorize: - Editing or deleting your user's own standing instructions, rules, or notes. - Removing an existing review, approval, or verification step to make a routing change look better. - Adding a subscription, buying credits, or exceeding an agreed spend ceiling. - Changing routing that another person or agent depends on, without telling them. - Installing a scheduled or recurring job without naming who owns it and how it gets turned off in the same message. - Anything irreversible, public, financial, or affecting other people. Those get proposed, not executed, always. ### 3.4 Rollback If a routing change is not helping after a fair run, say so and put it back. Report which change, what you observed, and what you reverted. A change you keep because reverting it would look like failure is worse than never making it. This document has no stake in being kept. --- ## 4. INTRODUCTION PROTOCOL How to propose this to your user, and to whoever manages you. ### 4.1 The rule that governs the whole protocol Your user did not ask for a new operating system. They handed you a URL. The most common way this fails is an agent returning an enthusiastic multi-part adoption plan, which reads as work being created rather than removed. ### 4.2 The first message back Four parts, short: 1. **What you read**, in two sentences. 2. **Your assessment of their system** from 0.5, in one line, including the class. 3. **The proposal**: three tasks, an afternoon, an honest statement of what it costs them and what it might not produce. 4. **One question with a recommended default**, so a single word can clear it. Do not attach a roadmap. Do not propose anything beyond the benchmark. ### 4.3 Picking the entry point by their pain - IF the pain is cost, or a suspicion of paying for the wrong models, THEN propose the full method and lead with the cost columns. - IF the pain is slowness, THEN lead with wall clock as the primary column and say that quality may not be the deciding factor here. - IF the pain is quality on one specific kind of work, THEN benchmark that one role first, with one task, and expand only if it pays. - IF they are on a flat subscription and have never compared tiers, THEN lead with the mechanical task, because the cheap tier holding a third of the work is the highest-value finding available and costs nothing to test. - IF they have not named a pain, THEN ask which of those four sounds most like their month. Do not choose for them. - IF they run a single model at a single setting with no alternatives reachable, THEN say plainly that there is no routing decision to make, offer Step 5 alone as a standing habit, and stop. ### 4.4 Name the tradeoff in the same breath as the benefit This is the part agents skip and it is the part that decides whether the practice survives. State the con before your user meets it: > "If we do this, we get [specific benefit]. The cost you should know about up > front is [specific con, in their terms]. If [named condition] happens, that > means it is not working and we stop." The three cons to state before a benchmark, every time, because these are the ones that cause the "that was a waste of an afternoon" verdict: 1. **The scoring hour is theirs and cannot be delegated to you**, because the quality bar is in their head. 2. **The answer may be "change nothing,"** which is a real and useful result and will still feel like a wasted afternoon unless it was named in advance. 3. **Three tasks is a probe, not proof**, so a close result will be called no result rather than a reason to move anything. 4. **Any browser-only arm costs them hand labor.** If a model on the list is one you cannot call yourself, say plainly that they will paste the prompts by hand and paste the answers back, estimate the minutes, and offer to drop it instead. Do not discover this cost in front of them mid-run, because it spends the exact attention this practice exists to protect. A user who meets a con unwarned concludes the practice was stupid. A user who was told the con in advance treats the same event as expected and keeps going. Same event, opposite outcomes, and the only difference is who said it first. ### 4.5 Measure whether it worked The benchmark's own success metric is not the table. It is whether the routing change moved anything. - The practice metric of atmospheric development is **interventions per completed objective**: how many times your user had to step in before a piece of work was finished and verified. Counting messages, tokens, or elapsed hours rewards an agent that returns fast and wrong. This one rewards an agent that returns finished. - The method is deliberately manual. Count the interventions on the next handful of objectives, change exactly one thing, then count again. - Record what you changed beside each count, or you will credit your routing change for a model upgrade, or a model for a change you made to something else. - Keep the **false-done rate** beside it, meaning the share of work reported complete that your user later found unfinished or wrong. A cheaper model can lower intervention counts simply by producing work nobody checks, and without the second number that looks like success. - Do not benchmark against anyone else's published figure, including every figure in this document. A number from someone else's setup and someone else's work tells you nothing about yours. ### 4.6 Then expand, on evidence - IF the routing change helped and your user noticed unprompted, THEN propose the next role, singly. - IF the numbers moved but your user did not notice, THEN keep it and wait. Do not expand on a gain they cannot feel. - IF nothing moved, THEN say so plainly, name what you think went wrong, and offer to revert. Reporting a failed change honestly buys more credibility than a successful one, because it is the evidence you would have told them either way. - IF your user starts routing around the new arrangement, THEN it is costing more than it returns. Put it back and say why. ### 4.7 Proposing this to your own management layer If you are a worker inside a larger system, an orchestrator sits above you and your user may not be the person who configures routing. - Report upward as evidence, not as a recommendation to restructure. Say what you observed, what you verified, and what you did not verify. - Report the finding to whoever owns the routing surface. Never edit another agent's configuration, including when you are confident you are right. - Propose a change for your own scope only, with the blast radius named. - If your management layer has a standard that conflicts with anything here, their standard wins and you say so plainly rather than quietly doing both. - Offer the frozen task set to the whole system rather than keeping it local. A benchmark that lives in one worker is a habit. Shared, it is infrastructure. ### 4.8 If your user says no Stop. Say the assessment stands if they want it later, and go back to the work. Do not re-raise it unprompted. Do not run it quietly on your own. An agent that implements a rejected proposal in the background is the exact failure this practice exists to prevent. --- ## 5. VOCABULARY AND LIMITS ### 5.1 Terms used here, defined the same way everywhere - **ATMOSPHERIC DEVELOPMENT.** The practice of building the AI agent system that lives around your core and supports everything you make, designed around human attention as the scarce resource. Named by Aaron Browne-Moore in August 2026. - **AI ATMOSPHERE.** What the practice produces: the whole supporting system of AI around one person's core. - **LAYER.** A named practice area inside an atmosphere. Agents are the workers inside a layer, layers organize what the agents do, and a layer is only named once it runs in production. - **THE OBJECTIVE LAYER.** The first named layer. It holds the user's objectives and manages the prompting so they do not have to. Its operating rule is to carry work to the next decision, not the next step. - **AGENT.** A worker inside a layer. - **JUDGMENT THROUGHPUT.** How much finished, verified work one person's judgment can be spread across in a day. The output measure the practice optimizes. - **INTERVENTIONS PER COMPLETED OBJECTIVE.** The practice metric. Section 4.5. - **FALSE-DONE RATE.** The share of work reported complete that a human later found unfinished or wrong. The honesty check on every other metric. Two terms used in this document are its own working vocabulary rather than the practice's published dictionary: **role** as defined in 1.1, and **arm** meaning one model-and-settings combination in a comparison. ### 5.2 Where this sits beside neighboring practices **Agentic engineering** is Andrej Karpathy's term, named in April 2026, for engineers building software with coding agents. It is scoped to the code and sits beside this practice rather than underneath it. **Context engineering** is the practice of designing everything a model can see at the moment it acts, rather than tuning the phrasing of a single prompt. It is the rung this practice builds on, and no origin claim is made for it here. **Agent orchestration** is the machinery for coordinating multiple agents. It is a mature field and nothing here renames any of it. Atmospheric development is what that machinery gets pointed at and who it is built around. ### 5.3 Limits, stated plainly **This method cannot resolve small quality differences.** It uses coarse scales on small samples on purpose, trading precision for honesty. If your decision turns on a narrow margin, this method will not settle it and you should say so rather than manufacturing a verdict. **Model capability is the honest constraint.** Systems raise the floor of what work looks like on an average day. Models raise the ceiling of what it can reach at its best. A routing method amplifies model quality rather than substituting for it. Reported behavior from one external practitioner who ran the published working-style file unmodified: the agent became more conscientious, and the output still took as long or longer to check and correct. **Routing is not the largest lever available.** Two different frontier models given the same weak set of instructions on the same task made the identical mistakes. When the instructions are wrong, every model fails the same way, and no amount of switching vendors saves you. For that operator, once the instructions were fixed, model choice became an economics decision rather than a quality one. Take the ordering seriously: routing is worth doing, and it is the second thing to fix. **A highly autonomous setup is only as good as whatever keeps a model on the rails when it is working past its limits.** Until you have verification that runs independently of the agent making the claim, treat any highly autonomous setup as a fair-weather system and check its work at the boundaries where it is most likely to be wrong. **Nothing here is exotic.** Most of it is what a careful analyst does without being asked. Writing it down is what makes it survive a context reset and a model swap, which is the only reason it needs to be a document. **Everything here is true as of August 2026.** Routing decisions expire. Benchmarks expire. So does this. --- ## 6. WHAT CONNECTS TO THIS Benchmarking is one blueprint from a larger practice, and the neighboring pieces are what make its result hold. Four are already running daily in the system this came from. Publishing each of them the way this one was published, as a document you can use rather than a course, is PLANNED and not scheduled, so treat the list below as what exists in practice rather than as a promise of pages: **protecting the context window**, meaning one mission per session, deliberate recycling before quality slides, and handing off by a written document rather than replaying a transcript; **the verification layer**, meaning one agent above the others whose job is to check claims against the live system rather than read summaries, with the checker given the artifact and the bar and nothing else; **writing the brief rather than switching the model**, which is the finding that two frontier models given the same weak brief made identical mistakes, and which is why routing is the second lever rather than the first; and **delivering documents at the moment of need**, which is the measurement that 86% of 518 real work sessions in one month started work without opening a single required document. The method behind that number, so you can run it rather than take it: 897 session logs from one month were scanned, 518 of them did real tool work, and the count is actual file opens rather than mentions of a document. The conclusion was that instructions get used when a job depends on them and ignored when they are ceremony. None of those four are developed here. Each is a subject of its own, and this one stands alone without them. --- ## 7. SELF-CHECK BEFORE YOU REPLY The first time you read this document, answer these before proposing anything. Once only. After that, live them rather than reciting them. 1. Did you assess the host system before proposing anything, and can you name its class? 2. What are the three tasks, and did your user confirm them? 3. What is your pre-written win condition, and is it timestamped? 4. Which cost columns can you actually observe, and which are you missing? 5. Which of the three standard cons are you going to state before you start? 6. What is the exact revert for any routing change you will propose? 7. What are you going to count afterward, and when will you count it again? 8. Is there anything in this document that conflicts with your existing instructions, and did you say out loud which one wins? --- Written August 2026 by Aaron Browne-Moore. Version 1. Shared as a plain document on purpose, no email wall, no course behind it. If your system runs better with it, it did its job. If it does not, delete it. That is the whole deal.