A vendor published a score last week. It arrived without a formula.
Here is the complete set of verifiable facts I can establish about Cognition's new AI efficiency grading system, the one positioned against what the industry has begun calling tokenmaxxing. Cognition, the company behind the autonomous coding agent Devin, announced that it has built a system to grade efficiency. That is the entire hard fact. Everything else in circulation โ that the system will shift focus away from raw token consumption, that it targets waste, that it represents a new philosophy of agent design โ is editorial framing applied to a noun.
I want to establish evidentiary status before I analyze anything, because precision about evidentiary status is the only thing separating analysis from commentary. There is no published metric definition. No numerator. No denominator. No sampling frame. No benchmark specification. No sample size. No confidence interval. No third-party attestation. No versioned methodology document.
When I audit a contract I run a fixed checklist: what is the bytecode, who holds the admin keys, what is the input domain, what is the output domain, and what is the reconciliation path between the last two. A metric requires the same discipline with different nouns. What is the numerator. What is the denominator. What is the sampling frame. Who defines ground truth.
Cognition's announcement supplies a name. A name is not a metric. A name is a container that a metric may eventually be poured into.
This is not a dismissal of intent. It is a statement about what a press release can support. In nine years of reading ledger state I have learned to keep three things in separate columns: the artifact, the claim about the artifact, and the narrative the claim was constructed to serve. Most analytical failures are contamination events โ the third column leaking into the first.
Column one contains one line. Column two contains nothing. Column three contains a great deal, and column three is where the actual engineering problem lives.
Context: a crowded market that cannot agree on what to count
Cognition occupies a specific and uncomfortable position in the AI coding market, and the position explains the announcement better than the announcement explains itself.
Devin was first to establish the autonomous agent mental slot โ the agent that executes a task end to end rather than completing your current line. That was a genuine category claim, made early, and it earned the company brand equity with enterprise buyers who wanted the thing that sounded like the future rather than the thing that sounded like autocomplete. The category claim has since been absorbed by everyone. Cursor holds the IDE layer with strong distribution and a superior editing surface. GitHub Copilot holds channel through Microsoft in a way no independent vendor can replicate. The model vendors ship their own agent surfaces with structural advantages that no wrapper can match โ native tool-call fidelity, first access to capability, and the ability to price at marginal cost. Open-source entrants compress the floor.
Model capability for code tasks has converged at the frontier. This matters more than any single product decision. When the underlying model is roughly interchangeable across vendors, the value added by an agent orchestration layer is reduced to latency, tooling, context management, and price. All four are contestable. None of them is a moat in the sense that a distribution channel or a model checkpoint is a moat.
A vendor in that position has three strategic options. Win on capability, which requires owning a model. Win on distribution, which requires a channel. Or win on the axis of comparison, which requires nothing but credibility and timing.
The third option is the cheapest. It is also the only one Cognition can execute this quarter.
There is a second context layer that matters, and it is financial rather than technical. The AI coding segment has spent two years raising on high revenue multiples. The question that has started appearing in diligence meetings is not whether agents can write code. It is whether the spend produces a return. Enterprise procurement has moved from experimentation budgets to line-item scrutiny, and line-item scrutiny demands a number. Nobody currently has one. The vendor that supplies the number supplies the frame in which every competitor is evaluated.
That is the whole strategic logic, and it is sound. The problem is that the logic of supplying a number and the logic of producing a defensible number are different disciplines, and only one of them is quick.
I should state my priors and their provenance, because readers deserve to know what is being inferred from where. In 2019 I spent roughly two hundred hours manually auditing the 0x protocol v2 smart contracts on GitHub as a second-year student. I found three critical logic flaws in the order matching engine and wrote them up. The lesson from that work was not about 0x. It was that the documentation and the execution are two different artifacts, and the gap between them is where every failure lives. A specification is a claim. The bytecode is the fact. The code does not lie; it only waits to be read.
In 2020 I modeled Compound's interest rate curves in Python across fifty thousand historical blocks. What that exercise taught me is that a rate quoted without its utilization state and its block range is decoration, not information. An APY is a function of a state; strip the state and you have a number that means nothing while looking precise.
In 2021 I tracked ten thousand NFT token URIs across the top hundred collections and found that roughly forty percent depended on centralized hosting vulnerable to a single takedown. In 2022 I traced a hundred thousand Terra transactions to the de-peg mechanism at the code level. In 2024 I tracked six months of daily IBIT flows and their correlation with realized volatility.
Five investigations, one invariant: a metric is only as meaningful as the state that produced it, and the state is almost never in the announcement.
Core: what is actually measurable, and where the chain breaks
The token accounting layer is the strongest part of this entire system, and it is worth being explicit about why, because the strength is real and it is narrow.
Every inference API returns usage objects. Prompt tokens, completion tokens, cached tokens, and increasingly reasoning tokens billed at a separate rate. These are integers. They reconcile against a bill. You can sum them across a session, attribute them to a request ID, and reconstruct the spend path after the fact. In forensic terms, this is a proper ledger: append-only, arithmetically closed, and independently verifiable by anyone holding the same API keys.
That is more than most software metrics can claim. It is also where the good news stops.
A token is not a unit. It is a tokenizer-relative count. This is not a semantic quibble; it is a unit mismatch that corrupts every cross-vendor comparison before methodology even enters the conversation. The same block of source code tokenizes differently under different families โ sometimes by twenty to forty percent on code heavy in punctuation, whitespace, and identifiers. A vendor reporting a token-efficiency score across multiple model families is comparing rulers with different tick spacing and averaging the results. The arithmetic will close. The comparison will not mean anything.
Add the compositional problem. A modern agent request is not one inference call. It is a loop: plan, call a tool, read the result, extend context, call again. Cached prefix tokens are billed differently from fresh tokens. Reasoning tokens may be billed but not returned. A single task can generate dozens of round trips with wildly heterogeneous cost composition. Any single scalar derived from that structure is an aggregation choice, and every aggregation choice is a value judgment wearing a lab coat.
So the input side of the efficiency ratio is auditable within one vendor's billing surface and non-comparable across vendors. Hold that constraint. It is load-bearing.
The attribution ladder: nine hops, one measurement
The output side is where the system stops being a measurement and becomes a positioning exercise. Consider what has to be true to say a token produced value.
Hop one: token spend. Measured.
Hop two: tool invocations and context growth. Measured.
Hop three: diff generated. Measured, at least in size.
Hop four: diff accepted by the human operator. Partially measured, and only if the interface instruments the rejection event. Most do not.
Hop five: tests passed. Measured if the repository has meaningful tests and the agent actually runs them, which is a conditional that fails more often than vendors admit.
Hop six: review approved. Human, subjective, essentially unmeasured.
Hop seven: merged. Measured in the version control system.
Hop eight: production incident attributable to the change. Measured only in organizations with real incident forensics, which is a small minority.
Hop nine: business outcome. Time-to-market, retention, revenue. Never measured at the task level by anyone, anywhere, in any product I have examined.
Each hop from three onward introduces survivorship. Discarded diffs leave less trace than merged diffs. A score computed on merged output is computed on a censored sample, and censored samples produce biased estimates in a direction that is rarely disclosed.
The human reviewer is an un-metered filter, and any score that ignores reviewer time will systematically reward agents that shift work onto the reviewer.
Run the arithmetic on a simple model. Agent A generates eight candidate diffs per task at forty thousand tokens each โ three hundred and twenty thousand tokens โ and a human spends six minutes selecting one. Agent B generates a single diff at one hundred and twenty thousand tokens with a twelve-minute review. On raw token efficiency, Agent A is more than two and a half times worse. On fully loaded cost โ tokens plus reviewer minutes at a loaded rate โ the ordering can invert entirely. The score, if it is token-denominated, will recommend the wrong agent. This is a model, not data; I flag it as such. But the direction of the bias is structural, not incidental, and it does not depend on the specific coefficients.
There is a second-order effect that is worse. An agent optimized against a token-denominated score learns to minimize context reads, skip call-site verification, and avoid re-reading files it has already seen. Those are precisely the behaviors that catch invariant violations. In my own audit work, the hours that found the three 0x flaws were spent reading call sites that the documentation said were irrelevant. A metric that penalizes that reading penalizes the only reliable method I know for finding the class of bug that matters.
The fork: a public benchmark is gameable, a private telemetry set is incomparable
Assume good faith. Assume Cognition wants a real measurement. It still faces a binary it cannot resolve.
Branch one: build the score on a public benchmark. The industry default is SWE-bench and its descendants โ curated issue-resolution tasks against real repositories, scored by whether the generated patch passes the associated tests. The known failure modes are severe. Contamination: benchmark instances leak into training corpora, and the leakage is difficult to detect because the instances look like ordinary code. Distribution shift: resolving a curated GitHub issue in a repository with a test suite is not the same task as shipping a change in a production codebase governed by undocumented invariants and an on-call rotation. And metric validity: pass at k measures sampling success, not integration success. A model that produces a passing patch on the third attempt scores identically to one that produces it on the first, even though the downstream cost of those two events is different by a factor of three.
Branch two: build the score on customer telemetry. Now the sample is real and the task distribution is real. But codebases differ in ways that are not normalizable. A monorepo with a mature test suite and a legacy service with none produce wildly different agent efficiency for reasons that have nothing to do with the agent. The score becomes incomparable across customers, which destroys its usefulness as a procurement comparison. Worse, the telemetry is private, which destroys third-party auditability. A number nobody outside the vendor can recompute is not a measurement. It is a disclosure.
There is no third branch that is simultaneously public and accurate, and the absence of a methodology document is the most efficient way to avoid saying so.
I have seen this fork before, in a different domain. When the industry tried to standardize proof-of-reserves after 2022, the same binary appeared: publish the address list and let anyone verify (which reveals nothing about liabilities), or publish an attested snapshot from an accounting firm (which is a point-in-time claim about a continuously moving state). Neither branch solved the problem. The exchanges that published reserves without liabilities produced numbers that were arithmetically correct and analytically empty.
Which brings me to the latency question. A score computed quarterly on stale telemetry is a lagging oracle, and lagging oracles are where DeFi's losses concentrate. The parallel is not decorative. In lending markets, the damage from a stale price feed is not that the price is wrong; it is that the wrongness persists long enough for someone to build a position against it. A procurement score that updates annually will be gamed in exactly the same way โ the target adapts faster than the measurement refreshes. Any efficiency index that does not specify its refresh interval has not specified its attack surface.
Who scores the scorer
Cognition is one of the entities being scored. This is a structural fact, not an accusation, and it produces a specific failure mode that governance literature has documented for decades.
The issuer-pays credit rating model failed in 2008 not because the analysts were dishonest but because the revenue relationship determined which methodologies survived inside the firm. Ratings that were harsh lost market share. Ratings that were generous gained it. No individual decision had to be corrupt for the aggregate outcome to be corrupt. The mechanism was selection pressure on methodology, not on people.
Apply the same lens. A rating published by a rated party is a governance artifact before it is a measurement artifact. It can be genuinely useful as an internal diagnostic โ a customer success tool that helps accounts identify waste and improve retention. That is a real product. But it cannot function as an external comparison standard without an independence layer, and independence is not a value statement. It is an organizational property: separate economic interest, separate publication rights, separate ability to be wrong without consequence.
I have a standing skepticism about decentralization claims that resolve to a permissioned node set coordinated by a single corporate entity. The technical properties โ uptime, finality, feed availability โ may all be excellent. The governance claim is still a governance claim. The same distinction applies here. Cognition's score may be technically excellent. It is still a score published by a scored party, and that is a category of artifact with a known reliability profile.
Integrity is not a feature; it is the foundation. A measurement system built without an independence layer does not have an integrity problem at the margin. It has an integrity problem at the base, and no amount of methodological polish on top of it changes the load-bearing properties.
The commercial function is pricing, not measurement
The most likely purpose of this system has nothing to do with efficiency. It has to do with the migration from seat pricing to usage and outcome pricing, and that migration requires a measurement layer to exist before it can be sold.
Seat pricing is what the first generation of AI coding tools used because it was legible to procurement: a fixed number of developers, a fixed monthly cost, a line item that fits an existing software budget category. It is also, for an agent product, an economic trap. A seat-priced agent with an unbounded cost of goods sold is a product whose margin is determined by customer behavior that the vendor does not control. Heavy users destroy the unit economics. Light users subsidize them. The vendor cannot tell which is which without a usage metric.
Usage pricing converts that uncontrolled cost into a pass-through. It is the correct commercial answer, and every infrastructure category has eventually arrived there. Cloud did it. Observability did it โ Datadog's entire position is built on the insight that the party who owns the meter owns the renewal conversation. Data warehousing did it. The pattern is stable: a category becomes unreadable, a measurement layer appears, and the measurement layer absorbs the adjacent traffic.
That is the correct frame for Cognition's announcement. The score is not primarily a customer service. It is the pre-positioning for a pricing change, and the reason it is being announced before it is specified is that its function at this stage is to establish the vocabulary. Once procurement documents contain a column for agent efficiency, the entity that supplies the column's definition has an advantage that no feature release can replicate.
But there is a market-structure problem the announcement does not address. The entity best positioned to own AI cost observability is not an agent vendor. It is whatever sits in the request path. Gateways, proxies, and the model vendors themselves already see every token. LangSmith, Helicone, Braintrust, and Datadog's LLM observability line are already doing cost and token tracking with no conflict of interest and no need to be believed. Cognition would be entering a category where it is not the incumbent and where its primary asset โ being one of the things being measured โ is a liability rather than an advantage.
The cost mirror
There is a structural fact about agent products that the announcement implicitly acknowledges and explicitly avoids: the efficiency narrative is the cost narrative viewed from the other side.
An autonomous coding agent consumes one to two orders of magnitude more tokens per task than a conversational interface. Multi-turn planning, tool feedback loops, context re-reads, retry logic, and verification passes all compound. If the agent is paying third-party API rates for that inference, gross margin is structurally lower than a chat product's, and the variance is driven by customer task complexity, which the vendor cannot forecast.
Under those conditions, an efficiency score does two things at once. Externally it is a customer ROI tool. Internally it is a justification framework for usage tiering, rate limiting, and cost control โ and it is a much more palatable justification than a rate limit, because it is framed as helping the customer rather than protecting the vendor's margin.
Here is the contradiction. If the score succeeds and customers migrate toward low-token usage patterns, and billing is usage-based, revenue compresses. The system would be optimizing against its own top line. The only way out is if the score is used not to reduce consumption but to segment it โ a low-token, low-latency tier and a high-token, high-verification tier, priced accordingly. In that configuration the score is not an efficiency measure at all. It is a price discriminator with a technical vocabulary, and the vocabulary is what makes the discrimination acceptable.
I flag that as inference, not fact. But it is the only internally consistent commercial reading I can construct from the available evidence, and it makes a testable prediction: watch whether the score is accompanied by tiered pricing within two quarters.
Goodhart, with receipts
I have watched metric gaming at close range in this industry, and the pattern is invariant enough to be treated as a law. Any metric that becomes a target acquires a farming industry within one quarter.
During the 2021 metadata investigation I tracked ten thousand token URIs across the top hundred collections. In the same period I was watching marketplace volume figures that were dominated by self-trades netting to zero economically while counting fully in the reported metric. Total value locked inflated through recursive lending and incentive mercenaries that departed the block after the reward window closed. Unique active wallets inflated by airdrop farming scripts. None of these were sophisticated attacks. They were ordinary responses to the existence of a number that someone was willing to pay for.
The AI coding equivalent is already foreseeable. If the score gates procurement, agents will be tuned to protect it. The cheapest protection is task refusal: a declined task costs nothing on an efficiency metric, while a failed task costs a great deal. The first behavioral effect of a published efficiency score will therefore be a rise in confident-sounding deferrals. This will read to buyers as appropriate humility. It will be score protection.
A second gaming vector is benchmark-shaped. If any component of the score derives from a static benchmark, fine-tuning on the benchmark's distribution is a known and cheap operation. The result is a score that rises while production performance does not, which is the exact failure mode that has degraded public model leaderboards for three years running.
| Measurement layer | What it sees | Auditability | Gaming cost | |---|---|---|---| | Vendor billing API | Exact token counts, per request | High within vendor, zero across vendors | High | | Gateway or proxy | Tokens, latency, model routing | High, third-party | Medium | | IDE instrumentation | Accept and reject events | Medium, depends on coverage | Low | | Version control | Merged diffs, review cycles | High, but censored sample | Medium | | Static benchmark | Pass or fail on curated tasks | High, but contamination-prone | Low | | Production outcome | Incidents, cycle time | Very low, rarely instrumented | High |
The table is the argument. The layers with the highest auditability are the ones that measure the least about productivity. The layer that measures what buyers actually care about is the least instrumented, which is precisely why it is not the layer being productized.
The contrarian angle: verifiability may be irrelevant to the outcome
I have spent most of this piece arguing that the score cannot be verified, cannot be compared across vendors, and cannot resolve its own ground-truth fork. All of that is correct and all of it may be beside the point.
Standards do not win on accuracy. They win on default status. The Gartner quadrant is not a measurement in any defensible statistical sense, and it moves billions in procurement because it is a decision shortcut that appears in slide decks and gets quoted by people who will never read the underlying methodology. If Cognition's efficiency score becomes the number that appears in a procurement template, it does not need to be right. It needs to be quotable.
That is the uncomfortable inversion. The absence of a methodology, which I have treated as an evidentiary gap, may be a strategic asset. A named score with no formula cannot be falsified. It can only be contradicted by a competing score, and competing scores require an organization willing to build one โ which is a much higher bar than writing a critique. Publishing a methodology invites reproduction. Reproduction invites contradiction. Silence is defensible.
A second inversion cuts against the premise rather than the execution. Token efficiency and productivity may be negatively correlated in the short run. Cheap tokens encourage exploration; expensive tokens encourage precision. An agent that spends more tokens running the test suite, reading adjacent call sites, and checking invariants is doing what I do manually when I audit a contract โ and that work is the difference between a diff that compiles and a diff that is correct. A score that penalizes token spend may penalize exactly the behavior that produces reliable code.
Correlation is not causation at the level of the proxy variable. The industry is about to optimize a number whose relationship to the outcome it claims to represent has never been established, and the optimization will be aggressive because the number will be visible.
Takeaway: watch the pricing page, not the press release
The score itself is not the signal. The signal is what appears next to it.
If Devin's pricing page shifts from seat-based to usage or outcome-based within two quarters, the grading system was pricing infrastructure and the efficiency framing was the sales motion that made the migration legible. If a methodology document appears with defined metrics, disclosed sample sizes, and a named third-party attestation, the system was a genuine measurement attempt and deserves to be evaluated as one. If neither appears within a quarter, the event belongs in the narrative column and should be cited as such.
The structural risk to watch is upstream. If the model vendors ship an official evaluation or observability surface โ and the incentives to do so are strong, because it lets them price their own agents against a score they control โ the third-party scoring layer resets to zero overnight. That is the same pattern as exchanges publishing their own attested reserves and rendering an entire category of attestation startups redundant. Infrastructure layers get built ahead of demand in this industry, and the ones that survive are the ones that were in the request path when the demand arrived.
There is also a slower signal worth tracking, and it is the one that actually matters to anyone buying these tools. Enterprise buyers are about to acquire their first quantifiable comparison framework for AI coding tools, and it will be supplied by a vendor. The correct response is not to reject it. The correct response is to build an internal evaluation set from real tasks in the real codebase and never let a vendor's number substitute for it.
Measurement is not neutral; it is a claim of jurisdiction. Whoever defines the denominator decides which agents get purchased. The question is not whether Cognition's score is accurate. The question is whether, by the time anyone can check, the number has already been written into a contract.