Abundance of Competence, Scarcity of Trust

What the UK government's AI Scenarios 2030 forgot to model

The Government Office for Science published AI Scenarios 2030 this month, an updated set of five futures built to help policymakers stress test their plans against the next half decade of artificial intelligence. It is a serious piece of work, assembled with a strong panel and a recognised foresight method, and it is candid about its own limitations in a way that not every government document manages. It is worth reading. It also, I think, leaves out one thing that turns out to matter a great deal.

The report's economic emphasis falls on distribution. Its recurring themes are concentration, gains accruing to frontier firms and capital owners, profits leaking overseas, and a widening split between winners and losers. That is a reasonable set of things to worry about. But taken together, the emphasis quietly assumes a roughly fixed quantity of value that gets divided up. What gets less attention is the possibility that the quantity itself grows, and changes shape.

For a foresight exercise about a general purpose technology, in a country whose central economic anxiety is sluggish growth, that omission is worth dwelling on.

The pie can grow, not just move

Here is the second order effect I wish the report had reached for. When the cost of producing software approaches zero, you do not simply get fewer development jobs. You get an explosion of software for use cases that were previously uneconomic, including plain deterministic software that has nothing to do with AI at all. Ideas that never cleared the cost of a build suddenly clear it easily. This is Jevons paradox applied to code: make production cheaper and total consumption tends to expand rather than contract.

The report half sees this. It notes that intermediation businesses may collapse and that execution bottlenecks fall away. It models the destruction. It does not model the creation. There is no concept of induced demand, no new market surface area, no theory of where the next thousand small firms come from.

Worse, it has no model of risk appetite, which is the variable that actually governs new formation. Lower the barrier to entry and you do not just let more ideas through the door. You change who is willing to walk through it. When the expected cost of a failed experiment drops by an order of magnitude, the rational risk appetite of the entire population of founders, and of risk averse incumbents, rises with it. Cautious organisations stop being cautious when failure becomes cheap. They run ten experiments where they used to run one. That is a behavioural feedback loop on the rate of new business formation, and it compounds: more attempts, more survivors, more visible playbooks, still more attempts.

The report's only behavioural lever is "adoption", a diffusion curve. It has a model of uptake and no model of appetite. So it cannot represent the single most interesting thing that cheap software might do to a slow growing economy, which is enlarge it.

The market splits, it does not homogenise

It would be easy to read the section above as an argument that incumbents are in trouble, and the section below as the opposite, so let me join them, because they are halves of one claim rather than two competing ones.

The collapse in the cost of competence cuts two ways, and which way it cuts depends on a single question: can the buyer judge quality for themselves, and what does it cost them to be wrong? Where verification is cheap and the stakes are modest, the barrier genuinely falls, and new entrants flood in. A technical founder or a small capable team does not need anyone to vouch for the output, because they can look at it and know. That is where the explosion happens, out in the long tail of new and previously uneconomic use cases. But where verification is hard and the stakes are high, abundance does the opposite. The buyer who cannot personally assess whether the competent looking thing is actually sound, and who carries real downside if it is not, finds that everything now looks good and that looking good has stopped being information. For that buyer the scarce good is trust, and trust accrues to whoever already has it.

So the same force produces opposite structures in different segments. The market does not converge. It stratifies. New entrants win at the low stakes edge; trusted players win at the high stakes core. The genuinely open competitive question of the next five years is whether the edge players can manufacture trust quickly enough to cross into the core, or whether the established players simply absorb the new capability before they get there.

Why trusted players hold the high stakes core

The report's reasoning on software businesses runs roughly as follows: AI commoditises the build, so the firms that sold the build are exposed. There is something to this, and some intermediaries genuinely will be hollowed out. But I think it points the wrong way for most incumbents, and the reason it does is the heart of the matter.

When the cost of competence collapses, competence becomes abundant. Every deck, every proposal, every app, every website is now well made. And the instant a thing that used to be scarce becomes universal, it stops being a differentiator. Value migrates to whatever is still scarce.

What is scarce in a world where everything looks good and everything is changing quickly? Trust.

Under volatility, stressed buyers do not expand their risk surface. They contract it. They fall back on what they already trust: known brands, incumbent contracts, the account manager whose name they know. The risk averse enterprise was never going to bet a business critical workflow on a six month old startup, and it is even less likely to now, precisely because trust has become the scarce good. So the established software firm with a sticky contract is not simply displaced by a clever new entrant. It loses ground at the edges of its market, where buyers can self serve, and it holds the core, where it absorbs the new capability, keeps the relationship, and watches the high stakes buyer stay put.

There is an obvious objection, and it is in fact the engine of the argument. Cheap creation also means cheap slop, a flood of competent looking noise that drowns discovery. Quite so. A glut of indistinguishable competence is exactly what forces buyers to fall back on trust to cut through it. The slop does not weaken the thesis. It powers it.

Which is why the differentiators that survive are brand, distribution, switching costs, contracts that are painful to churn, and human relationships. And why, when output quality is universally high, the binding question stops being "can it be built" and becomes "can it be trusted". That is assurance. Verification. Provenance. It is a market the report's framework cannot even render, because trust appears in its model only as a drag coefficient on adoption, never as the thing the whole economy reorganises around.

Each scenario scored from one to five on six axes and plotted as a polygon. Source: Government Office for Science, AI Scenarios 2030.

A note on the method

The method here is a recognised one, and the team uses it conscientiously. Three observations are still worth making, because they all point the same way.

The first is about the radar charts. Each scenario is scored from one to five on six axes and plotted as a polygon. It is a clear way to compare scenarios at a glance, and I understand the appeal. The report does not, though, show a rubric for how a given narrative becomes a given number. Without that, the scores are best read as illustrative rather than measured, and it is worth holding them a little more loosely than the precision of the charts invites.

The second is about independence. The report treats its six critical uncertainties as separate axes, and its own glossary notes that such axes should ideally be independent of one another. In practice, capability, adoption and labour displacement move together, and to the report's credit it acknowledges this link in its limitations. The consequence is that three of the five scenarios sit on the same capability trajectory with other dials adjusted around it. The futures are real and useful, but they are less independent than the framing suggests, and that is worth bearing in mind when using them to stress test.

The third is the assumption of no policy intervention. This is a deliberate and defensible choice: it keeps the scenarios neutral so that any policy can be tested against them. The report is explicit about it. The difficulty is that it then concludes outcomes are contingent on governance and institutional capacity, which is to say contingent on exactly the variable that has been set aside. There is a parallel worth drawing here with epidemic modelling, where assuming a population that does not adjust its own behaviour produces tidy curves and misleading ones. Firms and governments are adaptive agents. They restructure the moment an incentive becomes visible, often well ahead of any formal policy, and a set of scenarios that holds them still will tend to understate how quickly the ground moves.

One last thought, offered in the same spirit. A report about AI's analytical power is a natural candidate for using some of that power on itself: diffusion modelling, sensitivity analysis on the scores, a sweep across the uncertainty ranges it defines. That tooling is now inexpensive and fast. A future edition that leaned on it could turn some of the qualitative judgement above into something closer to evidence.

The live proof: KYC for AI

The story broke the same week the AI Scenarios 2030 report published.

It is worth grounding all of this in something concrete, and there is a useful example from the same week the report published.

Anthropic disabled access to its Fable 5 and Mythos 5 models for every customer worldwide, to comply with a United States export control directive issued under national security authorities. The reported trigger was a technique for jailbreaking Fable 5's safeguards. The directive itself ordered the company to suspend access by any foreign national, whether inside or outside the United States, including the company's own foreign national employees. It is contested, Anthropic called it a misunderstanding and is working to restore access, and it may yet be reversed. None of that matters for the point. The lever has now been demonstrated in the open.

Read the mechanism carefully. A restriction keyed to nationality rather than geography means that being inside the country no longer protects you. Only verified identity does. To comply with "no foreign nationals", a provider has no option but to verify the nationality of every user at the point of access. The directive does not suggest a permissioning layer. It mathematically requires one. That is Know Your Customer for AI development.

Now notice that this is the same dynamic from the other end of the capability spectrum. At the commodity layer, the market makes competence abundant, so trust becomes the scarce differentiator. At the frontier layer, the state makes capability deliberately scarce, so access becomes permissioned, and permission requires the provider to know exactly who you are. Two mechanisms, market from below and state from above, both converging on a single conclusion. In a world of abundant competence, the two scarce goods are trust and permission, and both require knowing who you are.

There is a second order effect here that compounds the first half of this piece. Only providers who can operate nationality verified, export controlled, identity gated access at scale can comply with a regime like this. That is not a moat the average startup can dig. So the same gravitational pull toward trusted incumbents that the market creates from below, the state now reinforces from above. The risk averse buyer routes toward big tech for trust. The regulator then makes compliance capability a precondition of supply. Concentration arrives twice, by different roads.

And it illustrates the independence point from earlier better than any argument could. That single event, one afternoon in June, was a security incident (a jailbreak) that triggered an export control (model access) imposed under national security authority (global cooperation), whose downstream effect pushes the market toward compliance capable incumbents (distribution and concentration). One real event moved four of the report's six axes at once. It is a vivid reminder that, in the world the scenarios are describing, those axes pull on each other, and the most interesting effects live in the wiring between them.

The question worth asking

The report's instinct is to ask how the gains are divided. The questions I would add are who grows them, who is now willing to attempt something they would not have risked before, and where value goes once competence is free.

My own answer is that value goes to whatever stays scarce. Right now that is trust and permission. The market is building a Know Your Customer layer for software from below. Last week, the state started building one from above. A 2030 foresight exercise has room for both, and the next edition would be stronger for making space for them. To borrow an image: the report describes the lighting of the carriage lamp very precisely, and does it well, on what may turn out to be the eve of the motor car.

---

Alex Brooker is the founder of Airside Labs, an AI security, testing and assurance consultancy. Airside Labs works on adversarial testing and evaluation of AI systems in business critical and regulated deployment.