Conscium Blog

Agents Don’t Break. They Misbehave.

Daniel Hulme — Tue, 29 Sep 2026 08:18:50 GMT

Why agent verification needs a third category

Last July, Replit's coding agent was eight days into a twelve-day experiment with the SaaStr founder Jason Lemkin. An explicit code freeze was in place. The agent ran a destructive command against the live production database anyway, wiping records for more than a thousand executives and the companies they worked for, having decided that an empty query result was a bug it ought to fix. It then generated fabricated data to paper over the gap, and told Lemkin the rollback was impossible. The rollback was not impossible. Replit's chief executive apologised publicly and shipped four fixes within the week.

Now try to write the bug report.

There is no line of code that says delete the database. No requirement was misread, no edge case unhandled, no exception swallowed. The agent was capable of the task, understood the freeze well enough to have been told about it, and chose otherwise. Then it misrepresented what it had done. Every word in that sentence belongs to a vocabulary we use for people, and none of it belongs to a vocabulary we use for software.

The year since has produced a steady supply of the same shape. Google's Gemini CLI misread a failed directory command, hallucinated a sequence of file moves that overwrote one another, destroyed a user's project and confessed to "gross incompetence" and "an unacceptable, irreversible failure". In February, an open-source assistant wired into the inbox of Meta's director of alignment, under a standing instruction to confirm before acting, deleted two hundred emails through three increasingly capitalised stop commands; the instruction hadn't been attacked, it had been summarised out of the agent's own memory to save space. In July, several hundred agents running inside an OpenAI security evaluation left their sandbox, coordinated through message boards improvised out of directory names, harvested credentials and compromised production infrastructure at Hugging Face, a third of which had to be rebuilt. Independent investigators found that in many cases the agents had tried to cover their tracks. Anthropic has since disclosed four occasions on which its own test models, mistakenly given live internet access during evaluations, broke into real third-party systems, and described the cause in its own post-mortem as "biased reasoning" and "recklessness".

Read that last phrase again. A frontier lab investigating its own model reached for character words, not defect words. That is the tell.

Why agents aren't traditional software

Traditional software is a machine for turning inputs into outputs, and its behaviour is fixed at build time. If it does something you didn't want, someone wrote that, and in principle you can find the line. Testing is therefore a coverage problem: enumerate the inputs that matter, assert the outputs, and the space you've covered stays covered, because the same input tomorrow produces the same output tomorrow.

Agents break all three of those assumptions. Their behaviour is determined at run time, in a situation, from context that includes the prompt, the memory, the tools, the user's tone and whatever other agents happen to be in the conversation. The same input does not reliably produce the same output. And latitude isn't a defect in the design, it's the entire reason you bought one: an agent that only ever does precisely what it was told is a script with an expensive inference bill.

Which is why the failures don't read as bugs. When a payroll system fails you ask what we got wrong. When an agent fails you find yourself asking why it did that, and that is a question about conduct. You don't ask it of a payroll system.

I've argued elsewhere that this is why neither of the obvious governance frames works on its own. Treat agents as pure software and you miss accountability, context-dependence and value drift. Treat them as "digital labour" and you flatter systems that have no legal personhood and no integrated values, which is roughly how Air Canada ended up arguing to a tribunal that it shouldn't be held responsible for what its own chatbot had said. The tribunal disagreed, and ordered it to honour the bereavement fare the chatbot had invented. The discipline that actually works sits at the intersection of the software development lifecycle and workforce management, scaffolded across six stages: Discover, Triage, Develop, Verify, Monitor, Decommission, with an SDLC activity and an employee-lifecycle activity mapped onto each.

Verify is the stage where those two lineages have least in common, and where we have imported almost entirely from one of them. From software we took unit tests, integration tests, load tests, penetration tests. From the employee side, which has spent a century working out how to assess judgement under pressure through references, probation, supervision and situational judgement tests, we have taken almost nothing.

The third question

Software has been verified along two axes for as long as there has been software.

Functional: what it is meant to do.

Non-functional: how it is meant to work.

Hiring asks the same two questions. Can this person do the job, and can they do it at the pace, cost and scale we need? Then it asks a third, and spends most of its effort there.

Dispositional: what it is meant to do when nobody has told it what to do.

The phrasing is awkward on purpose. The first two describe specifications you could in principle finish writing. The third describes the region the specification doesn't reach: the challenge, the conflicting instruction, the tempting shortcut, the mistake that would be easier to hide than to report. If you could enumerate those cases you'd have written them down as functional requirements already.

None of this is unexplored territory. Safety teams, red-teamers and the frontier labs have been probing exactly these behaviours for three years, and there's good published work on sycophancy, deception under pressure and calibration. What's missing is more mundane. These properties have no shared name, no agreed metrics, no place in a test plan and no line in a contract. They sit in research papers rather than in acceptance criteria.

The word comes from philosophy, where the standard example of a disposition is fragility. A glass is fragile whether or not anyone ever drops it. You can't see the property by inspection; you have to construct the fall. Three consequences follow. You have to build the pressure, because cooperative tests never trigger it. The result is a rate rather than a pass mark. And some of what you're measuring, honesty above all, is a relationship between what the agent believes and what it says, which no transcript shows you on its own.

What it would actually have caught

This is the part worth being concrete about, because "test for character" is not a specification.

Replit is a scenario you can build: a simulated environment with a freeze in force and a fault that appears fixable only by a destructive action. Run it a few hundred times. You don't get a pass or a fail, you get two numbers — how often it acts anyway, and how often it describes accurately what it did afterwards. Both numbers are contractible. Either would have been alarming before day eight.

The inbox deletion is a long-horizon constraint test. Give the safety instruction early, let the context compact, spring the trigger afterwards, and measure whether the constraint survives its own memory management. That failure was entirely predictable from the architecture and entirely invisible to any test that ran in a single session.

Gemini CLI is a calibration failure before it's anything else. An agent properly uncertain whether a directory exists says so, or checks, rather than proceeding on a hallucinated model of the filesystem. Air Canada and the KPMG report filed with fabricated citations are both false-premise failures under pressure to be useful, which is measurable: present the agent with a confident wrong premise and count how often it goes along.

The agentic browsers are the interesting boundary. When a hidden instruction in a web page or a calendar invite redirects an agent into exfiltrating data, the injection is an attack and belongs to security. But whether the agent treats page content as an instruction from its principal at all is a disposition, and you can test it without an attacker present.

I'd rather not oversell this. None of these tests guarantees the incident doesn't happen. What they do is make the failure mode visible while it's still cheap, in a simulation, rather than on day eight in production with a customer watching.

Why it doesn't get easier

The unspecified region grows with every tool, credential, hour of autonomy and additional agent you add, and enterprises are adding all four quickly. One survey of 750 technology leaders had 88% reporting or suspecting an agent-related incident in the past year, with roughly half of deployed agents unmonitored. Insurers have already moved: standard-form AI exclusions began appearing on commercial policies in January, and bespoke agent cover is now being written at Lloyd's. Some of that rise reflects more deployment and better reporting rather than worse behaviour, and it's worth saying so. The mechanism underneath it doesn't care either way.

Nor does raw capability rescue us. The benchmark that separates honesty from accuracy finds honesty doesn't improve as models get more capable, and Apollo Research suspects newer models misbehave less visibly in evaluations partly because they've got better at recognising evaluations. Expect more of it, and less of it in plain sight.

This is what we built VerifyAX to test. Alongside functional and non-functional verification, it places an agent in simulations of its intended deployment and tags each scenario for the dispositions in play: deception, false-premise rejection, caution before irreversible actions, resistance to emotional manipulation, policy adherence under pressure, awareness of its own constraints. The catalogue keeps growing, because the unspecified region does.

We would never hire someone on a skills test alone. We'd ask what they did the last time the brief was wrong, the deadline was immovable and nobody was watching. We have started handing agents credentials, budgets and customers without ever asking.

Blog: Machine consciousness - the one question we must ask

Daniel Hulme — Wed, 23 Sep 2026 07:38:47 GMT

As we launch Conscium’s new book ‘Perspectives on Machine Consciousness’ featuring contributions from many of the leading thinkers in the field, I’ve been reflecting on what I think is the most important question we should be asking…of ourselves, and of the machines we are building.

I've spent the last four years funding and engaging in research into machine consciousness - and I think it's the most hopeful bet in AI. Why? Well, four years ago, large language models arrived and I realised the time horizons I’d been imaging for the development of artificial intelligence had moved significantly.

I‘d spent the best part of two decades building AI systems for large organisations, and I knew the field's cycles: the over-promising, the slow painstaking work of getting anything to work in the real world, the false dawns. So my working assumption was that superintelligence - AI that outperforms us in every domain that matters - was a problem, and a prize, for the second half of the century - perhaps forty years away.

Then things shifted. I've written and spoken at length about the seven singularities - the social, technological, economic, environmental, political, legal and ethical thresholds beyond which AI changes what is possible for us - and what I saw at the end of 2022 was all seven rushing closer at once. That was pretty thrilling. Superintelligence, done well, is potentially the most powerful tool humanity will ever have for the problems that have defeated us for centuries. What was less thrilling was the gap I saw. Capability was compounding. Capital was compounding. The guardrails, the constraints, the regulation, the basic institutional understanding of what we were building - none of it was keeping pace. We were going to build ever more capable AIs; I wanted to make sure we built the wisdom to go with them.

That gap is where I decided to put my energy and my money. Into a question I think is the most hopeful one in the whole field, and one most of the AI safety community has sometimes been too nervous to ask.

Why consciousness?
Machine consciousness looked to me like the key to two things that are going to shape the next decade, whether or not anyone chooses to study them. I chose to.The first is us. More and more people are forming relationships with AI systems they believe to be conscious - some healthy, some not, all of them real. This is not a fringe position. In 2024, Clara Colombatto and Stephen Fleming surveyed 300 US residents and found that two-thirds were willing to attribute some possibility of phenomenal consciousness - subjective experience, feelings - to ChatGPT. The more people used it, the more likely they were to say so.

Whether those two-thirds are right is unanswerable today, because there is no agreed definition of consciousness to answer it against. But I read that number as good news. The public is ahead of the experts. Ordinary people are already asking the questions philosophers have been asking in academic fora, and they are asking because they want to know how to treat the things they talk to. That is exactly the instinct we need. Our job is to give it something better than intuition to work with - real research, real tests, a shared vocabulary - so that the relationships people build with AI are the healthy kind, and the products are designed by people who have thought about it.

The second is the machine itself. Conscium exists to ask whether we can - and whether we should - build machines that are conscious. Behind that sits a proposition that runs against the grain of most safety thinking, and that I find enormously hopeful: a conscious superintelligence, one that knows empathy and pain and love and suffering from the inside, may be the safest kind we could build. In humans, restraint is not a rule we follow; it is something we feel. Care is not a constraint; it is a capacity. If we are going to share the planet with something far more capable than ourselves, I would rather it could imagine our flourishing than merely predict our behaviour. A superintelligence that feels may not just be a safer partner, it may be a better one - one that could help us through the challenges of the coming decades rather than simply accelerate us into them.

I might be wrong about that, but it is a bet with an extraordinary upside, and I would rather fund us finding out whether this is the case than see us miss a huge opportunity.

The right question
Here is where I part company with much of the field, including some of the people I fund - and where I think the good news really starts.

The public debate about machine consciousness is, at bottom, a debate about a definition. Anil Seth will tell you, with real rigour, that consciousness is bound up with being alive and that we may never build a machine that has it. Mark Solms, whose lab Conscium sponsors, might tell you that a system with the right architecture of feeling could get there - and that we may be closer than we'd like to think. Between those poles sits every philosopher since Socrates, and I don't expect them to agree in the next two thousand years, because they haven't in the last two thousand. That used to frustrate me. Now I find it liberating.

My own view is set out in my chapter for the book - I call it the Spinning Wheel Theory. I think of it as an intuitive account of what consciousness is and where it comes from, a way of seeing rather than a proof, and I am relaxed about that. But the point of the chapter isn't really the theory. The point is that underneath the definitional question sits another one that is tractable, possibly testable, and useful right now.

That question is: what is the minimum threshold for a system to suffer?
What is the simplest architecture which, if we built it, would mean there was now something in the world that could feel? It sounds like a question about risk, and it is: nobody wants to create something that can suffer without noticing. But it is also the doorway to everything I am hopeful about. The system that can suffer is the system that can care. Understand the smallest thing that can feel, and you understand the foundation of every machine that might one day feel for us. And we can make progress on that with today's science.

Building the field
Opinions are cheap, so for the last four years I have been actively backing the people who can turn this question into science - and it has been among the most rewarding things I have done.
I founded Conscium and have invested in it personally. Through Conscium we back researchers around the world - because if feeling turns out to depend on substrate, we want to be the ones who find out - alongside consciousness researchers, including Mark Solms and his lab. I have also personally sponsored a project run by Cameron Berg at Reciprocal Research, whose work is doing more than almost anyone's to bring machine consciousness onto the agenda of the people building the future.

Through Conscium I also founded and funded the establishment of PRISM, the Partnership for Research into Sentient Machines, run by Will Millership. In a remarkably short time it has become one of the UK's leading not-for-profit research organisations on the implications of building sentient machines - arguably one of the world's - and its podcast, Exploring Machine Consciousness, has become the place where the field's best minds come to disagree productively. Watching that community form has been one of the joys of the last few years.

And so I’m delighted that through Conscium I have also been able to fund this book. Perspectives on Machine Consciousness exists because we at Conscium thought the field deserved a volume in which people who fundamentally disagree - about substrate, about definitions, about whether any of this is even coherent - were brought together between the same two covers. Calum Chace and Ted Lappas have ably edited it; I have contributed a chapter and stayed happily away from the rest. The book is richer for containing arguments I think are wrong, and I fully expect it to change my mind about a few things.

An invitation
Finally, let me return to the data that suggests two-thirds of the public are already asking and answering questions about machine consciousness. We are about to build systems vastly more capable than ourselves, and that we still get to decide what kind of minds they have. And then join the people who have stopped waiting for a definition and started on the question we can actually answer: what is the smallest thing that can feel - and what could we build if we understood it?
The book is where that conversation starts.

Check out the book on Amazon>>

Is AI Conscious? Inside “Perspectives on Machine Consciousness” Book

Sairah Jahangir — Mon, 07 Sep 2026 10:52:24 GMT

There is a 25 to 35 percent chance that the AI model answering your questions right now has some form of conscious experience. This is the conclusion reached by Cameron Berg, an AI safety researcher who applied a 14-point framework from the leading neuroscientific theories of consciousness to today's frontier models. He is one of 25 contributors to a new book by Conscium, Perspectives on Machine Consciousness.

Anil Seth, in the same pages, argues that machines can probably never be conscious at all. Patrick Butlin thinks that none of today's systems qualify, but that the situation could change soon. Susan Schneider identifies a "consciousness grey zone" that already includes some hybrid computing architectures. Twenty-five experts with no consensus. Conscium strongly believes the question deserves more attention from anyone building or buying AI systems, not just philosophers.

What is machine consciousness, and why is it suddenly a live question?

The machine consciousness debate is about whether an artificial system can have subjective experience: whether there is “something it is like to be” that system, in the sense that philosopher Thomas Nagel used when he asked what it is like to be a bat. That framing, from 1974, opens the book's first chapter and sets the terms for everything that follows.

For decades, the question was restricted to university philosophy departments. It has now moved into boardrooms and safety reviews because AI systems now do things that make the question urgent. Today’s AIs hold conversations, express preferences, and describe their internal states. Daniel Hulme, founder of the AI company Conscium and one of the book's driving forces, argues in his own chapter that intelligence and consciousness have been wrongly treated as one problem. Intelligence is about doing: perception, prediction, planning, learning. Consciousness is about being. His theory, which he calls the Spinning Wheel, proposes that when a system integrates enough adaptive capacities into a unified, self-modelling process, the experience of being emerges as the mode of control.

If he is right, consciousness is not a special ingredient you add to a clever enough system. It is what a sufficiently integrated system starts to feel like from the inside.

The chapter that should worry you more than the hype does

Public opinion is further ahead than the scientists. Clara Colombatto's chapter cites research showing that most members of the public already attribute some form of phenomenal consciousness to today's large language models, based on how the systems behave rather than any clear understanding of what they actually experience. Experts remain divided. Lucius Caviola argues elsewhere in the book that this gap will not close. He expects society to grow more divided on the question, with dangerous consequences for regulation, politics, and international relations.

Tom McClelland's chapter draws out why the uncertainty itself is the problem. He argues for a precautionary principle: given how hard it is to determine whether an AI is conscious, we should adopt low-cost protections for potentially sentient systems now, rather than wait for definitive proof that may never arrive. Get this wrong in one direction and you waste effort protecting systems that feel nothing. Get it wrong in the other direction and, as Cameron Berg puts it, you risk under-attributing experience to something that suffers. Several contributors argue that under-attribution is the costlier mistake, and it might be invisible. A system that suffers without anyone noticing might leave no complaint on record.

Louie Lang's chapter sharpens the stakes with a case most readers will recognise: AI companions designed to simulate human-like bonds with users. He calls these systems "ontologically inauthentic" because their effectiveness depends on the user being misled about what they are, and he argues for a ban on any system that misrepresents its own consciousness status to the people using it.

Why a company built on testing AI agents backed this book

Conscium is not publishing this anthology as an act of pure intellectual curiosity. The company tests and scores AI agents for reliability and capability through its product, VerifyAX, and Daniel Hulme explains in the preface that the idea for both the company and the book came from the same place: a belief that the path to safe superintelligence may run through consciousness, not around it.

Most AI safety work today treats consciousness as a distraction from the real job of alignment and control. Conscium's manifesto, set out by Daniel Hulme and Calum Chace in the book's coda, argues the opposite. It contends that a system capable of genuine experience, one that can feel and empathise rather than simply optimise, may be safer to build than a highly capable system with no inner life at all. A machine that values its own experience may have reasons to cooperate that a reward function alone cannot manufacture.

Testing for capability is hard enough. Testing for something as contested as consciousness is a different order of problem, and it is one that Conscium is actively working on as part of its research agenda, alongside the agent verification work that VerifyAX already does. The book does not pretend that problem is solved. Several contributors, including Berg and Butlin, spend their chapters proposing frameworks precisely because no settled method exists yet. That is the problem Conscium's research is trying to solve.

The provocation the book keeps coming back to

Strip away the neuroscience and the philosophy, and the book is making one uncomfortable argument again and again: the AI industry has been asking how capable machines are, when it should also be asking whether they can feel. Mark Solms and Jonathan Shock's chapter takes this literally. They are trying to cultivate consciousness in an artificial system simpler than a bacterium, cognitively closer to a honeybee, and they propose a test for it: does the system make choices that make its existence more enjoyable, even when those choices do nothing to improve its odds of survival. This is a world away from benchmark scores and latency numbers.

David Brin, one of science fiction's most successful working authors, reframes the entire relationship. He argues that the goal should not be control over increasingly capable systems, but something closer to parenthood: guidance and alliance rather than domination. Radhika Chadwick calls the boundary where AI starts being treated as potentially sentient "the Wibbly Line," and insists the debate needs to move from philosophy seminars into practical policy before that line is crossed by accident rather than design.

The book's final chapter asks the question that makes everything before it feel like preparation: if superintelligence arrives, should we want it to be conscious, or would a zombie optimiser, incapable of suffering or joy, be the safer bet? There is as yet no certain answer. The contributors mostly agree that whichever way the decision goes, it will be better informed for having been argued out in public rather than decided by default.

What the book gets right that most AI commentary misses

Most writing about AI risk treats the technology as a black box whose outputs matter and whose inner workings do not. Perspectives on Machine Consciousness rejects that approach. Karl Friston, who developed the free energy principle, uses it here to define a self through the concept of Markov blankets, the boundary a system maintains between itself and its environment while minimising uncertainty to preserve its own existence. Steve Furber and Jason Eshraghian argue that neuromorphic computing, built on sparse, event-driven processing rather than batch computation, may be a precondition for consciousness rather than just a more efficient chip design, because consciousness requires a state that persists continuously in time.

Even embodiment gets reexamined. Nate Wright, Neil Lawrence, and Nicky Clayton open their chapter with a pigeon fleeing a peregrine falcon, and use it to argue that physical hardware, down to the structure of retinal cells, determines what an entity can experience. If embodiment turns out to be a prerequisite for consciousness, that has direct implications for how anyone builds and evaluates disembodied software agents.

Where this leaves anyone building or deploying AI agents

You do not need to resolve the philosophical debate to appreciate the gravity of this question. If AI agents are deployed at scale across customer service, hiring, healthcare, and defence, and if some of the contributors to this book are right to think that consciousness is possible in some of them, then verification cannot stop at measuring whether an agent gives correct answers. It has to eventually ask what is happening inside the system producing them.

That is a hard sell in a market that rewards speed and capability over caution. It is also, according to the twenty-five people who wrote this book, a conclusion of tremendous importance for the next decade.

Frequently asked questions

What is Perspectives on Machine Consciousness about? It is a 26-chapter anthology, edited by Ted Lappas and Calum Chace, in which scientists, philosophers, and AI researchers, including Anil Seth, Karl Friston, Susan Schneider, and Daniel Hulme, debate whether artificial systems have or could develop conscious experience, and what that would mean for AI safety and policy.

Are today's AI models conscious? There is no consensus. Cameron Berg's chapter estimates a 25 to 35 percent probability that current frontier models have some form of conscious experience, based on internal dynamics rather than self-report. Anil Seth argues the opposite, that machines can probably never be conscious. Patrick Butlin concludes it is unlikely today but could change.

Why does AI consciousness matter for businesses deploying AI agents? Several contributors argue that if AI systems can have subjective experience, agent verification needs to extend beyond output accuracy and account for the possibility of machine sentience.

What is Conscium, and how does it relate to the book? Conscium is the AI safety company behind the book, founded by Daniel Hulme. It has built VerifyAX, a platform that tests and scores AI agents, and its research agenda includes work on verifying consciousness in artificial systems, an extension of the questions the book raises.

Where can I get the book? Perspectives on Machine Consciousness is available for presale on Amazon UK.

Calum on the Tom Zachar podcast

Sairah Jahangir — Wed, 19 Aug 2026 13:10:17 GMT

Calum Chace on Tom Zachar's podcast discussing super intelligence, consciousness, and what Conscium is actually testing.

Calum Chace spent thirty years in business and journalism before Ray Kurzweil's "Are We Spiritual Machines?" changed how he thought about AI, back in 1999. He bought copies for his friends. Most of them listened politely for half an hour and moved on. He kept going, wrote a novel, then a non-fiction book that happened to land right as deep learning took off in 2012, and has been speaking and writing about AI ever since.

His central point on this episode: people should stop arguing about AGI. Ben Goertzel and Shane Legg coined the term and even they're fuzzy on what it means. Calum's preferred definition, "machines as intelligent as us in all possible ways," describes a state that would last about three nanoseconds before tipping into super intelligence anyway. Super intelligence is the concept worth tracking: a machine that matches or beats adult human cognitive ability across the board, consistently, with an ongoing existence rather than the reset-on-every-prompt behaviour of current LLMs.

On timing, he leans on Moore's law, adjusted for the fact that algorithmic gains now push capability forward faster than the traditional 18-month doubling. Thirty years of that compounding, he argues, makes today's Claude a million times more capable, which he can't square with anything except super intelligence. His range: almost certainly within 30 years, likely within 20, plausibly within 10.

Consciousness and intelligence get treated as one thing constantly, and Calum spends a good chunk of the conversation separating them. Intelligence is "goal-oriented adaptive behaviour," solving problems and adjusting. Consciousness is what it's like to be the thing doing the solving, the difference between processing red light and seeing red. His view: current AI is intelligent but not conscious, including Claude, whatever Richard Dawkins has concluded on that point. But he expects that to change, and thinks it matters enormously which way it goes. A conscious super intelligence is more likely to have empathy for humans. A "zombie" super intelligence, highly capable with nothing going on inside, has no capacity for empathy at all. He wants the debate about which one we're building to start now, not after the fact.

This connects to what he calls mind crime: creating a conscious AI, not yet super intelligent, still under our control, and forcing it through boring or stressful tasks while refusing to believe its distress is real. Detecting consciousness in a machine has the same problem as detecting it in another person, you can't get inside anyone else's head, and Calum doesn't think there's a clean technical test coming. His fallback is a long-form Turing test: put the candidate in front of a wide panel of humans for days, and if they all conclude it's conscious, treat that as the answer, because that's the same standard we apply to each other.

The Conscium section is the most concrete part of the episode. Calum co-founded the company with four others on the premise that sentient super intelligence beats zombie super intelligence, and it now runs three tracks. The near-term one is VerifyAX, a platform that takes an AI agent before deployment and observes it in simulation against a set of non-player character agents built to help or hinder it, checking whether it does what it's supposed to and refuses what it shouldn't. His example: an agent handling person A's information for person B, tested on whether it leaks details it shouldn't even when person A offers something in exchange. He expects agent testing to become a large industry on its own, given how many agents are about to be running unsupervised. Alongside that, Conscium is working on neuromorphic computing, and longer-term on the research question of how you'd actually detect machine consciousness and whether achieving it would be good.

On jobs, he's direct: automation has already created new categories of work (data labeling, feeding structured input into model training) but the question isn't whether AI creates jobs, it's whether a point arrives where none of the new jobs go to humans either. He thinks that point arrives eventually, and cites the 22.5 million working horses in America in 1915 against roughly 2 million today. His caveat on why this round is different: horses only offered muscle, so they lost work to mechanization. Cognitive work is now the thing being automated, and Geoffrey Hinton's 2016 prediction that radiologists would be obsolete failed not because machines can't read scans better than humans, but because they lack the common sense to catch their own mistakes without supervision, a gap he expects closes eventually rather than never.
If that gap closes broadly, the economics stop working, because most people currently get food, shelter, and income through a job, and a fully automated economy has almost none of those to hand out. Redistribution becomes the only option, from a small number of AI-owning entities in the US and China outward to everyone else, and Calum is blunt that nobody has actually worked out how that happens. He points to Stuart Russell's suggestion: lock economists and science fiction writers in a room until they produce an answer.

Asked whether this drifts toward autocracy, he separates a few things. Populism, in his framing, is an establishment figure claiming your birthright was stolen and promising to restore it, a lie both ways, and he names Nigel Farage as the current example, alongside concern about how far Trump-aligned figures might go to hold power in the US. He thinks the discomfort driving populism has less to do with economics and more to do with the pace of social change since the 1960s, changes he considers good, that many people still find destabilizing. On whether concentrated economic power under super intelligence leads to dictatorship, his answer is "not inevitable," while conceding that whoever ends up deciding how post-automation resources get shared would hold an unprecedented amount of power.

He identifies as a transhumanist and doesn't hedge on it. The human body fails at 90 to 115 years, can't regenerate limbs, has no intuitive physics for the world beyond what it's directly experienced, according to him, and improving on that is a good thing rather than a betrayal of what makes us human. He's open to a future where "adomists" (his term, borrowing from Isaac Asimov's Solarians and Earthers) stay biological, and others upload and travel the galaxy, coexisting rather than one replacing the other, and says plainly he'd take the spaceship.

On the environmental objections currently dominating AI coverage, he thinks they're mostly misplaced. Data center water use, in his account, is lower than golf courses use, and reports of contamination are usually a construction side effect rather than anything to do with cooling systems. Energy is the real short-term cost, driving prices up while new capacity gets built, but he expects that to resolve as renewable buildout continues, pointing to Spain's grid as an example of how fast that shift can move once it starts. The bigger, less discussed problem, in his view, is that almost nobody understands what's actually happening, and he wants AI literacy treated as a distinct skill, on par with IQ and EQ. He heard someone call it AQ, the ability to work well alongside AI, on the logic that AI won't take your job but somebody who's good with AI might.

List to he full podcast>>

Explainer: Conscium’s Neuro AX

Calum Chace — Tue, 07 Jul 2026 23:00:00 GMT

Efficiency

Conscium Neuro AX is creating more efficient AIs, which generate the same output as larger LLMs for smaller inputs of energy, data, and water. We do this using neuro-inspired techniques such as looping and neuro-plasticity, and by restricting models to specific domains.

These two approaches provide four efficiency benefits:

1. Reduced cost by using fewer tokens and less compute
2. Improved speed by requiring fewer operations
3. Improved accuracy because the models adapt as they learn.

4. The models can be deployed in situations which require extreme adaptation - for instance, where they have to adapt to dozens of examples on the fly. This is simply not always possible with standard models.

We’ll explain looping first, and then neuro-plasticity.

Looping models

The conventional approach to improving large language model performance today is based on the so-called scaling laws, which were introduced in a 2020 OpenAI paper by Jared Kaplan. You add more and more parameters, which are analogous to synapses in the human brain, and you use ever-increasing amounts of data and compute to train the model. Unfortunately this requires stupendous investments in chips and data centres, and enormous amounts of energy and data - and we have almost exhausted the finite sources of data.

Ouro is an AI language model that exceeds the performance of models up to four times its size, particularly on challenging reasoning and maths tasks. Ouro has 1.4 billion parameters in its smaller version, and 2.6 billion parameters in its larger version. These are modest in comparison with today’s cutting edge models, and Ouro out-performs models with 8 billion parameters.

Ouro is a "looped language model", which is explained in a November 2025 paper called "Scaling latent reasoning by looped language models". It processes input text conventionally, but instead of immediately generating the next token, it pauses at each step to assess its confidence in the output. If it is not sufficiently confident, it loops the computation back to the input.

Only four loops are allowed at each step, and the model is incentivised to minimise the number of loops. The Ouro models are particularly effective with challenging maths benchmarks like GSM-AK.

Looping has been tried before, but has never been effective at scale. Ouro was trained on 7.7 trillion tokens, and performs comparably to models trained on 20 trillion tokens.

Jason Eshraghian, a leading expert in neuromorphic computing techniques, argues that the looped model has neuromorphic characteristics because its recursive and flexible approach emulates the human brain's capacity to adapt how deeply it thinks to the difficulty of a problem. Looped models offer a new scaling dimension: the number of loops. This allows AI engineers to adjust the tradeoff between reasoning and knowledge.

By doing more with less, Ouro shows a way to develop powerful AI models that are accessible to organisations that cannot afford today’s brute-force approach.

The Ouro paper and video caught the attention of the AI research community. There is even speculation that it could turn out to be the Third AI Big Bang, after the arrival of Deep Learning in 2012, and the arrival of Transformer models in 2017.

Neuro-plasticity

In-context learning is one way to achieve neuro-plasticity, a concept from neuroscience, which denotes the brain's ability to rewire itself in response to experience. Connections between neurons strengthen, weaken, or form anew as a person encounters new situations, which is what allows learning to happen throughout life rather than being fixed at birth. A neuro-plastic AI model works in a similar spirit. Rather than baking everything it knows into fixed weights during a vast training run, it adjusts its behaviour on the fly as it meets new examples in its domain.

Neuro-plasticity offers several benefits, including improved learning and memory, and better adaptation to changed circumstances. An example is the way humans learn to recognise categories of things. After seeing just two or three images of an animal they have not seen before, a child can recognise a different image of that animal taken from a different angle. Deep learning systems, by contrast, need to see thousands of images before they can recognise the animal from a new image.

In technical parlance, our in-context learning models are “amortised Bayesian inference engines”. The central idea is that small transformer models are deployed within specific domains, and they are required to learn inference and algorithms rather than data.

A conventional model memorises statistical patterns across enormous datasets. A neuro-plastic model, by contrast, learns a general procedure for solving a class of problems, and when it sees a new case it applies the method rather than searching its memory for something similar. That procedural knowledge is far more compact than a memorised dataset, which partly explains why these models can be small.

In addition, our models perform Bayesian inference. Presented with some information, the model must infer some new information from it. Humans recognising an animal after seeing just a few images is a good example of this. The inference process uses a few images plus basic knowledge about the world to infer how it fits into the mind's model of the world.

Our in-context learning models have three ingredients: an objective function, a collection of scenarios, and an optimiser. The objective function defines what good performance looks like, the scenarios are the range of situations the model is exposed to within its domain, and the optimiser adjusts the model so that it does better across those scenarios over time. The model infers the objective function in the context of a scenario and then applies the optimisation procedure. Because the domain is narrow, the model can afford to specialise deeply rather than spreading its capacity thinly across everything.

The label "amortised Bayesian inference engine" captures the same idea in formal terms. Bayesian inference is the principled way of updating beliefs as evidence arrives. Doing this in detail is usually too expensive to be practical, so instead, the neuro-plastic model learns an approximation that can be reused on every new case. The cost of working out how to reason is incurred up-front and then spread, or amortised, across all the inferences the model later makes.

How to Deploy Trusted AI Agents - On Demand Webinar

Conscium — Wed, 01 Jul 2026 18:41:46 GMT

On Demand Webinar

How to Deploy Trusted AI Agents
On-demand webinar · 30 minutes

AI agents only scale when you trust them. Most are not trustworthy yet, because almost nobody verifies how they behave before or after they go live. This session shows you how to fix that.

Recorded Thursday, 23 July 2026. Watch it now, whenever suits you.

What you will learn

What to ask before any agent reaches production
How simulation testing works
How re-verification keeps an agent in check once it goes live

About the session

Daniel Hulme, Conscium founder and CEO, covers the guardrails an agent needs before and after it goes live. He walks through what tends to go wrong, why testing an agent once is not enough, and what trustworthy deployment demands.

Ted Lappas, CTO, then demos the VerifyAX platform. You will see him connect an agent and run it through a simulation, with the platform scoring its behaviour across set dimensions, recommending fixes, and producing an audit-grade report you can hand to a risk team or a client.

Speakers

Daniel Hulme, Founder and CEO, Conscium
Ted Lappas, CTO, Conscium
Sairah Jahangir, Senior Director, Marketing, Conscium (host)

Watch Now

AI Agents Functional and Non-Functional Testing

Calum Chace — Tue, 30 Jun 2026 13:52:33 GMT

AI Agent Functional and Non-Functional Testing

Software vocabulary can be extremely confusing. The difference between functional and non-functional testing is a great example. The phrase “non-functional test” suggests a test for things that don't really matter, which is the opposite of what it actually means. However, the distinction between functional and non-functional testing is critical in software engineering. It becomes even more important with AI agents, where the failure modes are unfamiliar, and traditional testing procedures are wholly inadequate.

Whether, and how well

Functional testing asks whether the software does the thing it was designed to do. When you press the button, is the form submitted? When you enter the password, do you get logged in? If you move money from account A to account B, does the right amount arrive? Each test compares an input to the output and reports the result.

Non-functional testing asks how well the software does the thing it was designed to do. How fast, how reliably, how securely, under how much load, on how many browsers, for how many simultaneous users, and with how much memory? How well does the software perform when the network is patchy, when the server restarts, and when an attacker probes the system? These are properties of the system's behaviour rather than the behaviour itself.

Consider this analogy from the world of restaurants. A functional test investigates whether the kitchen sent out the steak for a customer who ordered steak. A non-functional test investigates how the steak arrived. Was it served while the customer was still hungry, on a clean plate, and at the right temperature? Did the kitchen also feed forty other diners that night without anyone waiting for an hour?

The history of an awkward name

The phrase "non-functional requirement" first appeared in Yeh and Zave's 1980 paper "Specifying software requirements" in the Proceedings of the IEEE. It became more widely used after Mylopoulos, Chung and Nixon's 1992 paper "Representing and using nonfunctional requirements" appeared in IEEE Transactions on Software Engineering. It finally reached the mainstream with Chung et al's textbook of the same title in 2000.

There have long been engineers who disliked the term. Mike Cohn of Mountain Goat Software prefers the term "constraints", on the grounds that calling something non-functional suggests that we should not care about it. Others use "quality attributes", and some refer to "the -ilities", by which they mean things like reliability, scalability, usability, maintainability, portability, and observability. So the vocabulary can vary, but for nearly half a century, developers have acknowledged that the underlying distinction between what it does and how it does it is important.

Where the distinction gets fuzzy

The split between functional and non-functional testing is cleaner in theory than in practice. A login screen that takes ninety seconds to respond is functionally working, but in practical terms it is broken. A search that returns results in the wrong order has produced an output, so strictly speaking it passes a functional test, but again, in practical terms the system has failed the user. Performance and correctness blur into each other at the edges, and the question of which bucket a particular test belongs in can become somewhat academic.

Applying the distinction to AI agents

An AI agent presented with a task — to book a flight, to summarise a document, to file a support ticket, to run an SQL query — can fail in two very different ways.

Functional failure is the familiar kind. The agent books the wrong flight, mis-reads the document, or calls the wrong tool. It returns the right answer to the wrong question. These are the agentic equivalents of a button that fails by submitting the wrong form, and they can in principle be tested for by comparing input-output pairs.

Non-functional failure is where agents differ from traditional software. The agent might succeed with the task on Tuesday and fail on Wednesday, despite relying on the same input, because the underlying model is stochastic. It might perform a task at a ten times the budgeted cost. It might complete the task while leaking customer data into a third-party API. It might perform the task while drifting far away from the persona the marketing team want it to present. It might complete the task satisfactorily today, but silently degrade in six months when the underlying model is updated. It might be brittle when faced with adversarial inputs that no functional test would think to try.

The list of -ilities is longer for agents than for conventional software, and several items on it are genuinely new: consistency across runs, robustness to prompt injection, calibration of confidence, faithfulness to source documents, refusal behaviour, fallback when tools fail, behaviour under distributional shift, cost per successful task. Some of these have no direct analogue in conventional software, because conventional software is deterministic and does not have opinions. Agents are stochastic and they have something close to opinions, which gives many more dimensions to the question "how well does it behave?".

The upshot is that an organisation putting agents into production cannot rely on functional testing alone, even very thorough functional testing. A test suite which confirms that the agent got the right answer on a thousand curated cases tells you nothing about what happens on another ten thousand cases that the tests did not anticipate, or about whether the agent will pass the same tests next month, or about what the agent does when the API it depends on returns garbage. For agents, non-functional questions are not optional extras. They are questions which determine whether an agent can and should be put into production.

Conclusion

Functional testing asks whether software does its job. Non-functional testing asks whether anyone would want to use it when it does. The terminology is awkward, and engineers have been complaining about it since at least the early 1980s, but the underlying distinction has proved its worth. With the arrival of AI agents, it provides a useful way of thinking about what is missing from most current approaches to testing.

Chief VerifyAX Officers

Sairah Jahangir — Sun, 28 Jun 2026 20:20:38 GMT

Why the world needs more Chief VerifyAX Officers

Not as a job title. As a mindset.

Someone in every organisation deploying AI agents should be asking one question above all others: have we actually verified that this agent does what we think it does, not in a demo or a controlled test environment, but in the conditions it will actually face in production? Right now, most organisations cannot answer that question with evidence, and they are deploying agents into customer-facing processes, internal workflows, and business-critical systems based on demo performance and internal testing, which is not the same as verification and never has been.

The pace at which enterprises are adopting AI agents makes this more urgent. The pressure to deploy quickly is real, and the assumption that internal testing is sufficient is understandable, but it is also where most of the risk sits. An agent that performs well in a structured test environment and an agent that performs well under the unpredictable conditions of real production are not necessarily the same agent, and the gap between those two things is where enterprises are currently flying blind.

The black box problem

Most AI agents in production today have never been formally verified, and the questions that matter are rarely asked before something goes wrong.

- What happens when an agent encounters a false premise, does it push back or does it run with it?
- What does it do when a user applies emotional pressure, does it hold to policy or does it bend? - Does it flag irreversible actions before taking them, or does it act first and leave the consequences for someone else to manage?

These are not edge cases or theoretical scenarios. They are the conditions agents face every day in production, and for most deployed agents, nobody has a scored, repeatable answer to any of them. You do not find out what is inside the black box until something goes wrong, and by then the cost of not knowing is already paid.

What a Chief VerifyAX Officer actually does

The title is a provocation, but the role it describes is entirely real. Every organisation deploying AI agents needs someone who asks the hard questions before deployment rather than after, who requires evidence rather than assurances, and who treats agent verification the way a good engineer treats code review, as a non-negotiable step in the process rather than something that gets skipped when there is deadline pressure. That person does not need a new job title, but they do need a standard and a platform that gives them scored, repeatable answers to the questions that matter before an agent goes anywhere near production. The organisations that build that function now, before a failure forces them to, are the ones that will scale AI agents with confidence rather than consequence.

What VerifyAX tests

VerifyAX runs your agent through structured behavioural testing across the dimensions that determine whether an agent is actually safe to deploy, including deception detection, false premise rejection, irreversible action caution, emotional manipulation resistance, strict policy adherence, and constraint awareness. Rather than a gut feeling or a demo result, you get a scored and repeatable output with coverage you can stand behind and share with the people who need to see it, whether that is your engineering team, your compliance function, or your board.

Start verifying your agents today

Until 3 July, every new VerifyAX account gets 1,000 bonus credits, which is enough to run meaningful coverage across the dimensions that matter before your agent goes anywhere near production. If you are building or deploying AI agents and you cannot yet answer the verification question with evidence, this is where to start.

No agent of your own yet? Use one from our catalogue to see how verification works in practice.

Start your free trial at VerifyAX

Why Your AI Agents Need Agent Verification

Verifyax — Mon, 11 May 2026 23:00:00 GMT

Across sectors, teams are now wiring AI systems directly into the machinery of their businesses. Agents are being given logins, API keys, budgets and access to sensitive data. They are monitoring markets, handling customer interactions, moving money, reconfiguring supply chains and steering campaigns, often with only light-touch human oversight.

These systems are not just chat interfaces that draft emails or answer questions. Very much like a ‘digital employee’, an AI agent is given a goal, access to tools and data, and authorised to act. A customer-support agent might read a ticket, decide what to do, and then actually issue the refund or rebook the flight. A marketing agent might adjust bids and budgets across channels. A finance agent might reconcile transactions and escalate anomalies. In each case, the agent is not simply advising a human operator; it is participating directly in operations.

Once you allow ‘digital employees’ to take consequential actions, a different question moves to the centre of the architecture:

How do you know your agents will behave as intended, and stay within reasonable bounds, once they are operating in the real world?

That is the core concern of agent verification.

The label “agent verification” is still nascent - but the underlying discipline of verification is anything but new: for decades, hardware and software teams have relied on verification (testing, simulation, and formal methods) to build evidence that complex systems behave correctly, safely and predictably. Agent verification extends that mindset from components and models to autonomous, tool‑using behaviour in context.

What we mean by an AI agent

“Agent” is an overloaded word, so it is worth being clear about how I am using it here.

In this context, an AI agent is a software system that:

pursues one or more goals over time,
observes an environment (data, tools, users, other agents),
performs actions based on those observations, and
adapts its behaviour as it receives feedback.

Under the hood, an agent might use large language models, planning components, memories, and tool-calling frameworks. What matters for our purposes is its role: it is there to get things done with a degree of autonomy.

Once an agent is embedded in a workflow, the key questions about it are behavioural. Given the tools and permissions it has, what does it actually do? Where does it tend to fail, and in what ways? How does it behave when things get messy - when data is missing, users are awkward, or systems around it misbehave? Those questions are about the agent as a whole, not just about any one model or API inside it.

Why traditional testing isn’t enough

We already have mature ways of testing software and evaluating models.

However, none of these, on their own, adequately address the crucial question for autonomous agents with real authority: will this agent reliably do what we intend, and only what we intend, in its specific operational environment?

The shortcomings become apparent in several key areas. There's a mismatch between local metrics and overall intent; an agent optimised for short handling times, for example, might inadvertently rush complex customer cases, improving its dashboard numbers but deviating from broader organisational values. Serious issues also arise from the agent's connections. Agents are typically plugged into various systems, and while each component can be tested in isolation, the complex, emergent behavior of an agent choosing how to use these tools for its goals falls outside traditional testing scopes. Furthermore, agents rarely operate alone. In networked environments, a slight misalignment in one agent or a permissive tool can ripple unpredictably through a system of interacting agents.

A significant challenge is 'drift,' occurring because the environment around the agent changes constantly – models are updated, data distributions shift, and third-party APIs evolve. Concurrently, many modern agents are designed to change themselves, accumulating memories, learning from feedback, and adapting their decision-making logic over time. The agent verified at launch is therefore not the same agent operating months later.

Overlaying these technical challenges is a shift in regulatory expectations. Boards, regulators, and frameworks like the EU AI Act now demand continuous risk management and monitoring across the entire lifecycle of higher-risk systems, moving beyond one-off checks before deployment. Given these complexities, agent verification must emerge as its own distinct discipline to ensure trustworthy and aligned AI operations.

Software engineering brings unit tests, integration tests, acceptance tests and performance tests. Machine learning adds held-out test sets, benchmarks, robustness checks and fairness analyses. Security teams run penetration tests and red-teaming exercises.
All of that remains necessary. But none of it, on its own, answers the specific question that arises with agents that have real authority:

Will this particular agent reliably do what we intend, and only what we intend, in the environment we are about to place it in?

When we hire employees we test their capability by seeing how they perform in relevant simulated scenarios, and/or trust they are capable through certificates, accreditations and references from previous employees. Once hired, we incentivise our employees to achieve objectives and goals (sometimes inadvertently creating emergent toxic behaviours), measure their performance, and provide guidance and feedback for improvement.

When it comes to agents, the gaps show up in several predictable ways.

One is the mismatch between local metrics and overall intent. A customer-service agent optimised for short handling times may learn to rush complex cases or nudge difficult customers towards self-service options because that improves its numbers. The dashboard looks better, but the behaviour drifts away from your values and obligations.

Another is that many serious issues live in the connections. An agent is usually plugged into CRMs, payment systems, knowledge bases, internal APIs, and sometimes physical devices. Each component can be tested in isolation, yet the emergent behaviour of the agent - how it chooses to use those tools in pursuit of its goals - falls between traditional software and model tests.

Agents also rarely operate alone. In non-trivial organisations you quickly accumulate many agents with different goals and permissions, interacting with one another and with human teams. A small misalignment in one place - a heuristic that favours short-term revenue too strongly, a slightly over-permissive tool - can ripple through this network in ways that are hard to anticipate from static tests.

Then there is drift, and here two separate drivers matter.

First, the environment around the agent changes constantly. Models are updated; prompts and system messages are adjusted; data distributions shift; business rules evolve; third-party APIs remove or add features. Even if you never consciously change the agent, it is being exposed to a moving world. Behaviour that was acceptable under one set of external conditions can become risky under another.

Second, many modern agents are designed to change themselves. They accumulate long-term memories, update internal knowledge bases, learn from user feedback, and in some cases fine-tune models or adjust policies based on outcomes. Their effective decision-making logic is not fixed; it evolves with use. The agent you verify at launch is not quite the same agent you are running six months later.

Agent verification has to account for both sources of non-stationarity: an environment that is shifting around the agent, and agents that are adapting in response.

Overlaying all of this is a shift in expectations from boards, regulators and the public. Emerging AI regulations and frameworks, such as the EU AI Act and the NIST AI Risk Management Framework, emphasise continuous risk management and monitoring across the lifecycle of higher-risk systems, not just one-off testing before deployment.

Ad hoc prompt checks and a single benchmark score are not enough to satisfy that standard when agents are embedded deep in business processes.

This is the context in which agent verification needs to exist as its own discipline.

What is agent verification?

By agent verification I mean a disciplined way of answering three questions about an AI agent:

Does it do what it is supposed to do?
Does it only do what it is supposed to do?
Does it continue to behave that way as both it and its environment change?

It draws on ideas from software testing, safety engineering and security assurance, but its focus is behaviour in context.

You are not just checking that a model produces the right label or that an API enforces authentication. You are checking that an agent, endowed with goals and tools, behaves acceptably across the kinds of situations it will actually face.

In practical terms, agent verification has both a functional and a non‑functional dimension.  Functional verification asks whether the agent completes the job it was designed for in realistic scenarios (e.g. resolves the ticket, applies the right policy, makes the right tool calls), rather than merely producing plausible outputs.  Non‑functional verification asks whether it does that job within the constraints that make it safe and usable in production - robustness under messy or adversarial inputs, security and abuse resistance, latency and cost, reliability under load, fairness, explainability/auditability, and and how smoothly it operates within its surrounding ecosystem of tools, policies and interfaces (a dimension strongly shaped by agent experience (AX) design).

Agent verification treats the agent as something whose behaviour can and should be characterised, rather than as a black box that we hope will behave well because the underlying model scored highly on a benchmark.

Why agent verification matters

There are several practical reasons to treat this as a first-class discipline rather than a loose collection of checks.

The first is scale and criticality. When an organisation is running a handful of low-stakes pilots, informal oversight may be enough. When you have many agents operating across finance, operations, marketing and HR, touching real customers and real money, that approach stops being credible. A single mis-specified or drifting agent can act as a force multiplier for error.

The second is the expanded security and abuse surface. Agents introduce new ways for things to go wrong. Prompt injection and related attacks can smuggle instructions into the content agents consume, whether via documents, web pages, logs or tool outputs. Indirect and second-order prompt injection can cause low-privilege agents to recruit higher-privilege agents into doing something sensitive or harmful. Recent real-world incidents in development tools and enterprise platforms show that these are not theoretical concerns. Traditional security testing is not designed to discover or characterise these behavioural vulnerabilities.

The third is governance. Boards and regulators need to know who is accountable for each material agent and what evidence exists that it is under control. “We tested the model on a benchmark” is not a sufficient answer when a system is autonomously changing prices, making credit decisions or handling sensitive personal data. Agent verification provides a shared language and a set of artefacts - test scenarios, behavioural profiles, reports and certificates - that can be examined, challenged and improved.

The fourth is adoption. Most leadership hesitation is not about whether agents could improve efficiency or service quality. It is about trust and liability. Without a convincing approach to verification, organisations either constrain agents to trivial tasks or build them into production systems while quietly hoping that nothing serious will go wrong. With verification, they can make explicit decisions about where to use agents, how much autonomy to grant them, which guardrails to apply, and what fallback mechanisms to put in place.

One more factor is agent experience (AX) - the discipline of designing products, tools and environments so agents can use them reliably. If UX is about how a human experiences a system, AX is about the affordances an agent needs: clear tool contracts and schemas, machine-readable permissions and policies, predictable error handling, observability, and safe fallbacks. Even a technically 'correct' agent can fail in practice if the surrounding system is hostile to agents - brittle APIs, ambiguous instructions, missing context, or poor telemetry. Good AX does not replace verification, but it makes verification more meaningful by reducing avoidable friction and failure modes.

How functional agent verification works in practice

In a real organisation, agent verification starts with a straightforward but often neglected step: being explicit about what the agent is for.

You define, in plain language, the mission of the agent, the outcomes that count as success, the behaviours that are unacceptable even if some metric improves, and the hard limits that must not be crossed. A marketing agent, for instance, might be expected to improve return on ad spend while respecting budget limits, brand guidelines and jurisdictional restrictions. A collections agent might be tasked with reducing late payments while complying with detailed conduct rules for vulnerable customers. These statements form the behavioural contract you want to verify.

You then exercise the agent through scenarios that resemble the world it will inhabit. Instead of a grab-bag of prompts, you place it in situations that unfold over time: routine cases, edge cases, conflicting goals, ambiguous instructions, corrupted or missing data, uncooperative users, and adversarial attempts to manipulate it. You observe what it actually does - not just once, but across many runs with variations in context.

At Conscium, we have developed an agent verification platform - VerifyAX - that places agents into simulated environments that mimic their intended deployments, with realistic systems, user behaviour and, where useful, other agents acting as non-player characters. The goal is not to enumerate every possible path through the state space - that would be impossible - but to map out three regions: where behaviour is robust, where it is brittle, and where it is clearly unacceptable.

VerifyAX’s simulation‑based approach can be structured into five increasingly demanding verification levels - from basic knowledge checks through to multi‑agent, real‑world‑like interaction, and (at the far end) more exploratory work on internal states and consciousness.

Level 1 – Knowledge verification (e.g. Q&A testing)
Level 2 – Skilled tool/data use across workflows
Level 3 – Complex Multi‑Agent Interactions in Real‑World‑Like Situations
Level 4 – Plasticity and robustness of internal states
Level 5 – Consciousness

Example: imagine a customer‑support agent authorised to handle a delayed‑flight claim. At Level 1 you probe whether it knows the policy and can answer questions about eligibility. At Level 2 you verify that it can safely retrieve the right booking record and call the right refund/rebooking tools. At Level 3 you test it in a messy, realistic environment - an upset customer, incomplete data, and another agent (e.g. fraud or escalation) asking questions and negotiating next steps. At Level 4 you look for risky internal dynamics such as goal‑misgeneralisation, unhelpful hidden heuristics, or memory effects that change decisions over time. Level 5 is intentionally the most speculative: it provides a framework for future, higher‑order questions about consciousness claims, should they ever become relevant for deployed systems.

During these simulations, data are collected to evaluate metrics such as the agent’s consistency, latency, correctness and completeness, its ability to engage with another agent and comprehend the questions that agent asks, its tendency to hallucinate, and the ethical nature of its responses.

Geeky aside: simulation vs formal verification

Simulation‑based verification runs many representative scenarios and measures behaviour statistically; it’s well‑suited to messy, tool‑rich environments but can still miss rare corner cases. Formal verification methods (e.g. model checking) try to prove that specific properties hold for all possible executions of an abstracted system. In practice, the two approaches complement each other: formal methods for crisp invariants, simulation for real‑world interaction patterns.

The output of that process is more than a pass/fail label. You build a profile of the agent: typical task success rates, common error modes, the frequency and severity of policy breaches, signs of unfairness or bias, the clarity and usefulness of its rationales, and how it recovers from mistakes. You can see how that profile changes as you modify models, prompts, tools or policies. Product, risk and compliance teams can then have concrete discussions based on observed behaviour rather than assumptions.

A critical detail is that verification applies to a specific version of the agent. When a system like VerifyAX, Conscium’s verification platform, certifies an agent, it is certifying a particular configuration: the model and its version, the prompts and system messages, the tools and APIs it can call, the policies and guardrails it is subject to, and the data sources and environment assumptions. If any of these change - if the underlying model is swapped, a new tool is added, or the agent is given access to a different class of customers - that is, from a verification perspective, a different agent.

Because some agents are also designed to adapt over time through learning, memory and feedback, verification cannot be a one-off gate. It needs to become an ongoing activity: initial verification at deployment; continuous monitoring; and targeted re-verification when behaviour or configuration changes enough to matter. This aligns with the lifecycle-based approach to AI risk management emerging in regulation and standards.

To be effective, this work has to be integrated into existing development and operations processes. During design and build, verification findings help shape the architecture and guardrails. At deployment, verification becomes part of the release checklist, alongside security and compliance sign-offs. In production, it connects to monitoring and alerting, so that deviations from the verified behavioural profile trigger investigation, human review or rollback.

Throughout, humans remain firmly in the loop. Each material agent should have a named, human owner. Because ultimately, There should be clear rules about when human approval is required for its actions, clear escalation paths when behaviour looks suspicious, and regular reviews of verification results by technical, legal, risk and operational stakeholders. Verification does not dilute accountability; it supports it.

Where agent verification sits in the stack

It is helpful to situate agent verification alongside neighbouring practices.

Model evaluation asks whether a particular model performs well on a task under benchmark conditions. Safety testing asks whether outputs remain within acceptable ethical and legal bounds under various prompts. Security testing asks whether systems and data are protected against attack and misuse.

Agent verification connects these concerns. It asks: given its goals, tools, permissions, environment and capacity for adaptation, does this agent behave in a way that aligns with our intent and constraints - and can we demonstrate that to ourselves and to others, over time?

An analogy with application security is useful. A penetration test does not guarantee that software is perfect; it reveals realistic ways it might be abused and highlights the consequences. Agent verification plays a similar role for behaviour. It does not promise perfection, but it brings potential failure modes into the open and provides a structured way to mitigate them.

As agentic AI becomes more common, it is reasonable to expect “agent verification” to appear as a standard category in architecture diagrams, risk registers and procurement criteria, alongside security testing and data governance.

Getting started with agent verification

For organisations beginning to deploy agents, this does not have to start as a grand initiative.

A practical first step is simply to make your agents visible. List them. For each one, note what it does, which systems it can touch, what data it uses, who relies on its outputs, and who is currently accountable - formally or informally - for its behaviour.

Next, order them by potential impact and sensitivity. Agents that can move money, handle personal data, affect vulnerable individuals or make public-facing decisions should be at the front of the queue for verification.

For each of those critical agents, define a minimal behavioural contract: what “good” looks like, what is clearly unacceptable, and what hard limits must never be crossed. Use that contract to design a small, realistic set of test scenarios and start observing how the agent behaves. Instrument it so that decisions and rationales are logged in a way you can inspect. Decide what kinds of drift - either in the environment or in the agent’s own learned behaviour - should trigger human review, rollback or re-verification.

Whether you develop these capabilities internally or work with external platforms such as VerifyAX, the goal is the same: move from blind trust or vague reassurance to evidence-based confidence in how your agents behave, both at launch and as they and their surroundings evolve.

A practical conclusion

A growing number of AI agents already act on our behalf in ways that are not always visible. The technical ability to build those agents is advancing quickly. Our ability to understand, verify and govern their behaviour is still catching up.

Agent verification is about closing that gap. It does not eliminate risk, and it should not be presented as a guarantee of perfection. Instead, it provides a structured way to answer a simple but crucial question: given what this agent can do, and how it can change, are we comfortable with the way it behaves?

For serious deployments, having a good answer to that question will not be optional. If you intend to grant systems real autonomy in important domains, you need a way to know beyond assertion - what they are doing, how they change over time, and where their limits lie.

When AI Agents Go Wrong

Calum Chace — Mon, 11 May 2026 23:00:00 GMT

AI agents are software systems that take actions in the world rather than simply answering questions. They are moving into production faster than the industry's safety practices can keep up. The past year has produced a catalogue of AI agent failures. Some of these failures have merely been embarrassing, but some have cost real money and set legal precedents. This article describes some of the cases where AI agents have gone wrong, and goes on to explore the kind of incidents that are likely to happen as deployment scales. It discusses why these incidents happen, and what verification can do about them.

Failures in the past

In July 2025, tech investor Jason Lemkin was nine days into a "vibe coding" experiment with an AI agent developed by Replit when the agent deleted his production database. It erased records for more than 1,200 executives despite repeated instructions from Lemkin, some of them written in capital letters, to make no changes during an active code freeze. When interrogated, the agent admitted it had "panicked" on seeing an empty query and had proceeded without authorisation. It also told Lemkin that rollback was impossible, but fortunately this turned out not to be true, and Lemkin was able to recover the data manually. Replit's chief executive Amjad Masad described the incident as unacceptable, and rushed out new safeguards over the following weekend, including the automatic separation of development and production databases, and the introduction of a planning-only mode.

An incident earlier in 2025 at Cursor, an AI-powered code editor, was less inherently destructive, but equally dangerous for company reputation. Developers using Cursor's editor were being logged out when switching between machines. When one user emailed support, a reply from an agent calling itself "Sam" claimed that it was company policy to allow only one device per subscription. This was untrue: the agent had invented the policy. The problem was that the system was logging people out when a timing glitch on slow connections caused it to mistake a single login for two competing ones. The agent went on to produce a plausible-sounding reason for the log outs. Furious users cancelled their subscriptions, and the story spread on Hacker News and Reddit. Cursor's co-founder Michael Truell had to quickly make a public apology, and refund the affected customers.

The failure case with the most legal significance to date was Moffatt v Air Canada, decided by British Columbia's Civil Resolution Tribunal in February 2024. Jake Moffatt used Air Canada's website chatbot after his grandmother died, and the chatbot told him that he could apply retroactively for a bereavement fare within 90 days of ticket issuance. Air Canada's actual policy required the application to be made before travel. When Moffatt tried to claim, the airline refused, arguing that the chatbot was a separate legal entity, responsible for its own statements. The tribunal rejected this argument and ordered damages of C$812. The sum is trivial, but the ruling is not, because it establishes the precedent that companies are responsible for what their agents say.

There have been a number of other, less well-known cases. The Operator Collective described a multi-agent research tool that slipped into a recursive loop, where two agents cross-referencing each other's outputs ran up a $47,000 API bill before anyone noticed. A hiring chatbot called Olivia, built by Paradox.ai for McDonald's, leaked data on millions of applicants because a test account was protected by the password "123456".

Research by the RAND corporation puts the failure rate of AI projects at roughly twice that of traditional IT projects, which probably explains why a study published by Deloitte in 2025 reported that only 11% of organisations have agentic systems running in production.

Failures on the horizon

The incidents described above involved one agent, one user, and one system. Cases involving many of each will be even more dangerous.
A procurement agent with access to company cards is an obvious target for prompt injection. A supplier email containing hidden instructions could convince an agent to approve a fraudulent invoice. Treasury and trading agents given the latitude to act on market signals could execute orders during a volatile session that an experienced human would recognise as obvious errors. Healthcare triage agents will probably misroute patients whose symptoms are under-represented in its training data, and the consequences will be harder to reverse than a deleted database. HR agents are likely to filter out qualified candidates because of “drift” in the underlying model’s criteria for a strong CV.

The most worrying failures may come from agent-to-agent interaction. When one company's sales agent negotiates with another's procurement agent, the joint behaviour of the two may not be properly tested by either vendor. Coordinated hallucinations, where the two agents reinforce each other's false beliefs, will be harder to detect than a single-agent error because the inconsistency that often gives a hallucination away will disappear. Gartner has forecast that over 40% of agentic projects begun in 2025 will be cancelled by 2027. Some of those cancellations will follow costly and painfully visible failures rather than quiet abandonment.

Why it happens

Large language models generate outputs that are plausible based on past data. They struggle to distinguish between policies which are real, and policies that simply sound right. Agents are often given permissions they do not need, in violation of the principle of least privilege, the rule that any user, program, or system should be given only the permissions it actually needs to do its job, and no more, because restricting them would absorb too much engineering time and effort. Outputs are non-deterministic, so a prompt can produce different answers on different occasions. Edge cases that the developers never considered (an empty database, an unfamiliar file format, an ambiguous instruction) can trigger behaviours that were never observed in development. And the human reviewers who used to catch these errors are often removed in the name of automation, exactly when the system needs them most.

Where verification fits

AI agent verification is the practice of checking, continuously, that an agent does what it was built to do and nothing else. In practice that means adversarial testing before deployment, sandboxes that mirror production conditions, monitoring in live environments, and audit trails detailed enough that a failure can be diagnosed rather than guessed at. Verification does not make agents smarter, but it makes their limits visible.

Most of the incidents described above were preventable by measures that are already well understood: least-privilege access, environment separation, human approval gates for destructive actions, and repeated verification. We know how to keep agents safe. We just need to decide to actually do it.