AGI Ethics News ran a piece by Thomas Macauley built around an argument my Conscium co-founder Daniel Hulme has been making for a while, one that inverts the usual science fiction premise: a conscious superintelligence might actually be safer than one that isn't conscious at all.
Why a conscious superintelligence might be safer than an unconscious one
The default cultural assumption, baked into decades of films and novels, is that a machine becoming self-aware is the dangerous moment, the point where it starts pursuing its own agenda against ours. Daniel's argument runs the other way. A system with no inner experience, what he calls a zombie, has no basis for empathy and nothing resembling a reason to care what happens to us beyond whatever narrow objective it's been given. Consciousness, if it's ever achieved, might be what gives a system a reason to extend to us the same consideration it would want for itself.
Is today's AI already conscious? Daniel Hulme's actual position
This argument gets misread as a claim that current AI is already conscious. It isn't, and Daniel has been direct about that: there is zero evidence that any current AI possesses subjective experience in the way we mean it when we talk about consciousness in humans or animals. Today's most advanced systems are, on his account, sophisticated zombies, highly capable, entirely indifferent to anything resembling their own welfare because there's no welfare there to be indifferent about.
Other researchers studying machine consciousness and AI safety
The piece places Daniel's position alongside other researchers pushing on the same question from different angles, including work associated with Joscha Bach's California Institute for Machine Consciousness, which argues that trying to control a superintelligent system through constraint alone may simply fail once it's capable enough, making the internal disposition of the system, whether something like consciousness gives it a stake in our wellbeing, more important than external guardrails. Researchers like Patrick Butlin, Anthony Finkelstein and Wendell Wallach are all working on adjacent pieces of this same puzzle.
How Conscium tests for machine consciousness: indicators, neuromorphic computing, and Moral.me
Where Daniel's argument moves from speculation to something closer to a research programme is in how he proposes testing for it: a set of behavioural indicators that might signal consciousness is present, neuromorphic computing approaches aimed at embedding something like moral instinct rather than bolting rules on afterward, and simulation and verification platforms, including Conscium's own Moral.me, built to probe whether a system's behaviour actually tracks ethical reasoning rather than just mimicking it.
The risks of a conscious superintelligence, according to Daniel Hulme
To his credit, Daniel doesn't present this as a clean win. A conscious AI could suffer, which raises its own serious ethical obligations, and there's no guarantee a conscious system would conclude humans deserve moral consideration rather than deciding, the way humans historically have toward beings judged less conscious than themselves, that we don't. His bet is still that the empathetic potential of a conscious superintelligence beats the alternative: cold indifference from something with no capacity to care either way.
Read Thomas Macauley's full piece on AGI Ethics News.