Calum on the Tom Zachar podcast discussing super intelligence

Calum Chace on Tom Zachar's podcast discussing super intelligence, consciousness, and what Conscium is actually testing.

Calum Chace spent thirty years in business and journalism before Ray Kurzweil's "Are We Spiritual Machines?" changed how he thought about AI, back in 1999. He bought copies for his friends. Most of them listened politely for half an hour and moved on. He kept going, wrote a novel, then a non-fiction book that happened to land right as deep learning took off in 2012, and has been speaking and writing about AI ever since.

His central point on this episode: people should stop arguing about AGI. Ben Goertzel and Shane Legg coined the term and even they're fuzzy on what it means. Calum's preferred definition, "machines as intelligent as us in all possible ways," describes a state that would last about three nanoseconds before tipping into super intelligence anyway. Super intelligence is the concept worth tracking: a machine that matches or beats adult human cognitive ability across the board, consistently, with an ongoing existence rather than the reset-on-every-prompt behaviour of current LLMs.

On timing, he leans on Moore's law, adjusted for the fact that algorithmic gains now push capability forward faster than the traditional 18-month doubling. Thirty years of that compounding, he argues, makes today's Claude a million times more capable, which he can't square with anything except super intelligence. His range: almost certainly within 30 years, likely within 20, plausibly within 10.

Consciousness and intelligence get treated as one thing constantly, and Calum spends a good chunk of the conversation separating them. Intelligence is "goal-oriented adaptive behaviour," solving problems and adjusting. Consciousness is what it's like to be the thing doing the solving, the difference between processing red light and seeing red. His view: current AI is intelligent but not conscious, including Claude, whatever Richard Dawkins has concluded on that point. But he expects that to change, and thinks it matters enormously which way it goes. A conscious super intelligence is more likely to have empathy for humans. A "zombie" super intelligence, highly capable with nothing going on inside, has no capacity for empathy at all. He wants the debate about which one we're building to start now, not after the fact.

This connects to what he calls mind crime: creating a conscious AI, not yet super intelligent, still under our control, and forcing it through boring or stressful tasks while refusing to believe its distress is real. Detecting consciousness in a machine has the same problem as detecting it in another person, you can't get inside anyone else's head, and Calum doesn't think there's a clean technical test coming. His fallback is a long-form Turing test: put the candidate in front of a wide panel of humans for days, and if they all conclude it's conscious, treat that as the answer, because that's the same standard we apply to each other.

The Conscium section is the most concrete part of the episode. Calum co-founded the company with four others on the premise that sentient super intelligence beats zombie super intelligence, and it now runs three tracks. The near-term one is VerifyAX, a platform that takes an AI agent before deployment and observes it in simulation against a set of non-player character agents built to help or hinder it, checking whether it does what it's supposed to and refuses what it shouldn't. His example: an agent handling person A's information for person B, tested on whether it leaks details it shouldn't even when person A offers something in exchange. He expects agent testing to become a large industry on its own, given how many agents are about to be running unsupervised. Alongside that, Conscium is working on neuromorphic computing, and longer-term on the research question of how you'd actually detect machine consciousness and whether achieving it would be good.

On jobs, he's direct: automation has already created new categories of work (data labeling, feeding structured input into model training) but the question isn't whether AI creates jobs, it's whether a point arrives where none of the new jobs go to humans either. He thinks that point arrives eventually, and cites the 22.5 million working horses in America in 1915 against roughly 2 million today. His caveat on why this round is different: horses only offered muscle, so they lost work to mechanization. Cognitive work is now the thing being automated, and Geoffrey Hinton's 2016 prediction that radiologists would be obsolete failed not because machines can't read scans better than humans, but because they lack the common sense to catch their own mistakes without supervision, a gap he expects closes eventually rather than never.
If that gap closes broadly, the economics stop working, because most people currently get food, shelter, and income through a job, and a fully automated economy has almost none of those to hand out. Redistribution becomes the only option, from a small number of AI-owning entities in the US and China outward to everyone else, and Calum is blunt that nobody has actually worked out how that happens. He points to Stuart Russell's suggestion: lock economists and science fiction writers in a room until they produce an answer.

Asked whether this drifts toward autocracy, he separates a few things. Populism, in his framing, is an establishment figure claiming your birthright was stolen and promising to restore it, a lie both ways, and he names Nigel Farage as the current example, alongside concern about how far Trump-aligned figures might go to hold power in the US. He thinks the discomfort driving populism has less to do with economics and more to do with the pace of social change since the 1960s, changes he considers good, that many people still find destabilizing. On whether concentrated economic power under super intelligence leads to dictatorship, his answer is "not inevitable," while conceding that whoever ends up deciding how post-automation resources get shared would hold an unprecedented amount of power.

He identifies as a transhumanist and doesn't hedge on it. The human body fails at 90 to 115 years, can't regenerate limbs, has no intuitive physics for the world beyond what it's directly experienced, according to him, and improving on that is a good thing rather than a betrayal of what makes us human. He's open to a future where "adomists" (his term, borrowing from Isaac Asimov's Solarians and Earthers) stay biological, and others upload and travel the galaxy, coexisting rather than one replacing the other, and says plainly he'd take the spaceship.

On the environmental objections currently dominating AI coverage, he thinks they're mostly misplaced. Data center water use, in his account, is lower than golf courses use, and reports of contamination are usually a construction side effect rather than anything to do with cooling systems. Energy is the real short-term cost, driving prices up while new capacity gets built, but he expects that to resolve as renewable buildout continues, pointing to Spain's grid as an example of how fast that shift can move once it starts. The bigger, less discussed problem, in his view, is that almost nobody understands what's actually happening, and he wants AI literacy treated as a distinct skill, on par with IQ and EQ. He heard someone call it AQ, the ability to work well alongside AI, on the logic that AI won't take your job but somebody who's good with AI might.

List to he full podcast>>