by Dr Ted Lappas
This article first appeared in Unite.AI (here) on 9 July 2025
Blake Lemoine jumped the gun
Three years ago, Google fired the software engineer Blake Lemoine for insisting that a chatbot called LaMDA was conscious. This was before OpenAI’s ChatGPT was released to the public, and most people simply assumed that Google had good reason for their action.
It remains the consensus view among computer scientists and neuroscientists working in the field that ‘large language models’ (LLMs) like ChatGPT are very unlikely to be conscious. LLMs and brains operate in fundamentally different ways. What chatbots do is learn how to imitatethe output of human mental activity. Even though they do this very well and very fast, there is no enduring internal state of an LLM that could be conscious. In Thomas Nagel’s famous phrase, there is almost certainly ‘nothing it is like’ to be a chatbot.
However, in the three years since Lemoine was fired, billions of people have interacted with ChatGPT and its competitors, like Gemini, Claude and Mistral. A small but vocal minority have convinced themselves that chatbots are now conscious.
Machine consciousness soon?
More importantly, many computer scientists and neuroscientists are openly suggesting that genuinely conscious machines may be developed in the next few decades – or even sooner. In case this does happen, we need to be prepared. We must avoid “mind crime”, which is the term for causing suffering to disembodied conscious entities. We must also try to ensure that these new entities present no threat to humans.
Since LLMs are unlikely to become conscious in their most basic form, a new paradigm may be needed to understand how AI consciousness could emerge. One of the leading research teams in this field is run by neuroscientist Mark Solms and physicist Jonathan Shock, of the University of Cape Town. They are scientific advisers to Conscium, and their research is partly funded by it.
Giving a damn
Solms and Shock argue that an essential ingredient for machine consciousness is that AIs must ‘give a damn’ about their choices and actions. One important aspect of this is to make their survival contingent upon their performance, just as the survival of living organisms is. This does not mean that they must have an abstract comprehension of their own mortality: most animals monitor and motivate their survival chances without thinking about it. The key is “affective consciousness”, or raw feelings. It feels good to be fed and safe, and it feels bad not to be. These feelings drive our behaviour. The survival imperative is nature’s fundamental driving force, and feeling is the yardstick by which we sentient creatures measure how we are doing.
We are entering the age of AI agents—digital entities whose ‘survival’ depends on their ability to perform tasks effectively. We are creating evolutionary environments in silicon, and we must be careful what we beget.
Free Energy Principle
To understand affective consciousness, the Conscium team is placing AI agents in simulated environments in which they must satisfy their own needs, such as maintaining their virtual temperature and power supply. To this end, they create and optimise internal models of themselves – their needs, abilities, and limitations – in relation to their environment.
These agents, simplified as they may be, monitor how well or badly they are doing by calculating a quantity known as ‘free energy’. The Free Energy Principle was developed by neuroscientist Karl Friston, another member of Conscium’s scientific advisory board. Free Energy is a measure of how far a system has diverged from its optimal state of being. It is an error signal, and it must be minimised. The agent constantly calculates “what will happen to my survival chances if I do this or that?”, and adapts its behaviour accordingly.
Confidence and Expected Free Energy
The degree of confidence that an agent has in its answer to this question is quantified in a measure called “Expected Free Energy” (EFE). The agent selects the answer with the lowest EFE – the answer that it is most confident in. Being confident is good for an agent that is trying to survive – unless it is deluded and its understanding of itself within its world is flawed. Importantly, this confidence value is wholly subjective: it is based only on the agent’s own needs, and its current and projected status. It matters to the agent only, and to no-one else.
Like biological organisms, the AI agent has a number of conflicting needs, which it must prioritise. Crucially, each of the agent’s multiple competing needs is treated as an independent ‘categorical’ variable, which means it is distinguished qualitatively. 80% of battery power is not equivalent in value to 80% of viable temperature. To balance them, the agent must project itself forward in time and forecast the consequences of any action it is considering taking.
Qualia
Because the agent’s subjective needs are qualitatively distinctive, they are analagous to ‘qualia’ in the philosophical and scientific jargon. These are generally considered to be the ground zero property of consciousness, so it is plausible that they might in future be experienced by an agent.
The agents developed by the Conscium team use sophisticated learning algorithms to operate in increasingly complex and uncertain environments. While the same can be said for many other agent architectures, these new agents are distinguished by a survival-driven imperative to look after their own Free Energy by continuously balancing their competing needs in a way that is meaningful to them and them alone. The agents are thus guided by an internal landscape of affective signals—akin to what we refer to as feelings—that reflect how well their needs are being met through every choice they make. As the sophistication and complexity of their needs and environments grow, so too will the richness of their internal states.
Evidence for feelings
The next step in the Conscium research programme is to develop a series of functional and behavioural tests which will provide evidence about whether or not an agent really does experience feelings when it identifies its most salient needs and makes its choices accordingly. We all know that consciousness cannot be observed externally or demonstrated objectively: each of us can only observe our own subjective states. However, the researchers believe it should be possible to agree upon some specific hypotheses which, if verified by experiment, would provide weighty evidence that the agent possesses feelings. Although there will always be room for doubt – given the subjective nature of consciousness – a rigorous testing process will aim at making this doubt increasingly unreasonable.
These experiments will seek to eliminate alternative hypotheses, making it less reasonable to attribute the agent’s internal states to mechanisms that do not require feelings. It will use adversarial tests to distinguish between sentient and non-sentient behaviour, and it will include control experiments with agents which do not possess the algorithmic aspects believed to be necessary for feeling, and blinded evaluation to reduce observer bias. Physical inspection of the agents’ internal states may reveal quantifiable indicators of affective processing. Findings must be independently reproducible, and the entire process will be subject to rigorous ethical oversight, especially in cases where the agent might conceivably experience true feelings.
This is how science works: by the experimental testing of falsifiable predictions. “We must be careful not to set a higher bar for the science of consciousness than for any other science; otherwise we run the risk of placing consciousness outside of science”, says Solms.
What if this succeeds?
If consciousness is found to arise in relatively simple AI agents in this way, a survival-driven agent could conceivably be conjoined with other architectures, including large language models, and lend them consciousness too. This is not something that we can afford to accidentally conjure into being, so understanding how to do it and also how to avoid it is vitally important.
If conscious agents – agents that give a damn – are developed in the near future, we will have to take their feelings – and their rights – into account. This is a major concern for Conscium: artificial consciousness must not arise by accident. The motivation of this research programme is not the creation of conscious AI, but to understand how it might come about, and the potential risks. On its website, Conscium has published an academic paper and an open letter setting out five principles to guide any organisation engaged in research that could lead to the creation of conscious machines.
Humans sometimes behave very badly towards other sentient beings, including other humans. If we walk blindly into this future, we risk doing the same to artificially conscious beings, with consequences that we may not even be able to comprehend.
