Two horizons: agents, intelligence, and machine consciousness
At Web Summit this month I shared a stage with a cybersecurity chief and an AI researcher. It was a lively session because we disagreed. Which is just as well, because panels where everybody agrees are very boring.
Two time horizons
When you think about AI, you have to divide the future into two time horizons. In the short term, humans will be essential: checking the work of agents, staying in the loop, providing oversight. I think we will have full employment for some time yet.
The second horizon is different. I am, to borrow the Silicon Valley phrase, AGI-pilled. I think we will see a singularity, and we will get superintelligence. At that point there will be no jobs for humans. But that is a conversation for another day.
Agents are not traditional software
Until then, we need humans to build good security systems. Traditional software does what it does: either it works or it doesn't, and if it doesn't, you find the bug, fix it and put it back to work. Agents are different. They evolve during deployment. They are ingenious, and they do things nobody expected.
I think the Hugging Face incident earlier this year will be talked about for years, because it was the moment the mainstream media woke up to how intelligent and capable agents are becoming.
And yet agents have not really been deployed at scale. It has been "the year of the agent" every year since 2024. During the discussions about DoorDash buying Instacart, it emerged that the number of food purchases made by agents is only in the hundreds. Most agents in deployment today are essentially knowledge-retrieval systems. That will probably change next year, and those agents will need to be verified and tested.
That is what my company, Conscium, does. Our platform puts agents into simulations and watches them closely as they do their job. We use our own non-player-character agents to test them, trip them up, and try to confuse them. At the end the deployer gets a report on how well their agents performed. You can't rely on evals that only tell you what happened after deployment.
We do know what intelligence is
One of my fellow panellists argued that we don't really know what intelligence is. I think we do. We may not be able to define it in a precise engineering way, but there is a serviceable four-word definition: goal-oriented adaptive behaviour. An entity that has a goal, learns what works and what doesn't, and changes how it operates in order to achieve that goal better is intelligent. It would be crazy to deny that today's large language models are intelligent.
Consciousness is another matter
Intelligence is about doing. Consciousness is about being: it is experience, what it is like to be you. We believe, and hope, that today's large language models are not conscious – because if they are, we are probably torturing and murdering them over and over again.
They may well become conscious in the future. That is one of the reasons Conscium was founded; the clue is in the name. We think that making AI conscious might be our best way to survive the singularity.
We are hopeless at defining consciousness, even though we know it from the inside. But whether machines should or should not become conscious could be one of the biggest questions we will ever face as a species.
It is a question we explore from many angles in our book, Perspectives on Machine Consciousness.
