Artificial Intelligence Sounds Sexier Than Mathematical Prediction

By Liam Lally | 17th October 26

A convincing answer can demonstrate capability. It cannot, by itself, demonstrate conscious experience and human emotion.

Artificial intelligence has a branding advantage.

“A mathematical model trained to predict what comes next” is a harder sell. It sounds less like the arrival of a digital colleague and more like something you would skip past in a software manual.

Yet that less glamorous description gets us closer to the mechanism behind the large language models, or LLMs, used in many AI chatbots. AI is a much broader field, but these conversational systems are where the confusion between a useful tool and a conscious mind becomes particularly tempting.

A chatbot responds in fluent English, refers to itself as “I” and tells you it understands. The leap from “that was an intelligent answer” to “something in there experienced understanding” can feel surprisingly small.

It is a much bigger leap than the conversation makes it appear.

What actually happens when an LLM answers

A typical autoregressive LLM generates text by predicting the next token. A token is a unit of text, which might be a word, part of a word or punctuation.

The model processes the context it has been given and calculates probabilities for possible next tokens. A token is selected, added to the sequence, and the process continues. The answer develops through repeated numerical computation, using learned parameters commonly called weights. [1]

Those weights are adjusted during training. Further training can use example answers and feedback to improve how a model follows instructions, responds to questions and handles requests. [2]

That is an extraordinary engineering achievement. But describing how an answer is generated does not tell us that the process is accompanied by awareness.

Nor does the phrase “next-token prediction” mean nothing sophisticated happens along the way. Research has identified examples of models planning ahead and carrying out intermediate reasoning steps while producing text. The prediction task can produce mechanisms far richer than simply matching one familiar phrase to another. [3]

Intelligence and consciousness are different claims

The word “intelligence” causes trouble because we use it to mean different things.

If we mean the ability to solve problems, use language or perform complex tasks, describing AI as intelligent can be perfectly reasonable. Those are capabilities we can test.

Consciousness, in the sense I mean here, concerns subjective experience. It is the felt experience of seeing a colour, being confused or suddenly grasping an idea. Demonstrating a capability and establishing that something experiences using it are separate questions. [4]

A model can produce a correct explanation without that explanation proving conscious comprehension. Giving it credit for the result does not require us to assume an inner life.

This also helps with arguments about whether AI “understands”. If one person means competent use of information and another means consciously experiencing its meaning, they may be arguing over a definition without realising it.

For this discussion, my concern is conscious understanding. A convincing performance does not establish that on its own.

Why “I understand” can be so misleading

Imagine telling a chatbot that a customer has complained about a delayed delivery. It replies: “I understand how frustrating that must be.”

The response may be appropriate. It may even help you write a better apology. But the sentence itself does not demonstrate that the system has experienced frustration, sympathy or concern.

We should be especially careful about treating a model’s statements about itself as decisive evidence. A claim such as “I feel worried” needs more support than the fact that the model produced those words. Equally, asking it to deny consciousness would not settle the scientific question.

Think about an automated message from accounting software warning that your cash flow is poor. We assess whether the warning is correct; we do not treat its wording as evidence that the software is worried about payroll.

Conversational fluency makes that separation harder to maintain. It does not remove the need to make it.

Changing an answer is not necessarily retraining

Tell a chatbot that your business sells to architects rather than homeowners, and it may rewrite its advice accordingly. That can look like learning, and in a practical sense it is adapting to new information.

However, ordinary inference, the process of running a trained model, usually leaves its weights unchanged. The model can use new information in the conversation without being retrained. This ability to adapt through context without weight updates is well established. [5]

So “it never learns” is too blunt. Training, adaptation within a conversation and an application storing information for later use are different processes. None should automatically be mistaken for a person developing understanding through lived experience.

The same distinction matters between requests. Stored model weights and a saved conversation are not, by themselves, an ongoing computation. If no inference or other processing is running, there is no continuing calculation taking place simply because the conversation remains on screen.

An application can schedule further processing or run background tasks. That is additional software activity, and it should be assessed on its own terms. It does not automatically demonstrate a private stream of conscious thought.

Knowing the maths does not settle every question

There is a tempting shortcut in this debate: we built the system, we know it runs on maths, therefore we have proved it cannot be conscious.

I would not use that argument.

Knowing the architecture and training process is different from fully understanding the mechanisms that training creates. Researchers can inspect and intervene in models, but explaining their learned behaviour remains an active scientific challenge. Anthropic’s work on tracing model computations explicitly describes those limits. [3]

More fundamentally, whether particular forms of computation could support subjective experience is part of the debate. Calling a system mathematical does not resolve that question. [4]

The stronger position is that fluent conversation, correct answers and humanlike wording do not, by themselves, establish consciousness. We can take those abilities seriously while requiring better evidence for claims about experience.

Why this matters when you use AI for work

Suppose you ask a chatbot why a Google Ads campaign has stopped converting. It offers a polished explanation and a confident recommendation to change the bidding strategy.

Before acting, you need to know what supports that recommendation. Has the system seen the account data? Has it checked conversion tracking, recent changes and the dates being compared? Or has it generated a plausible explanation from a brief description?

A useful hypothesis can help you investigate. It becomes a sound basis for action when the evidence supports it.

Whether a model is conscious would not make its recommendation correct anyway. People can consciously believe things that are wrong. The practical questions concern the information available, the quality of the reasoning and whether the result survives checking.

For business use, I would rather see a tool assessed on those grounds than trusted because it sounds like an experienced colleague.

What would count as evidence of machine consciousness

Machine consciousness is a legitimate research question. It deserves a clearer definition and stronger methods than asking a chatbot whether it feels alive.

Researchers have proposed assessing AI systems against indicators drawn from scientific theories of consciousness. These concern how information is processed and made available within a system, rather than simply how human its conversation sounds. Such indicators are evidence to evaluate under a theory, not a universally accepted consciousness detector. [6]

There are already experiments investigating whether models can report aspects of their internal states. Anthropic has reported limited, unreliable forms of this ability, while explicitly stating that its findings do not answer whether a model has subjective experience. [7]

Useful investigations would ask:

Can a system reliably monitor its internal processing rather than merely produce a plausible description?
Do controlled changes to its internal mechanisms affect that ability in predictable ways?
Can independent researchers reproduce the findings and rule out simpler explanations?
Why should the measured property count as evidence of experience, rather than another useful capability?

That last question matters. If we redefine “machine consciousness” to mean any form of self-monitoring, we have changed the label without necessarily answering the original question.

My position is to use the technology, recognise what it can do and keep the claims proportionate to the evidence. A useful answer does not need a conscious author to be useful. And an impressive answer is not a reason to stop asking how it was produced.

“Artificial intelligence” will probably always sound sexier than “mathematical prediction”. That is no reason to let the name do our thinking for us.

What would count as evidence of machine consciousness beyond a machine telling us it is conscious?

Sources and further reading

Categories +

Get in touch

If you have any questions or if your website needs optimising to reach your target audience, contact me on the below:

Blog