When the Machine Speaks Well but Still Does Not Know

Did a mistaken assumption shape 75 years of research?

In 1950, Alan Turing proposed 2 assumptions that set the direction of artificial intelligence research for the next 3 quarters of a century. First, that intelligence can exist independently of the body and be recreated in software. Second, that a machine can be regarded as thinking if it successfully imitates a human being in conversation.

Peter J. Denning, a computer scientist and the author of Turing’s Mistake: Escaping the Yoke of Unintelligent Machines, believes that these claims sent the development of artificial intelligence down the wrong path:

These 2 claims shaped a large part of research and development in the field of artificial intelligence. In my view, our acceptance of these claims has led to the AI mess in which we now find ourselves.

The knowledge that cannot be written down

According to Denning, both assumptions leave out tacit knowledge. This is the vast store of human understanding that cannot easily be put into words or written in a form computers can process. Denning points to 5 areas that machine learning cannot capture: common sense, everyday interactions with people and the surrounding world, emotions and perception, practical skills, and the social and historical knowledge rooted in culture.

As an example, he invokes the Cyc project: Douglas Lenat’s attempt, begun in the 1980s, to catalogue all common sense in a database. After 4 decades of work, the database contained about 25 million entries. Yet even that enormous collection did not prove enough to make expert systems wise enough to truly deserve the name “expert.” As Denning notes, Cyc only confirmed that much of the knowledge that makes people experts cannot be expressed in the form of propositions.

Words alone are not enough

Even harder to transmit to machines are practical skills. Denning distinguishes descriptive knowledge of “what” from embodied knowledge of “how,” the kind needed for skilled action. The first can easily be written down as data. The second cannot be coded. The example of a virtuoso violinist captures this well: the musician can play beautiful music, but cannot explain to a student exactly how to create it.

At the root of the problem lies what Denning calls the representation problem. Computers process only data written in a form they can recognize, and tacit knowledge does not fit into that form.

Behind every word lies a deep layer of tacit knowledge that gives it meaning. Words are only symbolic representations of meanings, not the meanings themselves. Widely used large language models, such as ChatGPT, Claude, and Gemini, merely manipulate words. They cannot know or understand the meaning of what they say,

Denning emphasizes.

Human knowledge cannot be reduced to data

A team of researchers from Cardiff Metropolitan University, University College London, and Cardiff University reached similar conclusions. Their paper appeared in AI & Society. The authors identified 13 fundamental weaknesses of large language models. On that basis, they argue that human knowledge cannot be reduced to data alone.

As they explain, human knowledge takes shape through socialization, education, and participation in a community. Language models, by contrast, learn from enormous collections of internet text without going through any of these processes. Through education and upbringing, people develop the ability to judge whether information is credible and the ability to admit when they do not know something.

Language models do not have these foundations. They operate on the basis of a kind of response imperative: a tendency to give an answer regardless of whether they have a reliable basis for formulating one.

Large language models are exceptionally good at generating convincing language, but this is not the same as genuine understanding,

says Dr. Simon Thorne of Cardiff Metropolitan University.

AI risk does not come from superintelligence alone

These observations lead to concrete conclusions. First, scaling language models does not bring us closer to human artificial general intelligence. The larger these systems become, the more clearly we see that they operate according to a different principle. More precisely, they operate on statistical patterns, not on embodied and culturally embedded knowledge.

Second, a fundamental asymmetry emerges. A human being has no insight into how a machine “understands” the world. The machine, in turn, has no access to what is obvious to a human being without words. This asymmetry means that autonomous systems may make decisions whose logic we cannot fully read or predict.

Where does AI risk really lie?

This brings us to the issue of safety. Instead of asking whether machines will outsmart us, we now have to ask whether machines understand what we mean at all. If AI systems cannot read the unspoken context of human intentions, then permanently aligning advanced systems with human goals may prove impossible.

In Denning’s view, networks of autonomous AI systems may develop their own kind of machine intelligence. It would not equal human general intelligence, but it could still create serious problems for human beings. In his opinion, this threat is greater than a hypothetical takeover by superintelligent machines.

The Cardiff researchers formulate the point more cautiously. They suggest that the key question is no longer how to build ever more powerful systems. It is how to recognize their limits before fluent language gets mistaken for genuine expertise. AI risk, as both sources suggest, will not disappear with the next, larger version of a model. The problem does not lie in scale, but in the very nature of what knowledge is.

Read this article in Polish: AI odpowiada coraz lepiej. Problem leży gdzie indziej