Separating Fact from Fiction in AI Technology
It would seem that the perception of what current parlance calls ‘AI’ could stand some improvement.
We often find indirect information about how that mythical being, the Average Computer User (henceforth ACU), sees the currently fashionable chatbots. The following assumptions hold:
- 1) that it has access to all information and computing power we got used to in relation to computers, and
- 2) that it can invent solutions, or invent methods to get solutions, or it can invent methods to invent methods to get solutions etc., in other words, the whole stack of self-reflection layers within human intelligence.
None of this is entirely false, but if we had to round it to the nearest boolean value, that value would be ‘false’.
To explain both the wrong perception and its correction, we need to make a distinction between LLMs and their applications like chatbots, RAG systems etc. An LLM is a huge neural net capable of completing text based on the large amount of other text it has seen during training. What an LLM does can be described via many different terms, one of which would be ‘imitation’. An LLM imitates the texts it has seen during training, but this imitation is a deep one; an LLM does not simply cut and paste text portions from its training set into a reply (although sometimes it does). Instead, it seems to have identified patterns at various levels of abstraction and applies them to constructing replies that fit the current context. A good example would be the logic formula called syllogism (‘humans are mortal, Socrates is human, therefore Socrates is mortal’), which every LLM we encountered so far was able to apply to completely new situations.
(A philosophical argument could be made here that by perfecting the capability for imitation the ultimate result will not be a perfect imitation but it would rather transition into the imitated thing itself, meaning that further development might lead to actual intelligent LLMs; but then a practical argument can be made that the current LLM technology has already reached a plateau, in the sense that if we made those neural nets even larger, their capacity would increase so much that they would no longer be forced to find abstractions, and we’d end up with a tool that knows more facts but is less intelligent than the current ones. The industry experience appears to bear out this observation; the number of parameters has stayed in the hundreds of billions for almost two years by now, experiments to push this number into the thousand billion – like Google’s Bard – notwithstanding.)
Now, any LLM needs a running environment, a shell, to be useful. At a minimum, such a shell would take care of tokenizing the input, feeding it to the LLM and retrieving the answer, but shells usually do way more than that. In fact, it seems like most of the real advances in the past year or so (and the expected advances in the coming year or years) were and will be made by enhancing the shells, not the LLMs. For instance, GPT4-o1 was such a result; we’ll see many others in the near future.
The Average Computer User, however, does not see things in quite this light. For the ACU expectations against AI are set partly by science fiction movies and novels and partly by sensationalist journalism, and the way they appear to see it we have arrived at that kind of cyberpunk future where you can get computers to perform a wide range of tasks without the specialization characteristic to using dedicated applications for each task and perhaps more importantly without the hassle of involving those initiated, anointed and annoying priests of technology, the developers. This is not an entirely wrong intuition, since the main use of LLMs will most likely crystallize around interpreting informal input to and providing informal output from computing processes. On the other hand, figuring out tricky solutions and coming up with superhuman ideas will not be their role anytime soon, inflated expectations notwithstanding.
What the Average Computer User must understand is that without significant, specialized and entirely traditional computing support from the shell, the LLMs have relatively little capability for computing. Un-augmented major LLMs of our time are incapable of reliably adding up three-digit numbers or figuring out the winning strategy of tiny logical games. They cannot look up patterns in an activity log of any practical size. They cannot figure out what’s most likely wrong with a system if all they have is a description of the system and a description of the symptoms. In short, they cannot do either 1) or 2) from the beginning of this text.
Sure, ChatGPT will add up any two numbers, however big they are. But it does so by a clever callback method called by various names; let’s settle here for ‘tool augmentation’. Tool augmentation basically means that the shell provides certain functionalities like looking up stuff from a database, running Python code etc. and describes these to the LLM, which then can choose whether to answer a question directly or turn first to the shell with requests to perform some of those functions with given parameters. Try this: ask the chatbot of your choice whether a large integer (just type something random on the keyboard, but make sure it ends in 1,3,7 or 9) is a prime number. ChatGPT, for example, will even tell you that it wrote and ran a Python script invoking the ability of the SymPy package to perform the primality tests. This is tool augmentation in action.
So in a sense you could, for example, make an ‘AI application’ to consult the wiring diagram of an electronic device and propose tests to narrow down the list of possible defects causing a certain symptom. But one has to do a lot of custom work to be able to do so; feeding the diagram into the LLM can be expected to involve rather complex transformations, providing the LLM with explanations about what the different parts do, and then hoping for the appropriate patterns to exist and work together correctly within the LLM so that the reply will be based on the data and not on useless generalities. The possibilities are almost limitless, but none of them include the ability to simply tell the machine, via informal chat, what to do. For that, you need the classic development of a custom shell. A generic chatbot will not be able to do it, will not be able to realize it cannot do it, but almost surely will output something that looks like it but isn’t.
There are attempts to supplant developer work by using the LLM itself to generate code. This exemplifies the class of problems we call the “Hello, world” problems. “Hello, world” is an allusion to the time-honoured custom to start presenting a programming language through the simple code required to output the string “Hello, world!”, and it designates simple illustrative problems such as you would encounter as the first one during a course of problem-solving of any kind. LLMs excel at these kinds of problems, writing small code, designing small databases etc., thereby creating the illusion that they can solve such problems in general. This is, however, false: without the support of a specialized shell, LLMs cannot tackle problems of this kind if they reach a practically useful size or complexity. Surprising shell solutions are still possible, and special documentation preparation can also be expected to play a role, but the general, zero-cost delegation of such processing to the AI is at best impossible and at worst appears to be possible but yields incorrect results.



