Can We Really Trust AI to Code?

proramcode on a screen

Can We Really Trust AI to Code?

Let’s turn to the big promise of large language models: code generation.

The dream of specifying a computation in informal or almost informal language and still getting it done by the computer is as old as high-level programming languages themselves. The COBOL language already aimed at something like this (the idea being explicit already in the design of its ancestor, B-0 or ‘FLOW-MATIC’), although dire necessities posed by the meager state of the art in the late 1950s forced the introduction of highly formal declarative sections that, in turn sank the whole idea. A similarly overt attempt was made by SQL in the 1970s (there’s a reason it was originally called SEQL – Standard English Query Language). Less overt attempts at the same goal were made by declarative programming languages (’you just specify the what, not the how’), whereby, although there was no intent to do away with strict formality, there was still an intent to remove the need to explicitly specify the underlying algorithmic tricks. Then came the monster called ‘real-life applications’ and trampled over the whole idea back and forth.

Enter large language models.

Although information on this topic is somewhat scarce, it would seem that current mainstream LLMs are trained on datasets having one-quarter to one-third of their tokens coming from program code. (This does not include specialized coder LLMs, for which this ratio would be much higher, somewhat to the detriment of natural language abilities – sometimes these specialized models aren’t really multilingual, let alone multimodal.) Since at a very abstract level most programming languages share structure with natural languages, it is only a mild surprise that LLMs ended up being able to correlate the semantics of natural and programming languages; LLMs can effortlessly translate between any two natural languages they understand, and the process of turning an informal natural language specification of a task into a program code performing the same task is rather similar to just another sort of translation between languages. Or at least we are entitled to think so.

Nonetheless, the coding performance of LLMs is, to make use of a euphemism here, rather mixed. LLMs generate code at a much higher speed than any human can hope to, but conformance to specification is a toss-up: either you get correct code or you get the other kind. Some organizations insist that this is not a problem, and indeed, we can imagine situations when 5% of the output being wrong does not outweigh the increased speed at which requests of custom code are served, even if nobody can identify that 5% in time. This is dangerous thinking because, outside certain narrow specialities, humans aren’t really capable of evaluating real-life probabilistic statements.

One major image problem here is that when the code generated by an LLM turns out to be wrong, it is not just a little bit wrong but spectacularly, amazingly wrong. LLMs aren’t into minor coding errors like forgetting to update the cycle variable or committing off-by-one indexing errors; in fact, those kinds of errors, the bane of any beginner or just dog-tired programmer, are pretty rare in LLM-generated code. No: when the LLM creates wrong code, that code is likely to do something rather unrelated to what has been asked, or fancies library functions that do not actually exist and then attempts to call them, or invents completely fictitious database fields or even tables that would be convenient for the task at hand if they existed, or simply forgets to write part of the code for no discernible reason. When people see these rather iconic disasters, they tend to mentally amplify the 5% to something of much more sinister proportions because they’ll naturally assume that if the 5% can be this much wrong, maybe the remaining 95% is also wrong, only less evidently so.

Human programmers do occasionally produce buggy code. But their output is much more reliable as of yet, because they perform a large number of inspection cycles. LLMs don’t do this as a rule. They could be forced to review their own code and fix the problems they find, but a hard limit seems to exist here, too: performing such a pass is likely to introduce new problems, forget some of the specification details, etc., so the bugs will start moving around in the code but will always keep up a minimal population.

Systems turning informal specifications into formal program code are most likely to be used by people who either could not write the formal code themselves or at least have no time to verify it. So it seems that the 5% problem hits exactly those people who most need this kind of functionality. Those people will have enough trouble discovering how many implicit assumptions they take for granted. When the LLM comes up with code that does what one interpretation of the request says, the interpretation being one that the user would not have thought possible. Asking for the best-selling products of some product line in the last year might give you the one best-selling product, or all of them (although probably ordered decreasingly by sales volume), because the user did not say how many of them (top 3, top 10 …) he wants to see, and anyway this number depends somewhat on the total number of products (it is kind of funny asking for the top three products when there are only four of them). Asking the machine to compare two trends might yield the answer ‘Ok, I have compared them. What is your next request?’ or something similar. Users have to learn to prompt the LLM, and this process is not going to be easy either for them or on the people who try to sell such solutions and have to explain why does it sometimes behave like a contrarian child.

A major code-from-LLMs application area is text-to-SQL. Here, without very careful preparation, having only 5% of the code be wrong would look like an overly ambitious goal, unattainable on any non-trivial database. Blame for this state of affairs has to be shared between people who create – actually, more likely than not: snap together – text-to-SQL solutions, usually first as a pilot with reduced scope, and between people who maintain the databases that are to be used this way. The usual way to go about a text-to-SQL (or text-to-pandas, or …) pilot is to generate some form of schema description from whatever introspection functionality is possessed by the database manager, and then expect the LLM to figure out the workings of the database from that schema specification alone. This may be a reasonable expectation in the case of a toy database designed by an unusually fastidious computer scientist, but not for any serious database encountered in the wild. For instance, relationships between tables could be encoded in database constraints, but tradition calls for foregoing their use because ages ago (i.e., around 2000) verifying those constraints was somewhat costly in terms of performance, so the most orthodox programmers encoded but promptly disabled them, perhaps expecting the improvements in hardware performance that arrived in our time to allow for a re-enablement. The other kind of programmer did not even record them. But without knowing those relationships, a text-to-SQL translator will not be able to reliably – or indeed at all – figure out queries which require joins, i.e., pretty much all serious queries in a properly normalized database. Worse, the LLM is usually expected to figure out the semantics of the data from the table and field names alone, and this is very unlikely to succeed at a large enough scale (and no, having a naming convention, even a strictly enforced one, will not save you here). We encounter again the situation where the solution works for the first example in the beginners- textbook problem, but it does not scale up to industrial sizes.

What is really needed is a description of the database that fully captures the semantics, the logical data model, and its mapping to the physical data model. In more common terms, this would include a description of the main entities and transactions stored in the database, the way these are stored in tables, conventions about representing the data, the meaning of values in lookup tables, the key-to-key relationships between the tables, and the meaning of individual fields. Oh, and this description has to be completely up-to-date at all times. This kind of documentation is usually completely non-existent in most enterprises, although it can be seen how it would also be useful whenever a new programmer joins the IT team. In fact, the style guide for such a description should be part and parcel of the data governance arsenal of any enterprise – hopefully, the spread of text-to-SQL will further highlight the need for it. Naturally, since maintaining such a document in synch with the actual database schema and semantics would be very difficult otherwise, we should try to keep the extra information needed to generate such a database schema description within the database itself, presumably by adding information in the form of comments and aspect descriptors to the code used to create the database itself, and processing it with some adequate – in our days probably custom-made – tool into the special description text-to-SQL tools require.

This is not really the end of the issues. LLMs know an amazing amount of facts or fact-like patterns, but it is rather difficult to add to this knowledge base. One can only add so many rules and facts before the LLM starts randomly ignoring some of them. A full database description may well pass this threshold, in which case additional trickery is required – pre-filtering the query to figure out what tables might be needed, then composing a subset of the full database description on the fly and only show this relevant part to the LLM. This requires, for example, being able to find all tables that link to tables found relevant; not an insurmountable task, but a good indication of how strict and formal the database description really needs to be.

In closing, it takes a lot – sometimes a great lot – to get useful code out of an LLM. Information about the domain has to be encoded, carefully apportioned to avoid causing cognitive overload, users have to be trained in proper prompting, and then the whole output has to be viewed with a healthy amount of suspicion. This seems to relegate text-to-SQL-via-LLM applications to areas where the relatively high error rate can be tolerated, which aren’t likely to be core areas where automation would be easily profitable. On the other hand, text-to-SQL was a vivid research scene before the arrival of LLMs, and the main obstacle back then was understanding the user’s request, so some kind of hybrid solution, having the LLM understand the request and a symbolic AI application answering it, looks promising. We shall see.

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.