Treating LLMs as Black Boxes

Treating LLMs as Black Boxes

Treating LLMs as Black Boxes

As we said last time, LLMs are a bit like alien artefacts, black boxes whose function and proper use we are investigating in ways not dissimilar to the way our stone age ancestors would puzzle over a plumber’s toolbox. The situation is familiar for those who have had to deal with any but the most trivial machine learning methods; most of those resist any attempt to extract the actual rules and patterns they discovered during training. Partial and specialised methods exist and research is ongoing, but overall we have to confess that we do not understand how the perfectly knowable and known low-level details of the functionality add up to the high-level behaviour we observe. LLMs make this problem even more intractable on account of their enormous size and complexity, therefore, their capabilities are best investigated by treating them as a black box. This process usually results in new ways to use the LLM, not new ways to structure and train it; in other words, the changes influence the application environment driving the behaviour of the otherwise inert LLM. We usually call this surrounding application the shell, and it is this component that needs to be adapted to the use we want to put the LLM to. One of these uses, or rather the architectural solutions serving one of these uses, is called RAG.

The term RAG stands for Retrieval-augmented generation, and loosely speaking it covers all usages of large language models whereby an external mechanism resident in the shell intercepts the question posed by the user, looks up relevant text (or image) fragments from some repository and hands over those fragments to the LLM together with the original question; the LLM can then provide a well-informed answer, not relying solely on its intrinsic knowledge. Enterprises can then build systems that accept informal queries about facts found somewhere in a potentially large corpus of documents and reply summarising those facts in human language, supplying, if necessary, the references to the original loci in the documentation. This concept is used in a wide variety of shapes, from document consultation through diagram interpretation, all the way to intelligent Internet search results. It all sounds simple and straightforward, right?

Well, yes and no. RAG is one of the unquestionably useful applications of LLMs, but it is anything but straightforward to implement one. It is important to emphasise this fact because if one goes by what the Internet claims, putting together a RAG system is a matter of clicking a few times on some Web UI. There are also tutorials and easy-to-use toolkits which allow one to put together a RAG solution quickly and easily. The trap is that the RAG thus configured will work, but usually will not perform very well, and it is rather difficult or even impossible to improve it step-by-step towards a solution that satisfies the requirements of the enterprise. It is understandable if, after such an experiment, enterprises decide that the whole idea is just a fancy trend of uselessness and give up on the whole idea. Understandable, but dead wrong.

First of all, it is important to pay attention to the system instruction (some call it system prompt). This is a text that tells the LLM what it is and what it does. There’s no single system instruction that would serve every need, and so customisation is called for. Traps abound: some LLMs understand system instructions better if put in a certain language, regardless of the expected language of the resulting system; others find it difficult to reconcile the two. All LLMs have some level of difficulty with injunctions, but every one of them has a different preference as to what circumlocutions they will accept instead. Some LLMs can tell the language of a query (for when they are instructed to reply in the same language they were asked in), others not so much and have to be helped out by the shell. This list could be continued pretty much indefinitely. Tuning the system instruction is an arcane art, with very few general rules and a huge reliance upon intuition.

Another, perhaps even more important component of such a solution is the preparation of document fragments (usually called chunks) in the case when a RAG solution is used for search in a document collection. Most enterprises have most of their documentation in PDF format, and the usual quick-and-dirty solution in this case is to grind the PDF through some PDF-to-text tool (from pdftotext of pdftools fame all the way to LlamaParse). This is often inadequate for the needs of a RAG solution, and more involved solutions are called for; we’ll return to this issue later. Anyway, having obtained the text of the document, we cut it into overlapping fragments according to some heuristics that aim to avoid splitting text units (paragraphs, sentences, etc.) whenever this is possible. This kind of fragmentation has to satisfy so many constraints that it is no wonder if its point-and-click mock-up fails pretty often: there’s a size limit imposed by the embedding mechanism used, which can usually be silently overstepped, and all that happens then is that the chunk gets the wrong embedding vector; there’s an optimal size that will most likely ensure that individual topics stay within a single chunk and chunks are most likely to contain one or at most a very few topics (to make things worse, this may vary with the author of the document; some people are long-winded, others are terse to a fault); too much overlap may increase the number of chunks and – more importantly – cause too little information to be presented to the LLM because the number of chunks added to the query is usually fixed/limited and if they are slightly shifted snapshots of the same text portion then other, also relevant texts may not make it into the prompt; too little overlap may hopelessly cut content from context; and the list could go on. All these problems have solutions, but the problems need to be recognised, the proper solution chosen and implemented in the RAG shell.

RAG shells usually search for relevant text fragments using a technique called text embedding, and although other methods exist, they are almost always used in tandem with this one. Text embedding is an unfairly ignored major achievement of modern AI; it might even be claimed that it is a greater achievement than LLMs themselves, in the sense that embedding comes closer to our notion of ‘the machine understanding the text’ than whatever LLMs do. Text embedding maps text fragments to numeric vectors (lists of numbers having a fixed length) in such a way that if two text fragments have similar meanings, refer to similar topics etc. then their embedding vectors will be close to each other, for some exact meaning of ‘close’ (out of a few alternatives). This property of embedding vectors is used to find text chunks relevant to the question posed by the user by identifying the vectors of chunks closest to the vector of the text of the question. Embedding models are separate from LLMs – you can mix and match pretty much any embedding model with any LLM, but they come with their own caveats and limitations. There’s usually a length limit on the text to be embedded, although this length limit is very soft, meaning that some embedding models will not inform you when you overstep it, silently generating less reliable embedding vectors instead. Probably all embedding models support English, but for any other language you better check because the embedding model will not tell you if the language of the text fragment is unsupported (or if it is supported but the text is too dadaist to make sense – all texts have an embedding vector and that’s what you will get). Some embedding models require standard prefixes to be prepended to the text because they work better if they know whether the text is a query or a text fragment containing information relevant to the answer. Again, we have a list of caveats that could be continued for quite a while.

Other techniques are occasionally needed. Sometimes the system has to answer questions about the documents, not from the documents. For instance, one could ask when/in which version of a contract the SLA level last changed. This will not work at all with the basic RAG solution, and much more sophisticated logic has to be implemented in the shell. Questions may be too terse for direct processing and may make extensive use of informal terminology, shorthands and acronyms not present in the document repository. This won’t work with the simple setup either, but it is likewise solvable in the shell.

By this litany of pitfalls, we did not intend to discourage experimenting with the use of LLMs in an enterprise environment. These aren’t reasons RAG won’t work for you, they are reasons why the quick-and-dirty proof-of-concept implementation doesn’t seem to hold promise. Don’t give up. These issues are solvable, although there are no fixed recipes and, as of yet, no Great Book of Tricks to look up the solutions. There is, however, one particular aspect that can be crucial if your reliability requirements are high, and this is the way PDFs are converted to computer-friendly text. This is a huge topic on its own, and we’ll return to it in a separate instalment.

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.