How Not To Use LLMs with IDP

Large Language Models (LLMs), like the one behind ChatGPT, have elevated the capabilities of AI systems, setting new standards from acing Wharton's MBA exam to replacing a nutritionist. Many Intelligent Document Processing (IDP) vendors, including Infrrd, have announced integrations between their knowledge graphs and LLMs. Now that the initial hype has settled, let's delve into the practicalities of this technology.

Can I Get Some More GPUs Over Here?

First things first, AI models are computationally intensive, and when I say intensive, I mean they have a voracious appetite for computing power. This shouldn't be surprising – those billion parameters used for training these models need to be applied during prediction to some extent. If you're using an externally hosted LLM like ChatGPT, you might not realize it, but by some estimates, OpenAI spends $700,000 a day to answer all the queries it receives. For companies like us that manage our LLM infrastructure, we know that the infrastructure costs to run these models are prohibitively high.

Why it matters for IDP:

The primary goal of IDP solutions is to provide scale and cost reduction for customers who traditionally rely on people to read documents. Since AI-based IDP solutions reduce manual labor, there is a cost to consider. If your document processing costs exceed or come close to human costs, it significantly impacts one of the main drivers for IDP adoption – ROI.

Clearly, You Are Hallucinating!

Have you ever asked the same question multiple times to an LLM and received different answers? It's great when you're seeking ideas, but not so great when you need consistent answers from documents. This inconsistency can compromise the accuracy and predictability of document processing.

Why it matters for IDP:

Some IDP providers, using proprietary algorithms, have come close to achieving almost perfect accuracy. The challenge for these models lies in the training data needed to achieve this accuracy. While LLMs provide a great start, other models offer a solid finish. The former matters during the sales cycle, and the latter is crucial for retaining satisfied customers.

Please Take A Token!

While these limits are gradually expanding, most LLMs have serious constraints on the number of tokens they can process in one go. This poses challenges in processing large volumes of documents concurrently. The limit stems from the expensive computing power required to run LLMs.

Why it matters for IDP:

Many IDP customers, such as mortgage companies and banks, need reliable data from IDP solutions in real time to run business processes. These processes cannot afford to wait for data to arrive after a few minutes. Token limits can potentially hinder IDP processing at scale.

I’m Sorry, Can I get a Little More Context, Please?

Arising from the same challenge of expensive computing leading to token limits is the difficulty of maintaining context. When an IDP solution processes a 40-page document, it may find the relevant answer on the 13th page, but an LLM attempts to find an answer within every token limit it processes. This can lead to more hallucinations, where it tries to return an answer that is not within the given token set.

Why it matters for IDP:

If you ask any customer about the most important aspect they look for in an IDP solution, their answer, without a doubt, will be accuracy. Higher accuracy means lower costs, happier customers, and better business. The risk of accuracy is quite high, especially with varying documents.

The Great Start Problem

Hundreds, if not thousands, of books have been written about the way the human psyche works. Daniel Kahneman has done some amazing, Nobel-winning work in this area. You can witness some of it in action when people try an LLM for the first time. They try one variation of the document, ask a specific question that can only be answered for that particular variation, and quickly jump to the conclusion that LLMs are awesome. However, as they spend more time with it, it becomes apparent that much more work is needed to make LLMs a viable, reliable, and economical business technology.

Here is real test data on the prediction of LLMs and IDP algorithms for four values against complex documents with a lot of variations:

‍

‍

You can clearly see the LLM struggling in three out of four tests. The pattern becomes more apparent with complex documents.

Why it matters for IDP:

Before AI-based IDP systems, and in some cases still true for some IDP systems, most tools worked off templates. Any customer who has worked with these systems will tell you how much it damaged their trust in technology. Using LLMs blindly will lead IDP systems and customers down the same path.

But You Promised Me!

One of the really cool things that IDP systems offer is their ability to constantly learn from corrections based on your data. Even if you start with low accuracy, as long as you keep using the system, accuracy improves using this ML Feedback Loop. This loop takes a big step backward with the use of LLMs. It is cumbersome to fine-tune LLMs for each customer's data. You need to rely more on fundamental enhancements to LLMs rather than small, incremental steps like retraining, which make a huge impact for customers.

Why it matters for IDP:

Customers derive more value from a 70% accurate model that can reach 99% accuracy in a few weeks than using an 85% accurate model forever. This can significantly flatten the ROI curve for customers.

‍

‍

LLMs are here to stay, and they represent a significant step forward for every AI company. However, there is a right way to employ technology and a blind way to do it during a hype cycle. Due to the stage we are in, we see a lot of the latter these days. At Infrrd, we are working on multiple in-house LLMs for different problem areas to help advance our IDP platform. Done right, they can make our customers' lives a lot easier.