Why an LLM should not be your whole AI system

One of the most common confusions I see today is thinking that implementing "AI" means connecting an API from OpenAI, Anthropic, Google, or any other provider.

But that is like saying that building a digital company means "having a website."

Technically it can be part of the system. But it is not the system. To explain this, I want to use an example: radiology.

Not because this is an article for radiologists. It is not. I want to use it because:

1. I get to reuse research I had to do a few months ago.
2. Radiology shows very well something that applies to almost any industry: manufacturing, retail, finance, government, or logistics.

Namely, that the question any team in these sectors should ask is:

What kind of problem are we solving, with what kind of data, at what level of risk, and with what architecture?

Not every AI problem is a language problem, and not every language problem is solved with an LLM alone.

Radiology as the example

From the outside, automating a radiologist's workflow can look simple:

1. An X-ray comes in.
2. Someone looks at it.
3. They write a report.

Even Geoffrey Hinton, one of the most renowned researchers in artificial intelligence, winner of a Nobel Prize in Physics and the Turing Award, predicted that it made no sense to keep training radiologists, since their work would be automated.

So it would be easy for any of us to think:

"Perfect, we pass the image to an OpenAI model and have it write a report."

The first problem is that nobody mapped the real workflow. Silly as it sounds, this is as easy to do as it is to skip: being clear about the flow from start to finish.

A radiologist's real workflow looks more or less like this:

Medical order → medical image → radiologist review → comparison with previous studies → report dictation → editing → signature

The second problem is not understanding the types of input and output data.

1. An X-ray, a CT scan, or an MRI are not just "images." They are specialized medical data.
2. And the output report is not just "text." It is a clinical document that can influence a real medical decision.

An AI system cannot limit itself to sounding smart.

• It has to be correct.
• It has to be verifiable.
• It has to know when not to answer.

The third problem is that many of us are confusing automating tasks with automating jobs and responsibilities.

• The radiologist's job is patient care, and one of the tasks of that job is producing reports.
• Also, every report carries responsibility. If the report says crazy things, the radiologist is the one responsible.

I doubt most of us would fully delegate our health, accounting, finances, and so on to a model that can hallucinate, without looking at its work.

The interesting thing about radiology is that the job did not disappear. In many health systems almost the opposite happened: the volume of studies went up, operational pressure went up, and the radiologist's role remained necessary.

The role of many robust systems today is to try to automate tasks, not to replace responsibility.

What using an LLM in a predictive radiology model could look like

There are several possible architectures, depending on which problem you want to solve.

But here I will share one used by a startup competing against Microsoft's MAIRA models and Google's Gemma.

Concepts, evidence, and a constrained LLM

In this architecture, instead of letting the LLM freely generate the report:

1. The system first extracts clinical concepts from the image.
2. Then it connects those concepts with visual evidence.
3. And then it uses a language model to write, but under constraints.

Something like this:

image → clinical concepts → evidence → constrained generation → verifiable report

This architecture tries to separate the roles better.

1. The vision model identifies signals.
2. The SAE creates an evidence layer.
3. The LLM helps with the writing.
4. The human reviews and signs.

---

SAE stands for Sparse Autoencoder. At a simple level, it is a model that tries to find more interpretable internal patterns inside the representations a vision model already produced.

The "sparse" part matters because the system tries to activate few relevant signals at a time, not a huge mix that is hard to understand. That lets some features be associated with clinical concepts like pleural effusion, cardiomegaly, or the presence of a pacemaker.

So the LLM is not writing from scratch. It is writing about signals another model extracted from the image beforehand.

---

The challenge is that this architecture is also more complex, because you will need to:

1. Verify that the evidence is real.
2. Prevent the system from showing explanations that look convincing but are not causal.
3. Handle rare cases.
4. Monitor drift.

But that complexity, in an industry where every mistake can hurt a person, is part of the requirements.

The lesson for companies

Why does this matter if your company is not in healthcare? Because the same principle applies in other industries.

• If you are in retail and want to forecast demand, you do not need to ask ChatGPT to "guess sales." You need forecasting models, historical data, seasonality, inventory, and external signals.
• If you are in finance and want to detect fraud, you need models on transactional data, behavior patterns, graphs, or anomalies.
• If you are in logistics and want to optimize routes, you need mathematical optimization, operational constraints, and real-time data.

Start by asking: "What is the problem, what is the data, what are the risks, and how will we measure whether it worked?"

Once you have that mapped, choose the type of architecture you need according to the process, the input data, and the output data.