AI Engineering Is About Knowing When Not to Use AI

One thing I keep noticing in AI implementations is that we are becoming very good at asking what AI can do, but not always asking whether AI is the right component for the problem.

An LLM can generate SQL. An agent can execute a fixed workflow. A model can evaluate a business rule written inside a prompt.

That does not mean we should design systems this way.

The challenge in production AI is not to use AI everywhere. It is to find the right boundary between traditional software and probabilistic systems.

Start with the problem, not the AI pattern

Traditional software already solves many problems extremely well.

Databases store and query facts. Code performs calculations. Rules engines enforce known policies. Workflow engines execute defined processes. APIs provide controlled access to systems.

These are deterministic problems, and deterministic components usually give us predictable results.

AI becomes valuable when we reach a different type of problem.

A customer can describe the same problem in hundreds of different ways. A document can contain information in an unpredictable structure. A support incident might require reasoning across several pieces of evidence. An automated process might need to select its next action based on information that was not available when we designed the system.

These problems contain uncertainty.

That is where AI starts to make sense.

A simple hierarchy

I use the following hierarchy when thinking about a new requirement:

Code → Search/Retrieval → LLM → Agent

Start from the left and move to the right only when necessary.

If the answer can be calculated or determined from known rules, use code.

If the answer already exists and the problem is finding it, use search or retrieval.

If the system needs to understand language, interpret ambiguous information, or reason about something uncertain, use an LLM.

If the system must decide which actions to take and adapt those actions based on what happens, consider an agent.

Moving to the right gives the system more flexibility, but it also introduces more uncertainty.

That trade-off matters.

Code when the answer can be determined

Consider a customer asking for the status of an order.

If we already have the order number, the application can query the database or call an API. There is little reason to ask an LLM to generate a database query first.

The same applies to calculations, permissions, pricing rules, data validation, limits, and known business rules.

However, the situation changes if the customer says:

“What happened to that large order I placed sometime last week?”

Now we have uncertainty.

An LLM can interpret what the customer means and extract useful parameters. The application can then use deterministic code to retrieve the actual order information.

The LLM handles the ambiguity.

The software handles the fact.

Retrieval when the answer already exists

Suppose an employee asks for the company's international travel policy.

The information already exists. The first problem is finding it.

Search or retrieval can solve that problem.

If the employee instead asks whether specific expenses for a five-day client trip qualify under the policy, an LLM can become useful. Retrieval finds the relevant policy, while the LLM helps interpret it against the employee's question.

This is where RAG can provide real value.

Retrieval provides the evidence. The LLM interprets the evidence.

LLMs when interpretation is required

Natural language is inherently variable.

A customer might say:

“The thing you sent doesn't fit and I just want my money back.”

An LLM can identify this as a refund request and extract the reason.

That does not mean the LLM should decide whether the refund is allowed.

If the company has a defined refund policy, code or a rules engine should make that decision.

This gives us an important boundary:

Let AI interpret intent. Let software enforce truth.

The same principle applies to permissions, compliance rules, financial calculations, pricing, and other decisions where the organization already knows the correct rules.

Agents when the next action is uncertain

Agents become useful when the system cannot fully determine the execution path in advance.

Consider an IT incident where users in one region cannot access a business application.

The system might first check application health. If the application is healthy, it might inspect authentication. If authentication is normal, it might check regional networking. The evidence from each step determines what the system should investigate next.

This is different from a fixed five-step process.

If the process is always:

Receive → Validate → Process → Approve → Notify

we already know the workflow.

Use a workflow engine.

Asking an agent to decide what to do after every step adds probabilistic decision-making to a process that was already deterministic.

Agents make more sense when the path itself cannot be completely known in advance.

Production AI is hybrid architecture

This is why I do not think AI architecture should be treated as a replacement for traditional software architecture.

A production AI system will still have databases, APIs, queues, caches, search systems, rules engines, workflow engines, security controls, and observability.

It will also have LLMs and, where justified, agents.

The important engineering decision is where each component belongs.

A customer request might use an LLM to understand intent, retrieval to find relevant information, code to validate business rules, and an agent only if the remaining task requires adaptive action.

The real AI engineering skill

As AI becomes easier to implement, I think this distinction becomes more important.

The ability to connect an LLM to a database, add RAG, or create an agent is no longer the difficult part.

The difficult part is deciding whether those components should be there in the first place.

Good AI engineering requires understanding both sides of the architecture: probabilistic AI systems and traditional deterministic software.

The question should not be:

“Can an LLM do this?”

It should be:

“What is the simplest component that can solve this problem reliably?”

Sometimes the answer is an LLM.

Sometimes it is an agent.

Sometimes it is retrieval.

And sometimes the best AI architecture decision is to write some code.

Good AI engineering is not about replacing traditional software with AI. It is about knowing where traditional software should end and AI should begin.