The Reductionist Trap: Why Your AI Optimization Is Breaking Everything Else

 

Three years ago, a fintech company created a model to predict when customers would leave. This model was good. It was correct about ninety-two percent of the time. The company was able to get it up and running in six weeks. The team that made it even won an award inside the company.

The model was able to figure out which customers were going to leave before they did.

By the month, the sales team had stopped using the model completely.

This was not because the model was wrong. The problem was that nobody had thought about what would happen when the sales team got a list of customers who were about to leave on the day that the accounting team finalized the targets for how much the company could spend. The accounting team had decided not to hire any people so they could meet their targets. The sales team needed to keep those customers to meet their targets for how much money the company could make. The model said that the sales team should try to save these customers by offering them discounts and special service. The finance team said that the company could not spend that much money. The sales team asked who would pay for the discounts if they were offered to save a customer that spent two hundred thousand dollars. The finance team did not have an answer.

So, the sales team just ignored the model.

The model was perfect, in a way. It did not work with the rest of the company. The company had problems that the model could not fix.

This is what happens with a lot of intelligence projects.

The Reductionist Illusion

A reductionist thinks that if all the parts are working well, the whole system will work well.

This is usually not true when people are involved.

A company that sells things in stores found this out in a way. They made a computer program that could figure out how much of each thing to keep in stock. It could predict what people would want to buy, which store they would buy it from and what day they would buy it. The program was very good at math.

They used the program. It worked pretty well. They did not have as much extra stock as before, which saved them money. They also did not run out of things much, which made customers happy. This was great.

Then they started using the program in two hundred stores.

After a couple of weeks, the people in charge of the stores started calling. They said the program was telling them to keep a lot of winter boots in stock during the summer. This would cost a lot of money to store them. What really happens is that people look at the shelves when they are in the store. If they do not see what they want, they will not come back later to look for it. The people in charge said that the program only looks at numbers; it does not understand how people think and behave.

The program was actually right about how many winter boots people would want to buy in the fall. It was wrong about how people think and behave during the summer. It only tried to save money on storage without thinking about how people would react.

The company did not need to make the program better. They just needed to add some rules. They told the program that it always had to make sure that people could see the things on the shelves. Now the program had to work with the way things really are.

The company learned that the retail company system had to work with the retail company reality.

The retail company system had to optimize within the retail company reality.

The retail company had to make sure that the retail company system understood how the retail company people behave.

The Org Chart Problem: Where Alignment Breaks

Here is where a simple idea like reductionism meets a problem like organizational gridlock.

Most companies are divided into teams that do things:

• The Sales team is in charge of making money. They look at how much sales they might make and how often they close deals.
• The Product team is in charge of getting people to use their products. They look at how people use the features and how happy they are with the product.
• The Operations team is in charge of making things run smoothly. They look at how much it costs to do each transaction.
• The Data Science team is in charge of making sure the information is correct. They look at how precise and accurate the information’s

Each team does what it is supposed to do. Each team has goals that make sense. Each team will keep trying to do

Then you start using a churn prediction engine.

The engine finds 50 accounts that’re at high risk of leaving. It is 90% correct. Now watch what happens:

The Sales team thinks: I need to give these customers $15,000 in discounts and support to keep them.

The Operations team thinks: That is $300 per customer. It is a lot of money for this group of customers.

The Product team thinks: If we have to spend that money to keep customers, our product is not good enough. I should use our engineers to make features that will keep customers instead.

The Data Science team asks: Why are people not using the prediction engine? It is accurate. I need to make it even better.

The Finance team says: Who is going to pay for the things we need to do to keep these customers?

Nobody is wrong. Everyone did what they were supposed to do. The system still got stuck.

I know a company that had this problem. The churn prediction engine was good. It took 14 months to get it working. Not because it was hard to build. Because the Sales, Product and Finance teams could not agree on who would pay to keep the customers. In those 14 months, customers left the company. The algorithm said they would leave. Nobody did anything. Everyone was doing what they thought was right. The company lost.

Real Failure Mode: The Drift Nobody Caught

Here is what actually breaks systems:

A company used a demand forecasting algorithm. On test data, it was about eighty-nine percent of the time. That is a signal. So, they put it to work.

The month it was right about seventy-six percent of the time. That was expected because data changes and the model adjusts.

By the month, it was right about sixty-four percent of the time. That was worrying.

By the month, it was right only about fifty-three percent of the time. That was a disaster.

What went wrong? Nobody was checking for changes in the data. The idea was that what happened before would happen again. During those six months, the demand forecasting algorithm had three big problems:

1. A competitor started selling. That changed how customers behaved
2. The company made a change to their product. That changed how users interacted with it
3. There were problems with getting products to people. That changed when people bought things

The demand forecasting algorithm was good at being right about old data. Not good at being right in a world that is changing.

Some people say the answer is to make the demand forecasting algorithm better by adding details.

A better way is to build a system that watches the demand forecasting algorithm and checks when real life is different from what the demand forecasting algorithm predicts. The system should automatically update the demand forecasting algorithm. Tell people when things change. It should have a way to catch changes before they affect customers.

This way is harder. It is not about the demand forecasting algorithm it is about the system that watches the demand forecasting algorithm. It is the difference between something that works for a little while and something that works for a long time.

What Systems Thinking Actually Looks Like

It’s not philosophy. It’s asking one hard question after you’ve built something:

“Now that we have this algorithm, what has to be true about how people use it, how it connects to other systems, and what success actually looks like—for this to work in the real world?”

Then you redesign based on the answer.

For a forecast algorithm:

Don’t assume sales will adopt new workflows. Co-design the workflow with them first. Before the algorithm launches. Build monitoring for demand shifts. Define what “accurate enough” means for your economics—one wrong forecast on a $1M order costs you $100k? That changes the accuracy target.

For a churn prediction system:

Assign the owner of the intervention budget before deployment, not after. Define what “act” means: discount? Retention call? Product feature request? Different levels have different owners. Build feedback loops that track what the algorithm predicted vs. what happened. If accuracy degrades, retrain and alert.

For inventory optimization:

Add constraints based on psychology, not just cost. Minimum shelf visibility. Protect high-discovery items. Build fallbacks for when supply chains break. What happens when your suppliers can’t deliver? Your algorithm doesn’t know. Your system needs to.

The Bottom Line

You do not choose between reductionism and systems thinking. You need both reductionism and systems thinking.

To build intelligence, you should use reductionism. To deploy intelligence, you should use systems thinking.

You should optimize the pieces of the intelligence very carefully. The artificial intelligence model should be as accurate as you can possibly make it. However, when you deploy the intelligence, you should never forget that you are putting it into a system that has its own logic. Money goes through budgets. People have incentives that are tied to outcomes. Information travels through channels. Humans make the decision.

If you break the logic of the system, no amount of model accuracy can save you from problems.

For example, that financial technology company that broke their customer churn system did not rebuild the algorithm. Instead, they built a system where the algorithm actually mattered. The company that deployed a forecast engine and watched it get worse over time needed to have monitoring and retraining loops, not training data.

The retail company that had problems with the algorithm did not add features to the algorithm. They added constraints to the algorithm instead.

You should start with reductionism. Finish with systems thinking. This is the difference between a data science project and a business outcome that actually works. Reductionism and systems thinking are both necessary for success.

Agents Are the New Microservices

 

A team I talked to recently took their working AI feature (one model, one prompt, one call) and split it into twelve cooperating agents. A planner, a researcher, a critic, a writer, a few tool-callers. The demo was beautiful.

Then they tried to run it in production and discovered they no longer had an AI problem. They had a distributed systems problem. The exact one the rest of us solved, painfully, a decade ago, and called microservices.

If you sat through the microservices years, the agent conversation in 2026 should feel like déjà vu. Same pitch, same gleam in everyone's eye, and, I'd bet, the same bill coming due.

The industry decomposes on a cycle

Step back and software architecture is a pendulum. Monoliths to SOA to microservices to serverless. Every wave breaks the system into smaller, more independent pieces, sells the same benefits (independent deployment, team autonomy, scale the hot parts) and quietly ships the same costs. Agents are the next swing.

But there's a real insight underneath the hype, and it's worth saying clearly. Microservices decomposed your code. Agents decompose your work. A microservice is a slice of functionality waiting to be called. An agent is a slice of a decision, pursuing a goal. That's a genuinely bigger idea, and it's why the pitch is seductive: you stop wiring logic for every edge case and instead hand a goal, some tools, and constraints to something that figures out the path. When it works, it's remarkable.

The tax comes back, with interest

What the slide never mentions, then or now: the moment you split one process into many that talk to each other, you inherit every problem of distributed systems whether you wanted them or not. Unreliable calls, timeouts and retries, debugging across boundaries, tracing, state that used to be a variable and is now a message someone might not receive. Conway's law shows up on schedule: your twelve agents start mirroring your org chart instead of your problem.

We paid for those lessons in outages. Agents inherit the whole invoice. But they also change the shape of failure, and this is the part that should keep an architect up at night.

A microservice fails loud. HTTP 500, a timeout, an alert. You know. An agent fails quiet. It returns something fluent, confident, and wrong. There's no error code for "plausible but false." One agent hallucinates, the next treats it as fact, and the mistake compounds down the chain like a game of telephone with no exception thrown. You don't get a red dashboard; you get a subtly bad outcome three steps later.

That's because the "contract" between two agents is natural language. No schema, no types, no compiler to catch a mismatch. The thing distributed systems spent twenty years learning to depend on, gone. And the work isn't deterministic, so the same input can take three different paths, which means you can't unit-test your way to confidence. Add that each hop is a model call, so latency and cost stack, and your elegant twelve-agent design can burn fifty inference calls to answer one question.

Honest version: agents can be microservices with brains, or microservices with worse tooling. Which one you get depends entirely on the discipline you bring.

What's genuinely better (To tax, you need income)

It would be lazy to only sound the alarm. Some of the upside is real and new.

Agents adapt without a redeploy: change a goal or a prompt, not a CI/CD pipeline. They can wrap a legacy monolith that has no API, driving it as a tool, which resurrects systems you'd otherwise pay millions to rewrite. They can self-heal and triage, routing around failures instead of just reporting them. And service discovery gets smarter: instead of looking up payment-service-v2 by exact name, an orchestrator can ask for "something that can process a payment" and match on capability. These aren't small.

So agents won't replace microservices

The headline is provocative, but the literal version is wrong, and the options that claim agents simply replace microservices are selling the same over-decomposition that gave us the distributed monolith. The accurate version: agents won't replace microservices. They'll sit on top and consume them. The deterministic, transactional, regulated heavy lifting stays in boring, reliable services. The cognitive, orchestrating, user-facing layer becomes agents. The future is a hybrid, and the architect's job is drawing that line well.

The lessons we already bought

We're not starting from zero. We paid for this education once. The trick is using it.

Start with the monolith. The hardest microservices lesson was "don't decompose first." One well-built agent beats twelve that need a committee meeting to answer a question. Split only when a real seam forces it.

Schema the handoffs. If agents must pass work, pin the boundary to typed, validated structure, not free prose. Give the natural-language soup a contract where it crosses a line.

Buy observability before you scale, not after. With agents you need more than logs and traces; you need the reasoning and inputs at each hop, or you'll never reconstruct why the system did what it did. Tracing a request becomes tracing a thought.

Don't decompose by hype. We split into microservices because it was fashionable, then spent years merging them back. Don't earn that scar twice. Decompose by genuine boundaries (different scaling, ownership, rate of change), not because "multi-agent" sounds advanced on a slide.

The actual point

The architects who came out of the microservices era well treated it as a tool, not a dogma. They knew "distributed" is a cost you pay for a benefit, and they paid it only when the benefit was real. The ones who suffered adopted the pattern because the conference talks were exciting.

Agents are at exactly that fork. The capability is real, the architecture is familiar, and the failure mode is predictable, because most of us have lived it once already.

So before you split your working system into a swarm of clever agents, ask the same question that should have been asked in 2015: do you have a problem that actually needs distributing, or do you just like the diagram?

AI Is Replacing Traditional Software - Here's What Comes Next

 

For thirty years, software meant the same thing: a team writes deterministic rules, ships a UI, and the user adapts their workflow to fit the tool.

That model is breaking.

AI-native applications don't ask users to conform to a rigid interface. They interpret intent, generate the interface on demand, and execute a workflow that used to require a dozen manually-configured screens.

The product isn't the UI anymore. The product is the outcome.

Three shifts are already visible:

1. The interface becomes disposable.
When software can generate its own UI per task, "features" stop being a durable asset. Screens and dashboards become ephemeral artifacts generated on the fly. This inverts decades of product strategy built around owning screen real estate.

2. Vertical SaaS unbundles at scale.
Products that survived on workflow lock-in—niche compliance tools, industry-specific dashboards, specialized add-ons—are exposed. If an AI agent can read a spec, query a database, and generate the same workflow in minutes, the switching cost was never the workflow. It was the lock-in. And that's collapsing.

3. Value moves to data, judgment, and orchestration.
The defensible layer isn't the interface anymore. It's proprietary data, domain-specific evaluation logic, and the systems that coordinate multiple AI agents reliably. Companies that only sold "a nicer way to click buttons" are the most exposed.

What comes next:

Pricing models shift from per-seat to outcome-based or consumption-based. The winners aren't the companies with the most polished UI—they're the ones that control the data pipelines AI depends on and the compliance expertise AI can't fake.

Traditional SaaS didn't survive the cloud transition by defending the old model. It won't survive this one by defending the interface.

The question isn't whether AI will replace traditional software. It's: Does your company control the moat AI actually needs?

Why Most AI Investments Will Look Like ERP Projects by 2028

 

Right now, enterprise AI feels like the iPhone moment. A demo lands, a room gasps, someone signs a budget, and everyone pictures a clean, magical product that just works.

Here's my prediction. By 2028, most of those investments won't be remembered as the iPhone. They'll be remembered as SAP.

If you were in a boardroom in the late 1990s or 2000s, you know exactly what that means, and your stomach probably just dropped. ERP was going to unify the company, kill the silos, and give leadership one version of the truth. Some of it delivered. A lot of it ran years long, cost multiples of the estimate, and ended in a system nobody quite loved and everyone learned to work around. The software was never the hard part. The company was.

AI is walking into the same story. Not because the technology is weak, but because of what always happens when powerful technology meets a large organization: the innovation becomes operations.

Why ERP is the right rhyme, and the iPhone is the wrong one

The iPhone was a product you bought and used. ERP was a program you implemented. That difference is everything.

A product pays off the moment you unbox it. A program only pays off after you've rebuilt your processes, retrained your people, cleaned your data, and rewired how decisions get made. The technology is maybe a fifth of the work. The other four-fifths is organizational surgery.

Enterprise AI is a program, not a product. The chatbot demo is the product. Getting it to actually change how your claims get processed, your contracts get reviewed, or your supply chain gets planned, safely and at scale and with someone accountable, is an implementation. And implementations rhyme. The technology works. The organization doesn't. That was the real story of ERP, and it's about to be the real story of AI.

This isn't even the first time. ERP, then CRM, then cloud, then the data-warehouse wave: each arrived as innovation and left as governance. Salesforce didn't fix bad sales processes; it exposed them. "Lift and shift" to the cloud didn't cut costs until companies rebuilt what they moved. Every wave starts as a breakthrough and matures into an operating-model change. AI is simply reaching that stage faster than anyone expected.

The specific ways it will rhyme

Budgets balloon and timelines slip. The demo cost a rounding error; the deployment will not. Wiring AI into real workflows means data pipelines, access controls, evaluation, monitoring, and endless edge cases the demo never showed. ERP taught us the pilot is the cheap 10%. AI is teaching the same lesson to anyone paying attention.

A consultant economy appears overnight. ERP built Accenture and Deloitte as we know them. The same thing is happening now: the "AI transformation practice" is being staffed as you read this. The irony your CIO will feel personally: after a decade of selling "disruption," they'll end up hiring the exact roles that made ERP work: enterprise architects, business analysts, data stewards, integration specialists, program managers, change managers. The revolution will be delivered by the people who ran the last one.

Customization eats the promise. The pitch is a general model that does everything. The reality is that your data, your compliance rules, and your processes are unlike anyone else's, so every serious deployment becomes a custom build. The gap between "standard package" and "our special requirements" that made ERP balloon is waiting for AI, in the same place.

The value is in the redesign, not the tool. This is the deepest rhyme. ERP only paid off for companies that used it as a reason to simplify how they actually worked. The ones who bolted it onto their existing mess just automated the mess, expensively. AI is merciless about this. Drop it on a broken process and you get a faster broken process. Bad knowledge management becomes bad RAG. Bad documentation becomes confident hallucination. Bad workflows become automated chaos. The model doesn't fix the rot; it scales it. Most of the return will come from the redesign the AI forces, not the AI itself.

And then it becomes table stakes. The part executives least want to hear. ERP stopped being an advantage the moment everyone had it; it went from competitive edge to the cost of staying in business. Enterprise AI is on the same curve, only faster. The general capability you're paying a premium for today will be a commodity your competitors also have by 2028. The advantage was never the software. It was what only you could do with it.

Where the analogy breaks, to be fair

It isn't a perfect match, and I'd be doing bad analysis if I pretended it was.

AI is cheaper to start with, adopts bottom-up before any big program begins, and iterates in weeks where ERP iterated in years. A team can get real value from an off-the-shelf tool tomorrow, with no eighteen-month rollout. That's genuinely different, and it's good.

But that changes the entry, not the endgame. Easy pilots are exactly what will lull companies into underestimating the enterprise-scale version, which is where the ERP dynamics come roaring back. Cheap to start is not the same as cheap to deploy across a regulated, political, legacy-bound organization.

What to actually do with this

If your AI program is being budgeted like a software purchase, it will fail like an ERP project. Budget it as the organizational change it really is. Assume the model is the cheap part and the change management is the expensive part, and staff for that.

Expect the pilot magic to die on contact with the org, and plan for that death instead of being surprised by it. Pick the processes you're genuinely willing to redesign, not the ones you just want to sprinkle AI onto. And stop treating the general capability as your moat, because it won't be one for long. Your moat is your data, your workflow, and the specific redesign a competitor can't copy.

The companies that came out of the ERP era ahead weren't the ones with the biggest implementation. They were the ones who used a painful technology program as an excuse to become a simpler, sharper business.

So the question worth sitting with, before the next AI budget gets signed: are you buying a product, or signing up for an implementation? Because by 2028, almost everyone will discover it was the second one.

When AI knows everything, what should humans learn?

 

My nephew asked me last week why he should study anything if the machine already knows it. He's twelve. Fair question, and I didn't have a clean answer, so I've been chewing on it.

The model knows the answer. It doesn't know if the answer matters. Ask it a bad question and it hands back something confident, well-written, and useless, and it will never tell you the question was bad. Knowing what to ask, and what isn't worth asking, is now the harder skill. The machine doesn't do that part for you.

Then there's judging what it gives you. AI is wrong often enough that you can't outsource trust to it. If you don't know enough to smell when a number is off or a claim is too clean, you'll ship its mistakes as your own. You still need real knowledge in your head, not to race the model on recall, but to catch it.

That is the part school got backwards. For a century we tested the things AI is now best at: remembering, understanding, applying. Those were never the point. They were just the easy things to grade. The skills we waved at and rarely taught, analyzing, evaluating, creating, are exactly the ones left standing.

And someone still has to own the decision. When the model says lay off the team or change the treatment, it doesn't carry what happens next. A person does. You can't hand that to something that feels nothing when it's wrong.

So the answer to my nephew is not "study less." It's study differently. Learn to ask sharp questions. Go deep enough in something real to know when an answer is garbage. Build taste. And find the nerve to decide and stand behind it. The best use of an AI tutor isn't getting the answer faster, it's one that argues back and makes you think harder.

I told him: learn enough to know when the machine is lying to you. He got it faster than most executives I've met.

The LEGO problem computers weren't supposed to solve

 

Hand a child a picture and a pile of LEGO, and they'll build something close to it. Ask a computer to do the same and you hit a wall that stood for decades.

It sounds trivial. It isn't. Turning a 3D shape into real bricks that snap together, hold their own weight, and don't collapse is a brutal combinatorial problem. Even a handful of bricks can be combined in so many ways that brute force chokes on it. So this sat for years in the pile labelled "computers can't really do this."

That label is coming off. Researchers at Carnegie Mellon built a system called BrickGPT that designs buildable LEGO models from a description. What makes it work isn't raw search. They trained it on over 47,000 brick structures spanning more than 28,000 unique 3D objects, and bolted on something like a physics inspector: it checks gravity, friction, and contact points, and when a section won't stand, it rolls back and redesigns that part. Then they had a robotic arm assemble one of its designs into a real object to prove the thing actually stands up.

Here is why I'd care if I ran a business, and it has nothing to do with LEGO.

Every company keeps a quiet list of things that are "just too hard to automate." Scheduling that one messy operation. Reading those non-standard documents. Planning a build nobody can write down as clean rules. Most of those lists were drawn up years ago and never looked at again. BrickGPT is a reminder that the line between "impossible for computers" and "done last year" moves faster than the list does, especially now that models can reason about real-world constraints instead of just pattern-matching text.

So the useful exercise isn't watching a machine build a LEGO guitar. It's pulling out your own "impossible" list and asking which items got quietly crossed off while you weren't looking.

Reference: https://avalovelace1.github.io/BrickGPT/

Why Shadow IT is increasing

 

Someone on your team is pasting company data into a chatbot right now. Not out of malice. Because it saves them an hour and IT hasn't given them anything better.

That is shadow IT in 2026, and it is growing fast. A few things are driving it.

The approved cloud tools are often too rigid. They are built to be standard, which means they rarely fit one company's actual workflow, so people build their own workaround in whatever is closest to hand. That closest thing is usually Excel. The most common ERP system on earth is still a spreadsheet someone made, because it bends to fit the job when the real system won't.

AI poured fuel on this. An employee opens a personal ChatGPT or Claude tab and is more productive in ten minutes, no ticket, no approval, no wait. When the unofficial option writes the email and cleans the data faster than anything IT handed them, policy loses. It loses every time.

And the gap keeps widening because technology moves faster than a large organization can approve anything. Shadow IT is the bridge people throw across that gap while they wait.

Here is the part leaders get wrong. They treat the tool as the problem and ban it. That just pushes the same behavior somewhere you can't see, where the data still leaks and you've lost the visibility too.

Shadow IT is a symptom. Someone had a job to do and the sanctioned path was slower than the shortcut. Fix that. Find out what the workaround is actually solving, get the safe version to be the fast version, and build a real route for those needs to reach the IT roadmap instead of hiding in a browser tab.

If your shadow IT is growing, your people have already told you your official tools are too slow. They just told you with their behavior instead of a survey.