Three years ago, a fintech company created a model to predict when customers would leave. This model was good. It was correct about ninety-two percent of the time. The company was able to get it up and running in six weeks. The team that made it even won an award inside the company.
The model was able to figure out which customers were going to leave before they did.
By the month, the sales team had stopped using the model completely.
This was not because the model was wrong. The problem was that nobody had thought about what would happen when the sales team got a list of customers who were about to leave on the day that the accounting team finalized the targets for how much the company could spend. The accounting team had decided not to hire any people so they could meet their targets. The sales team needed to keep those customers to meet their targets for how much money the company could make. The model said that the sales team should try to save these customers by offering them discounts and special service. The finance team said that the company could not spend that much money. The sales team asked who would pay for the discounts if they were offered to save a customer that spent two hundred thousand dollars. The finance team did not have an answer.
So, the sales team just ignored the model.
The model was perfect, in a way. It did not work with the rest of the company. The company had problems that the model could not fix.
This is what happens with a lot of intelligence projects.
The Reductionist Illusion
A reductionist thinks that if all the parts are working well, the whole system will work well.
This is usually not true when people are involved.
A company that sells things in stores found this out in a way. They made a computer program that could figure out how much of each thing to keep in stock. It could predict what people would want to buy, which store they would buy it from and what day they would buy it. The program was very good at math.
They used the program. It worked pretty well. They did not have as much extra stock as before, which saved them money. They also did not run out of things much, which made customers happy. This was great.
Then they started using the program in two hundred stores.
After a couple of weeks, the people in charge of the stores started calling. They said the program was telling them to keep a lot of winter boots in stock during the summer. This would cost a lot of money to store them. What really happens is that people look at the shelves when they are in the store. If they do not see what they want, they will not come back later to look for it. The people in charge said that the program only looks at numbers; it does not understand how people think and behave.
The program was actually right about how many winter boots people would want to buy in the fall. It was wrong about how people think and behave during the summer. It only tried to save money on storage without thinking about how people would react.
The company did not need to make the program better. They just needed to add some rules. They told the program that it always had to make sure that people could see the things on the shelves. Now the program had to work with the way things really are.
The company learned that the retail company system had to work with the retail company reality.
The retail company system had to optimize within the retail company reality.
The retail company had to make sure that the retail company system understood how the retail company people behave.
The Org Chart Problem: Where Alignment Breaks
Here is where a simple idea like reductionism meets a problem like organizational gridlock.
Most companies are divided into teams that do things:
- The Sales team is in charge of making money. They look at how much sales they might make and how often they close deals.
- The Product team is in charge of getting people to use their products. They look at how people use the features and how happy they are with the product.
- The Operations team is in charge of making things run smoothly. They look at how much it costs to do each transaction.
- The Data Science team is in charge of making sure the information is correct. They look at how precise and accurate the information’s
Each team does what it is supposed to do. Each team has goals that make sense. Each team will keep trying to do
Then you start using a churn prediction engine.
The engine finds 50 accounts that’re at high risk of leaving. It is 90% correct. Now watch what happens:
The Sales team thinks: I need to give these customers $15,000 in discounts and support to keep them.
The Operations team thinks: That is $300 per customer. It is a lot of money for this group of customers.
The Product team thinks: If we have to spend that money to keep customers, our product is not good enough. I should use our engineers to make features that will keep customers instead.
The Data Science team asks: Why are people not using the prediction engine? It is accurate. I need to make it even better.
The Finance team says: Who is going to pay for the things we need to do to keep these customers?
Nobody is wrong. Everyone did what they were supposed to do. The system still got stuck.
I know a company that had this problem. The churn prediction engine was good. It took 14 months to get it working. Not because it was hard to build. Because the Sales, Product and Finance teams could not agree on who would pay to keep the customers. In those 14 months, customers left the company. The algorithm said they would leave. Nobody did anything. Everyone was doing what they thought was right. The company lost.
Real Failure Mode: The Drift Nobody Caught
Here is what actually breaks systems:
A company used a demand forecasting algorithm. On test data, it was about eighty-nine percent of the time. That is a signal. So, they put it to work.
The month it was right about seventy-six percent of the time. That was expected because data changes and the model adjusts.
By the month, it was right about sixty-four percent of the time. That was worrying.
By the month, it was right only about fifty-three percent of the time. That was a disaster.
What went wrong? Nobody was checking for changes in the data. The idea was that what happened before would happen again. During those six months, the demand forecasting algorithm had three big problems:
- A competitor started selling. That changed how customers behaved
- The company made a change to their product. That changed how users interacted with it
- There were problems with getting products to people. That changed when people bought things
The demand forecasting algorithm was good at being right about old data. Not good at being right in a world that is changing.
Some people say the answer is to make the demand forecasting algorithm better by adding details.
A better way is to build a system that watches the demand forecasting algorithm and checks when real life is different from what the demand forecasting algorithm predicts. The system should automatically update the demand forecasting algorithm. Tell people when things change. It should have a way to catch changes before they affect customers.
This way is harder. It is not about the demand forecasting algorithm it is about the system that watches the demand forecasting algorithm. It is the difference between something that works for a little while and something that works for a long time.
What Systems Thinking Actually Looks Like
It’s not philosophy. It’s asking one hard question after you’ve built something:
“Now that we have this algorithm, what has to be true about how people use it, how it connects to other systems, and what success actually looks like—for this to work in the real world?”
Then you redesign based on the answer.
For a forecast algorithm:
Don’t assume sales will adopt new workflows. Co-design the workflow with them first. Before the algorithm launches. Build monitoring for demand shifts. Define what “accurate enough” means for your economics—one wrong forecast on a $1M order costs you $100k? That changes the accuracy target.
For a churn prediction system:
Assign the owner of the intervention budget before deployment, not after. Define what “act” means: discount? Retention call? Product feature request? Different levels have different owners. Build feedback loops that track what the algorithm predicted vs. what happened. If accuracy degrades, retrain and alert.
For inventory optimization:
Add constraints based on psychology, not just cost. Minimum shelf visibility. Protect high-discovery items. Build fallbacks for when supply chains break. What happens when your suppliers can’t deliver? Your algorithm doesn’t know. Your system needs to.
The Bottom Line
You do not choose between reductionism and systems thinking. You need both reductionism and systems thinking.
To build intelligence, you should use reductionism. To deploy intelligence, you should use systems thinking.
You should optimize the pieces of the intelligence very carefully. The artificial intelligence model should be as accurate as you can possibly make it. However, when you deploy the intelligence, you should never forget that you are putting it into a system that has its own logic. Money goes through budgets. People have incentives that are tied to outcomes. Information travels through channels. Humans make the decision.
If you break the logic of the system, no amount of model accuracy can save you from problems.
For example, that financial technology company that broke their customer churn system did not rebuild the algorithm. Instead, they built a system where the algorithm actually mattered. The company that deployed a forecast engine and watched it get worse over time needed to have monitoring and retraining loops, not training data.
The retail company that had problems with the algorithm did not add features to the algorithm. They added constraints to the algorithm instead.
You should start with reductionism. Finish with systems thinking. This is the difference between a data science project and a business outcome that actually works. Reductionism and systems thinking are both necessary for success.
Posts
No comments:
Post a Comment