TechnicalDebt
Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise."
— Ward Cunningham, 1992
A “debt” means that you traded acquiring something now for a long-term financial burden. This burden is not just about repaying principal but “interest” as well. Most of the time interest is not simple but compound. It means that, even if you pay your debt timely, you’ll pay more than you took, and if you don’t, your debt will keep increasing. And if you ignore a debt long enough, it will become unpayable and you’ll “bankrupt”.
TechnicalDebt is a metaphor referring to the consequences of poor system design, architecture and/or software development process within codebase. TechnicalDebt includes those internal things that a DevTeam chooses not to do now, but which will impede future development and/or maintenance if left undone. TechnicalDebt doesn’t include deferred functionality, except when corner cases are left for future.
TechnicalDebt happens when you “buy” more software features than you have resources (Dev team’s time).
Despite the populist feelings against the credit system, epitomized in The Merchant of Venice by Shakespeare’s character Shylock, debt is a good thing. In any vibrant and growing society possibility of credit drive business, innovation and trade. With credit you can buy a machine to start producing something and then pay for the machine.
Similarly in software, you need to survive first to grow in future. In most of the cases you cannot cease flow of immediate money in favor of future money. TechnicalDebt is inevitable in any vibrant, evolving and growing software. Manage TechnicalDebt as you cannot make it ZERO.
Kinds of TechnicalDebt
TechnicalDebt can be classified on two criteria:
As you might have guesses by now, only intentional debt is good debt. If code has unintentional debt, you need to introduce technical talent in your team and/or educate your stakeholders.
Internal: All TechnicalDebts are not is self-incurred. Business pressure will force you to take short cuts, putting you in debt. If you do not start paying back soon and regularly, project will be in mess sooner than you think.
External: Today, you got a call from Network Security team and they want to apply a patch to all boxes to plug a recently discovered security hole. All of sudden, you are in debt, through no fault of yours.
Causes
Common causes of TechnicalDebt include:
Business pressures: Business considers getting something released sooner, even before all of the non-functional changes are complete, it builds up TechnicalDebt comprising of those internal uncompleted changes.
Lack of process or understanding, where businesses are oblivious to the concept of TechnicalDebt, and make decisions without considering the implications.
Lack of Technical expertise
Lack of knowledge, when the DevTeam simply doesn't know how to create an elegant design or write graceful code.
Lack of building loosely coupled components, where functions are not modular, the software is not flexible enough to adapt to changes in business needs.
Lack of alignment to standards, where industry standards, frameworks, technologies are ignored. Eventually, integration with standards will be implemented, doing sooner will cost less (similar to 'delayed refactoring').
Lack of coding guidelines, where each member of DevTeam is following its own coding guidelines, which will make integration and automatic debugging nearly impossible.
Lack of automation
Lack of comprehensive test suite, which encourages quick and risky band-aids to fix bugs.
Lack of automated code generation for boiler plate
Lack of continuous integration
Lack of automated deployment
Lack of automated code analysis
Lack of documentation, where code is created without necessary supporting documentation. That work to create the supporting documentation represents a debt that must be paid.
Lack of collaboration, where knowledge isn't shared around the organization and business efficiency suffers, or Dev Team’s junior members are not properly mentored
Parallel development at the same time on two or more branches can cause the buildup of TechnicalDebt because of the work that will eventually be required to merge the changes into a single source base. The more changes that are done in isolation, the more debt that is piled up.
Delayed refactoring – As the requirements for a project evolve, it may become clear that parts of the code have become unwieldy and must be refactored in order to support future requirements. The longer that refactoring is delayed, and the more code is written to use the current form, the more debt that piles up that must be paid at the time the refactoring is finally done.
Top down project execution methodologies
Lack of product vision
Fast pace changes
Technological
Business environment
Not paying debt regularly
Most of the time, TechnicalDebt is created to achieve short run goals sacrificing, long term solution. But remember, you cannot have a long term if you do not survive in short term. But also do not forget – What you sow is what you reap.
BigUpfrontDesign
In the first glance it seems very plausible that BigUpfrontDesign will resolve the issue of TechnicalDebt. You must avoid fallacy of - we are designing the product from very starting, so we will be avoiding lot of challenges which might come on the way during long course of development. Ask few questions:
Is requirement space frozen?
Are technological innovations halted?
Is business environment static?
If your answer is “NO” for any one of above question, BigUpfrontDesign will results in bigger TechnicalDebt.
Puxadinho
Puxadinho is a Portuguese word which means an extension of a construction done without any expert supervision, poor materials and generally illegally.
Pauxadinho is not restricted to construction only.
You can encounter pauxadinho in software everywhere. Pauxadinho is result of excessive TechnicalDebt. Every one suffers.
Affordable Care web sites were prime examples of Pauxadinho across the states and federal agencies. Few of the examples you can follow here:
Various US agencies had to spend great sum of money and time to rectify the various web sites with great loss to reputation.
But, you should also not ignore Reid Hoffman’s (Linkedin creator) advice – if you are not embarrassed by first version of your product, you’ve launched too late.
Festina lente
The Roman historian Suetonius, in De vita Caesarum, tells that first Roman emperor, Augustus deplored rashness in military commanders, and thus “Festina lente” was one of his favorite sayings.
Gold coins minted for Augustus bore images of a crab and a butterfly to attempt an emblem for the adage.
The constructive intent of the phrase is that activities should be performed with a proper balance of urgency and diligence. If tasks are overly rushed, mistakes are made and good long-term results are not achieved. Work is best done in a state of flow in which one is fully engaged by the task and there is no sense of time passing.
Broken Window
The broken windows theory is a criminological theory of the norm-setting and signaling effect of urban disorder and vandalism on additional crime and anti-social behavior. The theory states that maintaining and monitoring urban environments to prevent small crimes such as vandalism, public drinking and toll-jumping helps to create an atmosphere of order and lawfulness, thereby preventing more serious crimes from happening.
James Q. Wilson and George L. Kelling first introduced the broken windows theory in an article titled Broken Windows, in the March 1982 The Atlantic Monthly. The title comes from the following example:
Consider a building with a few broken windows. If the windows are not repaired, the tendency is for vandals to break a few more windows. Eventually, they may even break into the building, and if it's unoccupied, perhaps become squatters or light fires inside.
Or consider a pavement. Some litter accumulates. Soon, more litter accumulates. Eventually, people even start leaving bags of refuse from take-out restaurants there or even break into cars.
Replace building with code. A code base with spaghetti will get more. With not so good code everyone wants to hack it and get the work done. Who cares about long term?
First Law of Programming
Lowering quality lengthens development time.
Quality software takes the least amount of time to develop. If your code are simple as possible, complete tests and a design that fits just right (no over engineering please!), additions and changes happen in the fastest possible way because the impact is lowest. Consequently, if you hack something out, the more you hack the slower you go because the cost of addition or change grows with each line of code.
Adding features vs writing from scratch
In a well-written system, the cost of adding a new feature should be less than the writing from scratch in most of the cases.
Certainly previous statement is function of reusability of existing code. Even if there is very little code reusability, the patterns of the existing system can be reused to reduce the number of design decisions.
Of course, TechnicalDebt will increase if you do not refactor on continual basis.
Legacy Code
You might have thought of COBOL code is Legacy code. I have different take on it. Code which does not have well written code is Legacy code. You will find it difficult to refactor this code. You find it difficult to add features to this code.
If you refactor this this code, you have no idea what else is broken. You need tests to verify. To test it, you need to refactor. This is Catch 22 of fixing a Legacy code.
TechnicalDebt Quadrant
In response to Uncle Bob’s post saying a mess is not a debt, Martin Fowler came up with idea of TechnicalDebt Quadrant. TechnicalDebt Quadrant is an attempt to distinguish between irresponsible, incompetency and prudency for a team.
The original quadrant lists two quadrants and one half.
With my experience with various teams across industry and geographies, I overlay my ideas over original. This overlay represents teams various state of working under numerous environmental conditions.
TechnicalDebt Matrices
You can't manage what you don't measure. It is an old management adage that is true beyond business situations. Unless you measure something you don't know if it is getting better or worse. You can't manage for betterment if you don't measure to view what is getting better and what isn't.
Before jumping to matrices, I suggest to understand your project in little different way:
Is project in the active development phase?
Is project in the maintenance phase?
If project is in the active development phase, let’s assume you are using Scrum while if your project is in maintenance phase, you are using KanBan.
Let’s start with a project which is in active development.
Are you marking your user stories – whether story is TechnicalDebt payment one or not? Are you assigning TechnicalDebt Payment points with story points to stories? If your answer is NO. Tighten your belt, accept Red pill.
Start ASAP and with next Sprint Planning and Product Backlog Grooming sessions start assigning TechnicalDebt Payment points.
There are four attributes which should be assigned to a story w.r.t TechnicalDebt:
Is pure TechnicalDebt story: This attribute has binary value – Yes or No. YES signifies that this story is not adding any feature to the increment. NO means that apart from TechnicalDebt payment, this story will also be adding feature to the product. This attribute to be filled up during one of the Product Backlog grooming session.
TechnicalDebt points to be paid: This number represents team’s best intention estimate of TechnicalDebt points to be paid if this story to be executed. The value of this attribute to be estimated during the Product Backlog sessions. At the Sprint Planning meeting the final value must be assigned.
TechnicalDebt points paid: After completion of all tasks associated with this story and just before declaring story DONE, assign value to this attribute. This attribute’s value signifies TechnicalDebt paid off by executing this story.
TechnicalDebt points collected: After completion of all tasks associated with this story and just before declaring story DONE, assign value to this attribute. This attribute’s value signifies prudency of team in acquiring new TechnicalDebt.
In a mature and healthy team, “Is pure TechnicalDebt story” value should be “No”. Why to pay debt upfront if you are not getting any benefit. But in some cases like where synergies or dependencies among stories warrant, you will be having value “Yes” for this attribute.
In ideal world value for remaining three attributes should be ZERO, but sadly utopia does not exist.
The value of various TechnicalDebt points should be estimated using the same method, that you are using to assign Story Points: T-shirt sizes, Fibonacci sequence (1, 2, 3, 5, 8, 13, …), power of 2 (2, 4, 8, 16, 32, …), etc.
Now it is time to see the trend in TechnicalDebt. Plot TechnicalDebt points against time (per Sprint). You will be drawing three lines.
Total TechnicalDebt points in the Product Backlog: This trend line will provide glimpse of technical health of your product. In a technically healthy product, this line should be flat over a time unless some major technical or functional or non-functional changes occur in the product. You may also notice some spike in this trend line with introduction of new blood in the team.
TechnicalDebt points collected in a sprint: The trend line should be downward. You may also notice some spikes in this trend which must be avoided but due to business pressure or other reasons may not be avoidable.
TechnicalDebt points paid off in a sprint: This trend line depicts team’s focus on Technical heath of the product. This line should be understood with “Total TechnicalDebt points in Product Backlog” and “TechnicalDebt points collected in a sprint” lines.
·
Now it is time to look into a project which is in maintenance phase.
Kanban is becoming popular to manage support projects which have entered into the maintenance phase. In this scenario there are two possibilities:
In first case where maintenance team has historical information about TechnicalDebt of the project can continue the work from where development team left.
In both cases whether maintenance team has TechnicalDebt records or not, there will be lot of adhoc tickets to resolve issues arising daily basis and some enhancement requests.
Before bringing a story Task Board, ensure that TechnicalDebt related attributes have proper values. This will help you to build trend lines over a time period.
Techniques to prevent TechnicalDebt
It is always a good idea to take preventive measure to ensure that bare minimum TechnicalDebt is acquired by team over the product’s life cycle.
Emphasize in Collaborative Design techniques
Whiteboard sessions on design topics
Leverage collective experience of the team
Wisdom of the crowd
Fair mixture of subject-matter experts and novices
Utilize Emergent Architecture
Evolution over revolution
Gradual changes
Controlled Generic-ness
Avoid YAGNI (You Ain't Gonna Need It)
Strong Definition of “DONE”
Story
Sprint
Release
Release to Prod
“Done should cover “Completion Criteria” as well as “Acceptance Criteria”.
Automation
Boiler plate code generation
Deployment
Continuous build
Testing
Test Driven Development
Refactor on continual basis
Cultural reinforcement
Without unit test cases, no code is ready to be called “DONE”
We own this product end to end
System thinking
No free lunches. Pay now or later (bigger amount)
We will not allow TechnicalDebt until unless tactical or strategic business imperative demands so.
Aware business
Business should be made aware that short term quick fixes result in long term pains (TechnicalDebt)
Business should be made aware of benefit and necessity of paying TechnicalDebt in timely manner.
TechnicalDebt Management
Since you have already accepted the Red pill, be ready for the consequences.
Accept the truth, No product of a reasonable size can be TechnicalDebt free at all time. Product will be in TechnicalDebt.
Try to prevent accepting new TechnicalDebt.
If you fail in prevention, resist with all of your claws.
There are two popular ways to pay TechnicalDebt:
Because of the Red pill, third option of not paying TechnicalDebt is out of consideration.
Let’s devise a plan to manage TechnicalDebt.
In Product Backlog mark technical stories explicitly. Most of the time Technical stories help you to pay TechnicalDebt or prevent you to accumulate new TechnicalDebt.
In each sprint planning meeting pick up stories which will pay some TechnicalDebt apart from Functional stories.
If you are using cards for user stories, use TechnicalDebt paying stories on obnoxiously colored cards. We all love beauty. It will motivate you to write clean code.
Start assigning TechnicalDebt points to each story during Product Backlog and Sprint Planning sessions.
During Product Backlog Grooming sessions:
During Retro and Product Backlog Grooming sessions:
Prepare high level TechnicalDebt payment plan and get buy in from Stakeholders.
Now, it’s time to pay back.
Repeat the cycle.
Reference
Posts
