Thursday, November 21, 2019

Waste in Software Development and Maintenance – Part 2: Maintenance & Support

This is the second and final part of the “Waste in Software Development and Maintenance. In this post, I will focus on various wastes in software maintenance & support.

Here are the few of the wastes in software development:
  1. Relearning
  2. Partially Done Work
  3. Handoff
  4. Task switching
  5. Delay/Waiting
  6. Defects
  7. Red-tape
  8. Ineffective Communication
  9. Unnecessary Complex Solution
  10. Manual Execution of Repetitive Work
  11. Multiple Stakeholders
  12. Ineffective Monitoring & Lack of Unified Dashboard
  13. The Manual Pull of Logs/Traces

  1. Relearning: Here I want to run through at least two scenarios – Waterfall setup (distinct Dev and Ops teams) and Agile/DevOps setup (Dev team transforms into Ops team).
a.      Waterfall setup: Dev has completed the development and the system is in production. It is now Ops' responsibility to learn intricacies of the system. If the Dev team was kind enough, they might have documentation which is certainly not able to keep pace with the system's development speed. Dev team might organize a couple of knowledge share (sic!) sessions for the Ops team. Only God knows how much knowledge is gained by the Ops team in these sessions. For all practical purposes, the Ops team needs to invest a great amount of time and resources to learn about the system. What’s a waste!
b.      Agile/DevOps setup: during the frenzy of development, there were a whole lot of developers; Ops team was busy with existing systems which are already in the production. Now the new system is in the production, development is slowed down; the Dev team is ramping down. Even with the best intentions, only partial knowledge is retained by the Ops team

To tackle this type of waste, engage the Ops team with Dev teams early. Once the system is the production, embed Dev team members in the Ops team. If possible retain some of the dev team members in the ops team.

Wastes due to Partially Done Work, Handoff, Task Switching, Delay/Waiting, Defects, Red-tape, Ineffective Communication, Unnecessary Complex Solution, and Manual Execution of Repetitive Tasks during Maintenance & Support are similar to the Development phase.

  1. Multiple Stakeholders: Once the system is in production, multiple stakeholders use the system and each has slightly different expectations from the system. This leads to difficulty in prioritization of work, and backlog refinement. In additional political weight of stakeholders drives the system toward higher entropy. This waste is difficult to handle due to its very nature. To tackle this type of waste, ensure that the system has a super owner. 
  2. Ineffective Monitoring and Lack of Unified Dashboard: All systems require monitoring from various perspectives (e.g. performance, security, usage pattern, etc.). The insufficient infrastructure of monitoring and lack of inbuilt hooks in the systems to monitor lead to a great amount of waste. To tackle this type of waste development team must have a designed system for serviceability (remember DFX) and PRD must have sufficient monitoring capability and a unified dashboard. A unified dashboard will ensure that Ops teams don’t waste time and resources to shuffle around various monitoring tools. 
  3. Manual Pull of Logs/Traces: Pulling out the logs and trace in case of any incident is the normal course of action for any Ops team. If this task is not automated, it requires a significant amount of resources and skill. To reduce this type of cost, automate the pull of logs/trace. Also, automate the routine analysis of logs and trace, so the Ops team can focus on deeper issues in case of an incident.
 In this two-part series on the waste in software development and maintenance, I have listed various wastes and ways to reduce them. Let me know if I have missed anything.

Wednesday, November 20, 2019

Wastes in Software Development and Maintenance – Part 1: Development

There is a huge amount of literature on Lean Software. In this literature distinction between development and maintenance is not recognized. Yes, I know some of you will argue that with iterative delivery of software there is no difference in software development and maintenance. Allow me to emphasize the difference between software development and maintenance. During the development phase, delivery is focusing on rolling out newer and shiny features. With the delivery of each feature, business value delivery gets a jump. There might be some defect fixing but the focus is the delivery of new features. In software maintenance focus is on maintaining the steady-state or at the most slight rise in business value delivery with each enhancement and defect fix.  

Apart from the non-existent difference in development and maintenance, the literature of Lean Software development is almost 10 to 20 years old. During this period we have learned a lot especially with the rise of Agile.

Keeping this in the background, it is a good time to list different wastes and how to avoid them during software development and maintenance.

In this part of the series, I am focusing on development.

Here are the few of the wastes in software development:

  1. Gold Plating
  2. Relearning
  3. Partial Done Work
  4. Handoff
  5. Task switching
  6. Delay/Waiting
  7. Defects
  8. Red-tape
  9. Ineffective Communication
  10. Mismanaging Backlog
  11. Unnecessary Complex Solution
  12. Manual Execution of Repetitive Tasks

1. Gold Plating: Pick up any software system; you will notice 20% of its features are used 80% of the time. Remaining 80% features are hardly ever touched by the system’s users. Do we need to devote time and resources to the minimally used features? The counter-argument can be – because of these extra features our system is purchased even if those vanity features are not used. I agree this might be a valid case in most of the systems. There the question is - when to make a call and say it is enough, we are entering into the diminishing rate of return. Employ Lean Startup, and Minimum Marketable Feature (MMF) to find out how many and how deep vanity features are adding value.

2. Relearning: Every day in software development is a day of discovery and invention, we don’t write the same code every day. Even though we use the existing foundation blocks repeatedly though their fitment and combination in the system vary. For example, design patterns, coding practices, engineering practices, management-related policies, and processes are well documented across software development discipline. Why not relearn them once and utilize them repeatedly. No need to reinvent the wheel again and again

Word of caution: Too much emphasis on the elimination of this waste lead to strict standardization which leads to stagnation and scuttled innovation. Follow the standard becomes the normal way of life rather than continuous improvement (kaizen) and occasional disruptors (Kaikaku).

3. Partial Work Done: Partial work done: We all want to work on new and shiny. Most of us do not like to devote efforts to something which will not be recognized or some time care. This leads to partial work done. With time this style of working leads to technical debt and the system becomes fragile. To tackle this waste, organizations not only recognize the value of the delivery of features also the proper closure of work started. In addition, regularly refactor the systems to keep it robust. Insist on making a system antifragile.

4. Handoff: Team-One is a Java team focusing on development and Team-Onyx is made of database experts, they will maintain the schema and data of the application. This structure will lead to the frequent handoff of the work. With the cross-function team, handoff can be reduced.

5. Task Switching: I am Java expert, I don’t know SQL, need someone to write SQL query. Another scenario – I am facing some challenges in the current task, let’s start the task two. I will do the multi-tasking. We need to understand, humans have evolved to perform a task at a time. Multi-tasking looks attractive, essentially it piles up the work in progress and a little difficult task way longer to complete (if it ever gets completed). To reduce this waste there are several strategies. Work In Progress (WIP) limit, T-shaped skill set, Swarming, and cross-functional teams are a few of the ways to tackle this waste.

6. Delay/Waiting: Database team’s SLA is two days. Though the development team needs simple re-indexing to run an experiment on the performance of their new code which requires a few minutes from DBAs, the dev team needs to wait for a while. This type of waste can't be avoided completely because keeping all experts in the dev team is impossible also sometimes it is economical to keeps experts in one team. Though this type of waste can be reduced significantly by T-shaped skill set, cross-functional teams, calibration of shared/expert services, and calibration of SLAs.

7. Defects: We all know defects are a costly waste. They become costlier if discovered late in SDLC (yes, even in Agile, SDLC exists with overlapping micro-falls and with infinite loops). To reduce this waste not only we need to catch the defects early also do something which reduces the occurrence of defects. Test first (TDD, BDD, ATDD), Shift Left, test automation, code scanning & policy implementation in IDE, code review, continuous integration are great ways to find out the defects early and even scuttle their occurrence.

8. Red-tape: In most of the organizations (mostly in software consuming organizations), you will find the bureaucratic rules which have lost their relevance with the evolving context of software development. Recently, I encountered an interesting example of bureaucratic lethargy. In this organization, there is a dedicated UAT testing team. Members of the UAT testing team are not from the business but part of IT which keeps a close relationship with the business. UAT testing team needs to bless any and every functionality before it gets deployed into production. UAT testing team is a big bottleneck. How this team came into existence? Once in the past, this organization failed an audit. To pass the audit UAT testing team was created as a quick fix and got stuck. Instead of looking into the root cause, a quick fix becomes a permanent bottleneck. To handle this type of waste, you might need to do something similar to a code refactor to reduce bureaucratic debt.

9. Ineffective Communication: Software development is a social activity, unlike manufacturing. In manufacturing work is determined by machines’ operation, humans need to adjust their working as per the demands imposed by the machine. In software development, even with the highest level of automation, logic is developed by humans. This logic development and interaction of components (which contain smaller pieces of the puzzle to be solved by the solution) is performed by humans as a social activity. This requires very effective communication. Face to face communication is the best but due to various reasons co-location may not be possible. Any ineffective communication will cost huge. Never forget Conway’s law. To reduce this waste keep multiple channels of communication open, repeat the communication, contextualize the communication, remember cultural differences (the USA has shoulder on the side of the road but in rest of the world it is a body part), get the confirmation from audience that they have understood the meaning of the communication. In addition, try to put pictures in written communication to convey a message.

10. Mismanaging Backlog: In its simplistic form, Backlog is a list of requirements. In real life, Backlog is more than a list. A backlog is a container that enables the realization of business value if handled properly. A backlog requires continuous refinement – prioritization as per business value & feasibility and split & combine of Backlog Items. In this process, the Product Owner needs to engage customers/consumers, technical teams, business stakeholders, and any other stakeholders as required. If the Backlog is not managed properly, lower business value requirements get delivered into the market over higher business value features. To reduce this waste, invest in continuous refinement of Backlog, story mapping, INVESTable stories, decide as late as possible, and engaging various stakeholders.

11. Unnecessary Complex Solution: Complex solution: We as an engineer want to make a perfect solution. We forget perfection is not achievable. Race for perfection leads to an overly complex solution that is costly to develop and costly to manage. To reduce this waste, focus on fit for the purpose, emergent architecture, architecture review, refactoring, insist on the simple solution, decide as late as possible (set based architecture & design), DFX (Design for XXX), etc.

12. Manual Execution of Repetitive Tasks: Manual work is costly due to employee cost and its proneness to error. Let software professionals focus on the tasks which require their ingenuity. To reduce this waste, I use a simple rule – if a task getting repeated at least three times must be automated.

Let’s take a pause here. I will discuss the various wastes in Software Maintenance & Support in part two of this post.

Tuesday, November 19, 2019

Predictions of Next decade (2020 -2029) – 2020

1.     Agile transformation & Business Agility will continue to drive great profit for consulting businesses.
2.     AI will be pervasive.
3.     Algorithmic driven operations in the service industry will be the norm.
4.     Digital currencies will start challenging the national currencies.
5.     3D printing will be able to a decent speed of product but still be vanity.
6.     Fragmentation of the internet. There will be country/region-specific walls.
7.     Cloud tri-poly or quad-poly will continue.
8.     Social media will make people lonelier and rip the democratic and social fabric.
9.     China-based vs USA-based tech organization will open new business war theaters.
10.  Industry-specific cloud offerings will be the norm.
11.  Autonomous vehicles will start shaking the foundation of the laws and insurance industry.
12.  Voice and gesture-based interfaces will be the norm.
13.   The quantum computing race will continue.

Monday, November 18, 2019

Top Technology Trends of 2020

1.     March of Agile will continue
2.     Business Agility will be the hot topic in the Agile transformation circles.
3.     AI and UX professionals will be in great demand.
4.     Semantic expansion of Agile and DevOps will continue with greater strength.
5.     War of scaled Agile framework will be more or less settle in favor of SAFe in traditional businesses (banks, insurance, retail, manufacturing, transport, natural resource extraction, etc.)
6.     Virtual reality and augmented reality will continue its market gain in the enterprise training market.
7.     The merger of devices will continue.
8.     IoT will penetrate further in the home automation market.
9.     Privacy issues will continue to haunt in social media and home automation.
10.  The autonomous vehicles still be struggling to be street legal.
11.  Various options for AI as a service will emerge.
12.  The online non-human medical practitioners will register their presence.   
13.  NLP will scale new height.