Extreme programming practices that mitigate Technical Debt

Technical debt, first coined by Ward Cunningham(TD) [1, 2], can be described as the dichotomy between the best design that takes longer to implement and a quick design and implementation for short-term gain. The technical compromise between these two designs and extra development effort needed to make changes to the latter software system represents debt.

1.   What is Technical Debt?

Technical debt, first coined by Ward Cunningham(TD) [1, 2], can be described as the dichotomy between the best design that takes longer to implement and a quick design and implementation for short-term gain. The technical compromise between these two designs and extra development effort needed to make changes to the latter software system represents debt.

A popular metaphor is that of monetary debt. If TD is not repaid as soon as possible, “interest” is incurred — this relates to the effort needed to implement changes in the future. When TD is not addressed and systems are patched with functional and bug fixes, it leads to an increase of software entropy [3, 4], a condition that deteriorates over time. In some situations, when creating a proof-of-concept (POC), it is acceptable to incur some degree of TD. This, however, is known at design time and should be addressed when the POC evolves into an official project.

Another symptom of TD can be compared to the law of unintended consequences. More often than not, poorly designed systems have longer release cycles and have higher resistance to change. This is due to the fact that when necessary code changes are made to one part of the system, unintended changes occur and have to be addressed. This increases the time of the delivery cycle and drives the cost of change higher, often leading to more pressure, which in turn creates more TD.

Some causes of TD:

  1. Last minute emergency additions or omissions of functionality;
  2. Incompetent technical leadership and bad systems architecture;
  3. Efforts to minimize startup capital;
  4. Outsourced software engineering;
  5. Ignoring industry standards;
  6. Building tightly coupled code with modularity Loosely coupled code is easy to modify or change; and
  7. Non-existent test suite, continuous integration and deployment

 

TD manifests itself in the following examples:

  1. Countless branches of same code base, often for single customer releases;
  2. Database SQL stored procedure spanning a 1000 lines of code (LOC);
  3. Continuous version releases to fix bugs;
  4. Frequent patching of a legacy system; and
  5. Very large methods within

Figure 1: The author’s depiction of technical debt

2.    What is extreme programming?

Kent Beck [5] is widely considered as the creator of the XP software development methodology. XP is a frontrunner to many of today’s more modern software development methodologies. It is a lightweight, practical methodology with a focus on software quality and responsiveness to stakeholder requirements and change.

Intensive requirement analysis and extensive documentation are omitted from the process. Teams are small and focused, with a heavy emphasis placed on simplicity and short development cycles. User stories are a core doctrine of XP. Projects are initiated by the final users creating user stories describing the behaviour and functionality of the software system. Before any coding begins, functional testing of the requirements is conducted, with automated testing throughout the lifecycle of the project. Code is constantly refactored to reduce complexity, drive efficiency and adhere to standards. The result is extensible and maintainable code.

Good relationships between software engineers and stakeholders are also considered a cornerstone of XP as defined in the XP value system: communication, simplicity, feedback, courage and respect.

A software engineering team does not necessarily follow every XP practice. Self-organizing teams use and follow the XP practices that suit them at a point in time. As the software engineering team grows in maturity, so more XP practices are incorporated or omitted.

Given that most software systems are in a state of flux, XP adapts with the software system without being a technical barrier. XP manages this by having both technical and management practices. There are several XP practices, but for the purposes of this blog, the primary practices of XP are broken down in Table 1.

Table 1: Primary extreme programming practices

2.1  Modularity violations as a cause of technical debt

When code follows a good design and strictly adheres to standards such as the SOLID principals [6], changes to one module do not affect another module. A module can be described as a “unit whose structural elements are powerfully connected among themselves and relatively weakly connected to elements in other units.” [7]. This inherently creates an orthogonal design.

“Orthogonal design is the union of two principles, cohesion and coupling [8]”. To be more specific, loosely coupled code creates high cohesion, which makes responding to changes very easy and predictable. Positive side-effects of orthogonal design lead to clear and logical structure of relationships and create significant reuse.

Modularity violations occur when changes to one module unexpectedly induces changes to another module. This is not orthogonal design and a key indicator of TD. Changes to the code are not easily made and time estimation becomes inaccurate.

These violations must be identified and rectified by relentless refactoring, a cornerstone practice of XP. Strict and comprehensive coding standards must underpin the refactoring efforts of code, another important XP practice. The two XP practices complement one another and when used together lead to a direct and significant decrease in TD.

In the research paper Comparing Four Approaches for Technical Debt Identification, substantive findings are made by Zazworka et al. A very high correlation exists between Modularity violations and change prone classes, so much as 85% as shown in Table 2 below.

Table 2: Correlation between Modularity Violations and change prone classes [9]

2.2  Technical debt estimation variance

An integral function of software engineering is estimate development time for maintenance and new components. TD can have a significant impact on such estimations. Variances in estimation are depicted by the Cone of Uncertainty. Instead of being certain of the estimations given, TD creates increased uncertainty at the later stages of the lifecycle, where estimates should be more accurate (shown in Figure 2). This negatively impacts cost, productivity and project schedules.

Figure 2: Adapted Cone of Uncertainty impacted by technical debt [10]

Deadlines frequently being missed and so-called unforeseen changes to the code during the development cycle are indicative of time estimation variance due to TD. Accurately predicting development duration is not an easy task, TD makes this exponentially more difficult. Most estimates are given at the start of the phase, when TD is encountered estimates become inaccurate due to the extra work effort involved.

2.3  Measurement of Technical Debt

In the research recorded in Measuring Architectural Technical Debt [11], Kuznetcov defines TD as follows:

[TD] Principal – cost of eliminating TD (Code Debt, Design Debt etc.).

[TD] Interest probability – the probability that the type of TD will have an impact on the system (likelihood of imminent refactoring).

[TD] Interest amount – the extra cost incurred for addressing the TD (cost of modifying a module earmarked for refactoring as opposed to the cost of modifying after refactoring).

TD : { TDprincipal, TDinterest }

           man hours           work incurred           loss of productivity

TD can be expressed as an array where TDprincipal is independent of TDinterest. In keeping with the debt metaphor, the principal debt can be higher that the interest incurred or vice versa.

Various approaches are available to calculate the TD principal. Curtis et al. describes the TD principal value as a function of must-fix problems, the time required to fix, and the cost of the fix. Source code analysis provides actual counts and values calculated against input variables and the code itself. In the code, structural problems, modularity violations, LOC and duplication (to name but a few) all provide input for this analysis. TD can then be expressed as a product of this source code analysis. The principal amount of TD can be calculated using the high-level formula:

TDprincipal = Nmust-fix issues x Ttime required x Ccost incurred

2.4  Technical debt mapped to XP practices

All software systems develop some pain points over time. These pain points are based on the ISO 9126 [12] standard. (This standard has been revised by the ISO/IEC 25010 [13] standard.) A generic indicator of software quality can be defined by six characteristics: functionality, reliability, usability, efficiency, maintainability, portability [12].

These points are dealt with as they arise, and they are also indicators of TD. In Figure 3,  the author adapted the original diagram by Zazworka et al. [14] to reflect a mapping with the relevant mitigating XP practices.

By analysing the characteristics of each XP practice and variants of TD, a mapping can be made of the XP practices that effectively reduce the relevant form of TD.

Figure 3: Adapted Technical Debt landscape [14]

2.4.1  Coding standards – Directly influence the actual structure, composition and interaction of a software system at its lowest level. This can include guidelines and rules that the software engineer has to follow when writing code. This important practice touches on every part of the design, writing and testing of code.

2.4.2  Continuous integration – Facilitates in the feedback of system health and interaction in a continuous and predictable manner. As code is written and checked into version control, an extensive process of compiling, testing and deploying is started. Feedback is provided from all these sub-processes. Processes fail fast and provide immediate information.

2.4.3  Incremental design – Allows for the evolution and advancement of the software system in small manageable and measurable increments. Small releases are easy to scrutinize for TD. Also, small frequent releases are highly productive and stakeholders perceive a growing healthy software system.

2.4.4  Refactoring – Allows for the evolution and advancement of the software system on a code level and keeps the system at an optimal level of engineering excellence. If the code is frequently refactored, it becomes familiar to the software engineering team and TD is dealt with proactively.

2.4.5  Test-first programming – Software defects are kept at a minimum. The addition of new functionality is made simpler and more visible due to observable and trackable test results. As all the tests pass, the software system’s integrity is maintained and confirmed. As the system grows, the test suite grows too.

3.      Findings

Research indicates that there are various approaches to measure and indicate the extent of TD. This applies to both TDprincipal and TDinterest as described in Section 2.3. Only once there is a manifestation of a standard approach, can tools like SonarQube [15] come to fruition and be more scientific [11]. The SQALE[16] method has been widely adopted as a standard way of measuring and expressing TD. The adoption allows for increased feedback and evolution of the method.

Apart from the common causes of TD discussed in this paper, disconnect between technical staff and management is a major cause of project overrun and subsequently TD. This is exacerbated by the lack of further learning by middle management. Middle management also influences technology decisions and lack of understanding technology leads to an increase in TD.

When XP is implemented effective mitigation of TD occurs naturally. XP has had a significant impact on software engineering in the last decade. CI, Refactoring and Test-first programming have been instrumental in this, especially in the management of TD [17].

Each practice addresses one or more forms of TD [18]. Coding standards combined with refactoring and static code analysis directly increases code quality. Refactoring also aids in the writing of unit tests. Further, if done correctly, code will always be in a better state than before refactoring took place. Every new feature is built off a better base than the previous feature — this is true continued improvement. CI reduces the cost of integration and provides continuous feedback, and mitigates operational debt. Incremental design allows for the focus to be on functionality needed now, small increments and modularity counter design debt. Effective use of Test-first programming dramatically reduces defects and increases code quality, this mitigates code and test debt [19, 20].

In previous research [21] conclusive empirical evidence was provided in the written responses of interviewed software engineers. Applying a points system to the ranked XP practices provides a clear hierarchy of effective- ness. The Coding Standards XP practice is ranked first by a substantial margin. Refactoring and incremental design follows in rank respectively. Lastly, test-first Programming and continuous integration occupies the last two positions respectively.

Table 3: XP practices ranking

4.      Conclusion

The five XP practices are intrinsically linked in their ability and effectiveness in the mitigating of TD. The effect of following all these practices drastically reduces levels of TD. The power of these XP [22] practices lie in fact that they complement and follow on from each other. Most modern agile methodologies heavily incorporate these five practices, a testament to their importance.

In its simplest form: “Technical debt is the difference between what was promised and what was actually delivered” [23]. This difference can not only be very difficult to measure, but also difficult to find the root cause for the difference. TD will always be incurred at varying degrees of cost. An argument must be made that a software system is born from the code itself, not the idea of a software system. The code captures not only the core business idea or solution to the problem, but the reasoning and cognitive ability of a group of people, an amalgamation of intellect. Code must be guarded, maintained, optimized and treated with a level of respect. The empirical evidence presented substantiates that the five XP practices aid in this.

With time the measurement of TD and tools reporting on TD will increase in sophistication. There is no single solution to measure, report and reduce TD but rather a combination of measures and practices. The astonishing amount of research papers and commercial literature on the subject is proof of this.

Adapting to change is crucial to any software system evolving over time. Empirical evidence shows that this evolution can have negative consequences, but the practices we use to measure and mitigate the impact are also evolving in their effectiveness.

REFERENCES

[1]  W.Cunningham. “The WyCash portfolio management system.” ACM SIGPLAN OOPS Messenger , vol. 4, pp. 29–30, 1993.

[2]  Debt Metaphor – Ward Cunningham. URL https://www.youtube.com/watch?v=pqeJFYwnkjE. Last accessed: 1 October 2016.

[3]  Software Entropy. URL https://en.wikipedia.org/wiki/Software_entropy. Last accessed: 1 October 2016.

[4]  N. Ford. Evolutionary architecture and emergent design: Investigating architecture and design. IBM developerWorks. URL http://www.ibm.com/developerworks/java/library/j-eaed1/index.html.  Last accessed: 1 October 2016.

[5]  K. Beck and C. Andres. Extreme Programming Explained: Embrace Change, 2nd Edition. Addison Wesley, 2005.

[6]  Solid principles. URL https://en.wikipedia.org/wiki/SOLID_(object-oriented_design). Last accessed: 1 October 2016.

[7]  C. Y. Baldwin and K. B. Clark. Design Rules: The power of modularity. The MIT Press, 1999.

[8]  J. Coffin. Cohesion and Coupling: Principles of Orthogonal, Object-Oriented Programming. URL http:// www.jasoncoffin.com/cohesion-and-coupling-principles-of-orthogonal-object-oriented-programming/. Last accessed: 1 October 2016.

[9]  N. Zazworka, A. Vetro, C. Izurieta, S. Wong, Y. Cai, C. Seaman, and F. Shull. “Comparing Four Approaches for Technical Debt Identification.” Tech. rep., 2016. URL https://www.cs.montana.edu/ courses/esof522/handouts_papers/TDLandscape.pdf.

[10]  B. W. Boehm. Software Engineering Economics. 1981.

[11]  M. Kuznetcov. Measuring Architectural Technical Debt. Master’s thesis. URL www.ru.nl/publish/pages/ 769526/z-mscis-s4340132-mkuznetcov-2014-08-28.pdf.

[12]  ISO 9126 standard. URL http://www.iso.org/iso/catalogue_detail.htm?csnumber=22749. Last accessed: 1 October 2016.

[13]  ISO/IEC 25010 standard.     URL http://www.iso.org/iso/home/store/catalogue_ics/catalogue_ detail_ics.htm?csnumber=35733. Last accessed: 1 October 2016.

[14]  N. Zazworka. Technical Debt. URL http://www.nicozazworka.com/research/technical-debt/. Last accessed: 1 October 2016.

[15]  SonarQube. URL http://www.sonarqube.org/. Last accessed: 1 October 2016.

[16]  SQALE. URL http://www.sqale.org. Last accessed: 1 October 2016.

[17]  N. Brown, Y. Cai, Y. Guo, R. Kazman, M. Kim, P. Kruchten, E. Lim, A. MacCormack, R. Nord, I. Ozkaya, R. Sangwan, C. Seaman, K. Sullivan, and N. Zazworka. “Managing Techni- cal Debt in Software-Reliant Systems.” Tech. rep. URL https://pdfs.semanticscholar.org/f754/ db80f0e465cfcad4077c5703ff1cdfd8e902.pdf.

[18]  J. Holvitie, V. L. nen, and S. Hyrynsalmi. “Technical Debt and the Effect of Agile Software Development Practices on It – An Industry Practitioner Survey.” Tech. rep. URL http://conferences.computer.org/ mtd/2014/papers/6791a035.pdf.

[19]  C. Sterling. Managing Software Debt – Building for Inevitable Change. Addison Wesley, 2010.

[20]  J. C. Sanchez, L. Williams, and E. M. Maximilien. On the Sustained Use of a Test- Driven Development Practice at IBM. URL https://pdfs.semanticscholar.org/a00c/ 61b77e2df21b43d5e500341d5efec286c195.pdf. Last accessed: 1 October 2016.

[21]  C. Fourie. “EXTREME PROGRAMMING PRACTICES THAT MITIGATE TECHNICAL DEBT.” Tech. rep., School of Electrical and Information Engineering, University of the Witwatersrand, 2016.

[22]  Extreme Programming . URL http://www.extremeprogramming.org. Last accessed: 1 October 2016.

[23]  Escaping the black hole of technical debt. URL https://www.atlassian.com/agile/technical-debt. Last accessed: 1 October 2016.

by Carel Fourie

Leave a Reply

Your email address will not be published. Required fields are marked *