Extreme Programming

Extreme Programming (XP) is a software engineering methodology, the most prominent of several agile software development methodologies. Like other agile methodologies, Extreme Programming differs from traditional methodologies primarily in placing a higher value on adaptability than on predictability. Proponents of XP regard ongoing changes to requirements as a natural, inescapable and desirable aspect of software development projects; they believe that being able to adapt to changing requirements at any point during the project life is a more realistic and better approach than attempting to define all requirements at the beginning of a project and then expending effort to control changes to the requirements.

XP prescribes a set of day-to-day practices for managers and developers; the practices are meant to embody and encourage particular values. Proponents believe that the exercise of these practices?awhich are traditional software engineering practices taken to so-called "extreme" levels?aleads to a development process that is more responsive to customer needs ("agile") than traditional methods, while creating software of better quality. [1].

History

Extreme Programming was created by Kent Beck, Ward Cunningham, and Ron Jeffries during their work on the Chrysler Comprehensive Compensation System (C3) payroll project. Kent Beck became the C3 project leader in March 1996 and began to refine the development methodology used on the project. Kent Beck wrote a book on the methodology, and in October 1999, Extreme Programming Explained was published. Chrysler cancelled the essentially unsuccessful C3 project in February 2000, but the methodology had caught on in the software engineering field. As of 2006, a number of software-development projects continue to use Extreme Programming as their methodology.

Origins

Software development in the 1990s was shaped by two major influences: internally, object-oriented programming replaced procedural programming as the programming paradigm favored by some in the industry; externally, the rise of the Internet and the dot-com boom emphasized speed-to-market and company-growth as competitive business factors. Rapidly-changing requirements demanded shorter product life-cycles, and were often incompatible with traditional methods of software development.

The Chrysler Comprehensive Compensation project was started in order to determine the best way to use object technologies, using the payroll systems at Chrysler as the object of research, with Smalltalk as the language and GemStone as the persistence layer. They brought in Kent Beck, a prominent Smalltalk practitioner, to do performance tuning on the system, but his role expanded as he noted several issues they were having with their development process. He took this opportunity to propose and implement some changes in their practices based on his work with his frequent collaborator, Ward Cunningham.

The first time I was asked to lead a team, I asked them to do a little bit of the things I thought were sensible, like testing and reviews. The second time there was a lot more on the line. I thought, "Damn the torpedoes, at least this will make a good article," [and] asked the team to crank up all the knobs to 10 on the things I thought were essential and leave out everything else. ?a Kent Beck

Beck invited Ron Jeffries to the project to help develop and refine these methods. Jeffries thereafter acted as a kind of coach to instill the practices as habits in the C3 team.

Information about the principles and practices behind XP was disseminated to the wider world through discussions on the original Wiki, Cunningham's WikiWikiWeb. Various contributors discussed and expanded upon the ideas, and some spin-off methodologies resulted (see agile software development). Also, XP concepts have been explained, for several years, using a hyper-text system map on the XP website at "www.extremeprogramming.org" circa 1999 (website ^XPORG).

Beck edited a series of books on XP, beginning with his own Extreme Programming Explained (1999, ISBN 0-201-61641-6), spreading his ideas to a much larger, yet very receptive, audience. Authors in the series went through various aspects attending XP and its practices, even a book critical of the practices.

Current state

XP created quite a buzz in the late 1990s and early 2000s, seeing adoption in a number of environments radically different from its origins.

The high discipline required by the original practices often went by the wayside, causing certain practices to be deprecated or left undone on individual sites. Agile development practices have not stood still, and XP is still evolving, assimilating more lessons from experiences in the field. In the second edition of Extreme Programming Explained, Beck added more values and practices and differentiated between primary and corollary practices.

Goal of XP

Extreme Programming Explained describes Extreme Programming as being:

• An attempt to reconcile humanity and productivity
• A mechanism for social change
• A path to improvement
• A style of development
• A software development discipline

The main aim of XP is to lower the cost of change. In traditional system development methods (like SSADM) the requirements for the system are determined at the beginning of the development project and often fixed from that point on. This means that the cost of changing the requirements at a later stage (a common feature of software engineering projects) will be high.

XP sets out to lower the cost of change by introducing basic values, principles and practices. By applying XP, a system development project should be more flexible with respect to changes.

XP values

Extreme Programming initially recognized four values. A new value was added in the second edition of Extreme Programming Explained. The five values are:

• Communication
• Simplicity
• Feedback
• Courage
• Respect (the latest value)

Building software systems requires communicating system requirements to the developers of the system. In formal software development methodologies, this task is accomplished through documentation. Extreme Programming techniques can be viewed as methods for rapidly building and disseminating institutional knowledge among members of a development team. The goal is to give all developers a shared view of the system which matches the view held by the users of the system. To this end, Extreme Programming favors simple designs, common metaphors, collaboration of users and programmers, frequent verbal communication, and feedback.

Extreme Programming encourages starting with the simplest solution and refactoring to better ones. The difference between this approach and more conventional system development methods is the focus on designing and coding for the needs of today instead of those of tomorrow, next week, or next month. Proponents of XP acknowledge the disadvantage that this can sometimes entail more effort tomorrow to change the system; their claim is that this is more than compensated for by the advantage of not investing in possible future requirements that might change before they become relevant. Coding and designing for uncertain future requirements implies the risk of spending resources on something that might not be needed. Related to the "communication" value, simplicity in design and coding should improve the (quality of) communication. A simple design with very simple code could be easily understood by most programmers in the team.

Within Extreme Programming, feedback relates to different dimensions of the system development:

• Feedback from the system: by writing unit tests, or running periodic integration tests, the programmers have direct feedback from the state of the system after implementing changes.
• Feedback from the customer: The functional tests (aka acceptance tests ) are written by the customer and the testers. They will get concrete feedback about the current state of their system. This review is planned once in every two or three weeks so the customer can easily steer the development.
• Feedback from the team: When customers come up with new requirements in the planning game the team directly gives an estimation of the time that it will take to implement.

Feedback is closely related to communication and simplicity. Flaws in the system are easily communicated by writing a unit test that proves a certain piece of code will break. The direct feedback from the system tells programmers to recode this part. A customer is able to test the system periodically according to the functional requirements (aka user stories). To quote Kent Beck, "Optimism is an occupational hazard of programming, feedback is the treatment."

Several practices embody courage. One is the commandment to always design and code for today and not for tomorrow. This is an effort to avoid getting bogged down in design and requiring a lot of effort to implement anything else. Courage enables developers to feel comfortable with refactoring their code when necessary. This means reviewing the existing system and modifying it so that future changes can be implemented more easily. Another example of courage is knowing when to throw code away: courage to remove source code that is obsolete, no matter how much effort was used to create that source code. Also, courage means persistence: A programmer might be stuck on a complex problem for an entire day, then solve the problem quickly the next day, if only he or she is persistent.

The respect value manifests in several ways. In Extreme Programming, team members respect each other because programmers should never commit changes that break compilation, that make existing unit-tests fail, or that otherwise delay the work of their peers. Members respect their work by always striving for high quality and seeking for the best design for the solution at hand through refactoring.

Principles

The principles that form the basis of XP are based on the values just described and are intended to foster decisions in a system development project. The principles are intended to be more concrete than the values and more easily translated to guidance in a practical situation.

Feedback is most useful if it is done rapidly. The time between an action and its feedback is critical to learning and making changes. In Extreme Programming, unlike traditional system development methods, contact with the customer occurs in small iterations. The customer has clear insight into the system that is being developed. He or she can give feedback and steer the development as needed.

Unit tests also contribute to the rapid feedback principle. When writing code, the unit test provides direct feedback as to how the system reacts to the changes one has made. If, for instance, the changes affect a part of the system that is not in the scope of the programmer who made them, that programmer will not notice the flaw. There is a large chance that this bug will appear when the system is in production.

Assuming simplicity is about treating every problem as if its solution were "extremely simple". Traditional system development methods say to plan for the future and to code for reusability. Extreme programming rejects these ideas.

The advocates of Extreme Programming say that making big changes all at once does not work. Extreme Programming applies incremental changes: for example, a system might have small releases every three weeks. By making many little steps the customer has more control over the development process and the system that is being developed.

The principle of embracing change is about not working against changes but embracing them. For instance, if at one of the iterative meetings it appears that the customer's requirements have changed dramatically, programmers are to embrace this and plan the new requirements for the next iteration.

Activities

XP describes four basic activities that are performed within the software development process.

Coding

The advocates of XP argue that the only truly important product of the system development process is code (a concept to which they give a somewhat broader definition than might be given by others). Without code you have nothing.

Coding can be drawing diagrams that will generate code, scripting a web-based system or coding a program that needs to be compiled.

Coding can also be used to figure out the most suitable solution. For instance, XP would advocate that faced with several alternatives for a programming problem, one should simply code all solutions and determine with automated tests (discussed in the next section) which solution is most suitable.

Coding can also help to communicate thoughts about programming problems. A programmer dealing with a complex programming problem and finding it hard to explain the solution to fellow programmers might code it and use the code to demonstrate what he or she means. Code, say the exponents of this position, is always clear and concise and cannot be interpreted in more than one way. Other programmers can give feedback on this code by also coding their thoughts.

Testing

One cannot be certain of anything unless one has tested it. Testing is not a perceived, primary need for the customer. A lot of software is shipped without proper testing and still works (more or less). In software development, XP says this means that one cannot be certain that a function works unless one tests it. This raises the question of defining what one can be uncertain about.

• You can be uncertain whether what you coded is what you meant. To test this uncertainty, XP uses Unit Tests. These are automated tests that test the code. The programmer will try to write as many tests he or she can think of that might break the code he or she is writing; if all tests run successfully then the coding is complete.

• You can be uncertain whether what you meant is what you should have meant. To test this uncertainty, XP uses acceptance tests based on the requirements given by the customer in the exploration phase of release planning.

Listening

Programmers do not necessarily know anything about the business side of the system under development. The function of the system is determined by the business side. For the programmers to find what the functionality of the system should be, they have to listen to business.

Programmers have to listen "in the large": they have to listen to what the customer needs. Also, they have to try to understand the business problem, and to give the customer feedback about his or her problem, to improve the customer's own understanding of his or her problem.

Communication between the customer and programmer is further addressed in The Planning Game.

Designing

From the point of view of simplicity, one could say that system development doesn't need more than coding, testing and listening. If those activities are performed well, the result should always be a system that works. In practice, this will not work. One can come a long way without designing but at a given time one will get stuck. The system becomes too complex and the dependencies within the system cease to be clear.

One can avoid this by creating a design structure that organizes the logic in the system. Good design will avoid lots of dependencies within a system; this means that changing one part of the system will not affect other parts of the system.

Practices

Extreme Programming has 12 practices, grouped into four areas, derived from the best practices of software engineering:

Fine scale feedback

• Pair Programming
• Planning Game
• Test Driven Development
• Whole team

Continuous process

• Continuous Integration
• Design Improvement
• Small Releases

Shared understanding

• Coding Standards
• Collective Code Ownership
• Simple Design
• System Metaphor

Programmer welfare

• Sustainable Pace

Controversial aspects

The most controversial aspect of Extreme Programming is the change management aspect of the process. More formal software development processes require change requests to be analyzed and approved by a change control board (or equivalent). In Extreme Programming, the on-site customer requests the changes informally, often by verbally informing the development team.

Unstable Requirements: Proponents of Extreme Programming claim that by having the on-site customer request changes informally, the process becomes flexible, and saves the cost of formal overhead. Critics of XP claim this can lead to costly rework and project scope creep beyond what was previously agreed or funded.

User Conflicts: Change control boards are a sign that there are potential conflicts in project objectives and constraints between multiple users. XP's expedited methodology is somewhat dependent on programmers being able to assume a unified client viewpoint so the programmer can concentrate on coding rather than documentation of compromise objectives and constraints. This also applies when multiple programming organizations are involved, particularly organizations which compete for shares of projects.

Revision Funding: Change control boards might distinguish between discrepancy reports and change requests to determine funding. A discrepancy from stated requirements could be considered a "mistake" by developers for the work originally funded, whereas a change request could be considered a new requirement to be backed by new funding, especially for a large-scale change. The funding of software revisions could be controlled in that manner.

Other Aspects: Other controversial aspects of Extreme Programming include:

• Requirements are expressed as automated acceptance tests rather than specification documents.
• Requirements are defined incrementally, rather than trying to get them all in advance.
• Software developers are required to work in pairs.
• There is no Big Design Up Front. Most of the design activity takes place on the fly and incrementally, starting with "the simplest thing that could possibly work" and adding complexity only when it's required by failing tests. Critics fear this would result in more re-design effort than only re-designing when requirements change.
• A customer representative is attached to the project. This role can become a single-point-of-failure for the project, and some people have found it to be a source of stress. Also, there is the danger of micro-management by a non-technical representative trying to dictate the use of technical software features and architecture.

Scalability: Opponents of XP claim that XP would only work on teams of twelve or fewer people. However, it has been claimed that XP has been used successfully on teams of over a hundred developers. It is not that XP doesn't scale, just that few people have tried to scale it, and proponents of XP refuse to speculate on this facet of the process. ThoughtWorks has claimed reasonable success on distributed XP projects with up to sixty people.

Controversy in Book: In 2003, Matt Stephens and Doug Rosenberg published a book under Apress called Extreme Programming Refactored: The Case Against XP which questioned the value of the XP process and suggested ways in which it could be improved. This triggered a lengthy debate in articles, internet newsgroups, and web-site chat areas. The core argument of the book is that XP's practices are interdependent but that few practical organisations are willing/able to adopt all the practices; therefore the entire process fails. The book also makes other criticisms and it draws a likeness of XP's "collective ownership" model to socialism in a negative manner.

XP Evolution: Certain aspects of XP have changed since the book Extreme Programming Refactored (2003) was published; in particular, XP now accommodates modifications to the practices as long as the required objectives are still met. XP also uses increasingly generic terms for processes. Some argue that these changes invalidate previous criticisms; others claim that this is simply watering the process down.

Hybrid/Unified Methodologies: Other authors have tried to reconcile XP with the older methods in order to form a unified methodology. Some of these XP sought to replace, such as the waterfall method; example: Project Lifecycles: Waterfall, Rapid Application Development, and All That.

JPMorgan Chase & Co. tried combining XP with the computer programming methodologies of Capability Maturity Model Integration (CMMI), and Six Sigma. They found that the three systems reinforced each other well, leading to better development, and did not mutually contradict, see Extreme Programming (XP) Six Sigma CMMI.

Finally, XP is not the only controversial methodology, because XP is a response to the controversy and criticisms about other methods used in software development.

Application of Extreme Programming

Controversial aspects notwithstanding, Extreme Programming remains a sensible choice for some projects. Projects suited to Extreme Programming are those that:

• Involve new or prototype technology, where the requirements change rapidly, or some development is required to discover unforeseen implementation problems
• Are research projects, where the resulting work is not the software product itself, but domain knowledge
• Are small and more easily managed through informal methods

Projects suited for more traditional methodologies are those that:

• Involve stable technology and have fixed requirements, where it is known that few changes will occur
• Involve mission critical or safety critical systems, where formal methods must be employed for safety or insurance reasons
• Are large projects which may overwhelm informal communication mechanisms
• Have complex products which continue beyond the project scope to require frequent and significant alterations, where a recorded knowledge base, or documentation set, becomes a fundamental necessity to support the maintenance

Project Managers must weigh project aspects against available methodologies to make an appropriate selection. However, some XP concepts could be applied outside, such as using Pair Programming to expedite related technical changes to the documentation set of a large project.

ThoughtWorks' experiences with implementing XP in large distributed projects can be found at Using an Agile Software Process with Offshore Development