Agile Software Development

based on Chp 3 of Sommerville “Software Engineering” (9th edition)

CMPT 276, Spring 2019, SFU Surrey

Agile Methods

agile software development methods are popular for

• small teams
• fast-paced and changing environments
• rapid prototyping
• exploratory projects

agile methods

• are iterative and incremental, i.e. multiple small releases
• typically get customer feedback throughout the process
• involve lightweight and “low ceremony”, i.e. emphasize people over process and documents

some well-known agile methods

• Extreme programming (XP)
• Scrum

Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

(from https://agilemanifesto.org/)

Agile Methods

most Agile methods share the following characteristics …

customer involvement

• customers should be involved throughout the development process
• they provide feedback on versions, suggest features, help with requirements, etc.

incremental delivery

• system developed in a series of versions

people not process

• developers skills and talents should be highlighted
• should be allowed to work in the way they prefer (as opposed to being required to follow a prescribed standard)

embrace change

• assume requirements and design will always be changing, so work accordingly

maintain simplicity

• focus on simplicity in both software and process; “what’s the simplest thing that could work?”

from Sommerville, Software Engineering 9, Figure 3.2

The “worse is better” design philosophy

the “worse is better” philosophy is an approach to thinking about design proposed by software engineer Richard Gabriel (http://dreamsongs.com/RiseOfWorseIsBetter.html)

he suggests that design is concerned with at least these things:

• Simplicity
• Correctness
• Consistency
• Completeness

Gabriel then proposes two “schools” of design

The “Correctness” School (aka the MIT approach)

• Simplicity
  • The design must be simple, both in implementation and interface. It is more important for the interface to be simple than the implementation.
• Correctness
  • The design must be correct in all observable aspects. Incorrectness is simply not allowed.
• Consistency
  • The design must be consistent. A design is allowed to be slightly less simple and less complete to avoid inconsistency. Consistency is as important as correctness.
• Completeness
  • The design must cover as many important situations as is practical. All reasonably expected cases must be covered. Simplicity is not allowed to overly reduce completeness.

The “Worse is better” School of design (aka the New Jersey approach)

• Simplicity
  • The design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design.
• Correctness
  • The design should be correct in all observable aspects, but it is slightly better to be simple than correct.
• Consistency
  • The design must not be overly inconsistent. Consistency can be sacrificed for simplicity in some cases, but it is better to drop those parts of the design that deal with less common circumstances than to introduce either complexity or inconsistency in the implementation.
• Completeness
  • The design must cover as many important situations as is practical. All reasonably expected cases should be covered. Completeness can be sacrificed in favor of any other quality. In fact, completeness must be sacrificed whenever implementation simplicity is jeopardized. Consistency can be sacrificed to achieve completeness if simplicity is retained; especially worthless is consistency of interface.

these are quite different philosophies!

the key part of the “worse is better” is that simplicity is the most important property of the interface and implementation of the system

Unix vs Multics

the Unix operating system is often held up as a good example of a “worse is better” software project

Unix grew out of the Multics project, a big operating system project in the 1960s

Multics was a famously big, complex system that always tried to do the “right thing” in most cases

a lot of Multics code was written to handle possible but unlikely situations, and this was one reason for its complexity

the original Unix was much simpler than Multics in many cases

Unix often simply crashed, even in cases where it was known how to recover from a problem

failure in Unix typically returns an error code, and then you do it again

Unix doesn’t try to recover and fix things, which makes much of its code relatively simple

The C Language

the C programming language was designed for implementing Unix

and C has much the same flavor as Unix: it just crashes when things go wrong

many languages before C “fixed” many of the problems that C has

but C purposely chose to ignore those fixes in the name of simplicity

interestingly, the Multics oeprating system used the PL/1 language for much of its implementation

PL/1 was a large language with many useful features, and compilers for it were difficult to implement

thus, in practice, some subset of PL/1 was usually chosen to work with

Popularity of C and Unix

both C and Unix have proven to be extremely popular

not just because of their simple designs, but that surely helped

many of the perceived flaws in both were actually intentional design decisions that favored simplicity over complexity

it seems that simplicity is a good designed characteristic because it results in things that people can understand

understandable things are easier to fix, change, and use well

Extreme Programming (XP)

from Sommerville, Software Engineering 9, Figure 3.3

Principles and Practices of Extreme Programming

incremental planning

• uses a technique called the planning game
  • by calling it a “game”, it takes away some of the anxiety that often surrounds plans in more traditional software methods, where getting the plan right is an important, difficult, and thus scary party of the process
• requirements written on “story cards” (a story is a kind of usage case)
• priority, time available, and amount of work estimated to achieve the “story” are taken into account
• plans are made for about 2 weeks into the future
• if necessary, prototyping might be done to better understand what is needed

small releases

• get a working system up and running as soon as possible
• therefore focus on features that will enable this

simple design

• do the minimal amount of design, i.e. design just for the feature being implemented and nothing else
• YAGNI: You Aren’t Going to Need It
• “Do the simplest possible thing that could work”
• XP explicitly does not encourage flexible designs that support change!
• instead, it says get something working now, and refactor it later when necessary

test driven development (TDD)

• write test before you write code
• use the tests to specify your code, better understand it, and to test it
• tests are concrete examples, which can be taken from user stories
• tests are a good way to understand and think about a system concretely
• testing tools/frameworks are encouraged because tests will be run frequently

refactoring

• refactoring is rewriting code to make it better without adding features or substantially changing its functionality
• important in XP because software is, initially, intentionally simple and inflexible

pair programming

• developers work in pairs to check each others work and provide support
• 2 programmers sit together at the same desk in front of one computer

collective ownership

• anyone can change anything: any developer could work on any part of the project

continuous integration

• as soon as a feature is completed, it is added to the entire system and the system is tested
• could occur multiple times a day, so testing is interleaved with development

sustainable pace

• overtime is avoided
• too much overtime is often detrimental to the overall quality of the project

on-site customer

• developers should have access to a customer at any time so they can ask questions and check requirements

communication

• XP encourages face-to-face communication (or its digital alternative) over written documentats
• source code documentation is discouraged since if the code changes then programmers often don’t have time to update its documentation

Scrum

Scrum is a popular agile project management approach

Scrum doesn’t prescribe programming practices the way XP does, so an agile team could combine both Scrum and XP if they wish

the key idea of Scrum is that development is organized into a series of sprints, each sprint maybe a couple of weeks long

at the start of a sprint, the team chooses the features they are going to work on from the list of unimplemented features called the backlog

during the sprint, the developers are isolated from the customer and other distractions, allowing them to focus just on software development

the project manager, known as the scrum master, helps keep the team isolated by dealing with outside concerns for them

daily stand-up meetings are held; these are short meetings where, often, everyone literally stands up to make sure it doesn’t go on too long

in stand-up meetings, developers report to each other what they are doing, if they are stuck, etc.

at the end of the sprint, the results are reviewed and presented to stakeholders

Team Size

agile methods like XP and Scrum were designed for small teams

lots of anecdotal evidence (and some studies) show that these methodologies can indeed work well with small teams

however, it is not so clear if agile methods work well with large systems

Features of Large Systems

usually involve combining multiple systems

• can have geographically separate teams in different time zones
• communication (and politics) can be more challenging

external rules and regulations can be more of a constraint

• e.g. all SFU data must be stored on Canada-based servers
• e.g. final exams must be archived for at least one year after
• these are examples of a non-functional requirement

larger time scales lead to personnel turn-over

• people leave the team, people join the team

many different stakeholders

• e.g. a course management system like Canvas has teachers, TAs, students, designers, and administrators, and they all have different needs and interests in the system

Agile for Large Projects?

for larger projects, agile-like methods may work, but with some adjustments, such as:

• more up-front design and planning is needed
  • can’t just rely on doing everything in code
  • you need some documentation so different teams know what the other team is doing
    • reading the code isn’t always practical!
• more meetings
  • members of smaller teams are often sitting near each other and can talk at any time
  • different teams need to meet with each other to keep in sync
  • also need other kinds of communication, like email, instant messages, newsgroups (e.g. Discord/Slack), etc.
• continuous integration needs special support
  • e.g. software for allowing geographically distance teams to add and test features when they are ready

Why Doesn’t Everyone Use Agile?

even for small projects, not all companies use agile methods

some reasons for this are

• inexperience with agile methods, so it is risky to use them
  • if the current processes work, what is the value in changing?
• sometimes there are procedures and standards that must be followed, and these rules aren’t always easy to change in big companies
  • e.g. all projects in the company might be required to start with large up-front planning documents
  • e.g. some companies might simply forbid certain employees from interacting direcly with customers
• perception that agile methods work well only with best and most experienced developers
  • usually a wide range of developer skill and know-how at large companies
• misunderstanding of agile methods, e.g. might be using agile terms, but not really doing agail development

Detecting Agile BS

(from US Dept. of Defense document called “DIB Guide: Detecting Agile BS”, v0.4, 3 Oct. 2018)

software development is, unfortunately, one of those areas where there is a lot of “snake oil”

it is filled with buzzwords, and it can sometimes be the case that you are using buzzwords without really using, or understanding, the underlying practice

for many programmers, software engineering methodology is boring and just gets in the way of the fun part, which is making programs

this document offers some concrete and direct suggestions about how to tell if you are not really doing agile development

some key flags that you are not really doing agile:

• no one on the team is talking to users, or observing them use the software

• continuous feedback from users to the development team is not available

• meeting requirements is treated as more important than getting some working software into use

  • this implies that getting working software released should be the requirement with the highest priority

• end-users are missing in action

  • users should be involved in planning and feedback throughout the project, otherwise they could be unhappy with the final result

• tolerating manual processes that could be automated

  • e.g. version control should be handled by a tool like Git
  • e.g. testing should be automated
  • e.g. tools like Travis-CI can be used to help with continuous integration (Travis-CI monitors a Github repository and runs a script (usually for testing) whenever it is updated)
  • e.g. Chef/Ansible/Puppet for configuration scripts
  • e.g. Jira/Github issues for issue tracking (bugs, feature requests)

• some questions to ask a software development team

  • How do you test your code?

    • bad answer: the testing group does it

  • How automated is your development? What tools are you using?

  • Who are your users, and how are you interacting with them?

    • How long does it take to respond to their feedback?

  • How many programmers are part of the group the owns the budget and sets milestones?

    bad answers: zero, “we don’t know”

  • How do you measure progress and success?