Assignment 2

##Test Adequacy via Mutation Testing

For this project, you will perform a mutation adequacy study of an existing test suite for an open-source software project.

You will work in groups of two to four students, but the overall requirements will scale with the group size.

The following requirements apply to the open-source project.

  1. It must be hosted/sponsored/associated with an organization that is affiliated with Google Summer of Code 2012. That is the organization must have had GSoC projects active last summer. Any exceptions to this must be approved.
  2. The code base must be at least 20,000 lines of code (20 KLoC). A substantial test suite must exist for the project.

In finding which project to analyze, you should identify and consider the following:

  1. Identification of the open-source project.
  2. Identification of the supporting organization.
  3. Size of the code base.
  4. Proposed evaluation platform (OS, language).
  5. Build time to compile and link an executable from source code.
  6. Execution time for the test suite.

If you have questions about whether a particular project is a good choice, identify these attributes and ask.

Each group member should choose one routine (procedure, function or method) from the open source project and generate mutations to test it. Choose a complex routine that is likely to have errors. If it seems difficult to create sufficiently many mutants for one routine, you may choose one or two additional routines, but keep them in the same file.

Generate mutants manually by applying a small number of mutation operators to the code. Document your mutation operators. Each mutant should be identical to the original code except for the application of one mutation operator at one specific location. Each team member should generate at least 25 mutants for testing.

The source code for each routine tested should be submitted, together with a diff file showing a 3-line context diff for each mutant created. (diff -c)

Apply the test suite to all mutants. Document the results: how many mutants are killed for each of the chosen routines. For every mutant that was not killed, try to determine either (a) that it is an equivalent mutant that should not be killed, or (b) how to add a test to kill it.

Calculate the effectiveness of the test suite. Effectiveness should be calculated separately for each routine tested. Overall effectivness should also be calculated using the ratio of total mutants killed to the total number of non-equivalent mutants created. Discuss your results and your methods, including how you might automate the generation of mutants and/or the application of tests. You might also contrast the costs and benefits of mutation testing versus what you might expect from other techniques.

The assignment is due 11:59 pm on Monday, February 24, 2014. It is worth 1/3 of your overall assignment grade.

Much of the structure for this projects is based on plans by Rob Cameron