Assignment 2

Testing with Input Space Partitioning

For this project, you will examine and test an open source program of your choice that reads in a text file and transforms it or computes with it to produce output. The program must perform some nontrivial transformation or computation based upon the provided file (and possibly command line options). For instance, the cat command would be grossly insufficient. In previous semesters, students have had success looking at programs that, for instance, convert CSV formatted files to XML or JSON formatted files (or vice versa). Many such csv2xml, csv2json, etc. programs for converting data formats can be found on the internet.

Following the processes outlined in class, this assignment incorporates the following components.

  1. Specification of the Program Under Test
  2. Input Space Partitioning
  3. Combinatorial Test Generation using ACTS
  4. Test Report

Students may work in pairs or in groups of three for this project.

Specification of the Program Under Test

Provide a concise specification of the input/output requirements of the program or portion of a program that you are testing, with references to appropriate documents.

Input Space Partitioning

Recall the approach presented in class for input space partitioning.

  1. Clearly identify the component you are testing along with all of its inputs, whether from parameters, user input, the environment, etc.
  2. Identify the characteristics for partitioning the input space.
  3. Clearly present your partitioned input domain model along with any constraints that are necessary for making it consistent.

You should clearly identify the partitions resulting from your characteristics and also explain why you added any constraints as a part of your report. Your model must include at least some constraints. Your input domain model should be functionality based.

Combinatorial Test Generation using ACTS

ACTS is a tool provided by the National Institute of Standards and Technology (NIST) for assisting in combinatorial testing. The documentation page for the tool is available here. The actual manual for ACTS is available here. You can download the (Java based) tool by following the instructions here. To run ACTS using the GUI interface, you can use the following command as described in section 3 of the manual (page 11):

    java -jar acts_gui.jar

Using ACTS, you will generate pairwise tests for the program that you test. You can use the scratch mode of ACTS to build a new test suite "from scratch." ACTS also allows you to specify the test generation constraints. You should encode your constraints specified in your model within the interface/language that ACTS provides.

Note, ACTS only explicitly supports enums, booleans, and integers, and the data that we want to work with may include string or more complex data. This is okay. Recall the different ways that we looked at formulating the triangle classification problem. Even complex structures like shapes can be represented using only enums and booleans, although you may have to do more work to translate the results from ACTS into actual test data.

Once you have generated a plan for which tests to run, create an actual test for each planned test frame. You should run these tests on your selected program.

For example, to test a csv2xml program, one approach is to create your tests within a TestData/ directory with three subdirectories: TestFiles/, ExpectedOutput/, ExpectedMessages/. The TestFiles/ directory contains all the input files for your test cases with one input file per case. The ExpectedOutput/ directory contains an expected XML output file for every input file. The ExpectedMessages/ directory contains an expected message file for every test file, with the kind of error message expected. You should design the structure of these files to work with a test harness script/program that you also write. This test harness calls the csv2xml on each input file and produces the outputs into directories TestOutput/Files/ and TestOutput/Messages/. These results can then be easily compared with the expected results, e.g. using diff.

Export your ACTS project as an XML file and include it in your submission.

Test Report & Submission

Using the generated test suite, evaluate the program under test and report the results. How many tests did you generate? How many of these tests were successful/passing? How many tests would have been generated if you didn't use pairwise testing? What tradeoffs did you make as a result of pairwise testing? Give concrete examples and explain the implications.

You should document and explain each component of your assignment in an assignment report that you submit along with all components of the project. Note that this report should not simply be a list of bullet points. It should address questions raised in all sections of this document with justified explanations.

Finally, include a brief reflection on the experience. What was easy? What was difficult? What did you feel would have made your experience better?

The assignment is due 11:59 pm on February 13.