For this project, you will examine and test an open source program of your
choice that reads in a text file and transforms it or computes with it to
produce output.
The program must perform some nontrivial transformation or computation based
upon the provided file (and possibly command line options).
For instance, the cat command would be grossly insufficient.
In previous semesters, students have had success looking at programs that,
for instance, convert CSV
formatted files to XML or
JSON formatted files (or vice versa).
Many such csv2xml, csv2json, etc. programs for converting data formats can be
found on the internet.
Following the processes outlined in class, this assignment incorporates the following components.
Students may work in pairs or in groups of three for this project.
I am open to the idea of testing something else that may not transform a text format (more complex and more interesting but not less). Contact me for approval if you have something in mind.
The software that you test may be written in any language. The code that you write to perform testing may be written in any language. (You may find some easier than others.)
Provide a concise specification of the input/output requirements of the program or portion of a program that you are testing, with references to appropriate documents. Specifically, you will be interested in any standards documents that specify the formats.
Do not simply say, e.g., which formats you are converting between. The actual specifications of those formats are necessary for identifying relationships between the input and output as well as how they form partitions of the input space. Many students assume that they already understand formats like CSV without recognizing the corner cases that arise from actual specifications.
There are also frequently ways in which corner cases in the input affect the output and vice versa. You should be taking these into account. How do the input and output relate?
Recall the approach presented in class for input space partitioning.
You should clearly identify the partitions resulting from your characteristics and also explain why you added any constraints as a part of your report. Your model must include at least some constraints. Your input domain model should be functionality based.
ACTS is a tool provided by the National Institute of Standards and Technology (NIST) for assisting in combinatorial testing. The documentation page for the tool is available here. The actual manual for ACTS is available here. You can download the (Java based) tool via CourSys here. To run ACTS using the GUI interface, you can use the following command as described in section 3 of the manual (page 11):
1
java -jar acts_3.0.jar
Using ACTS, you will generate pairwise tests for the program that you test. You can use the scratch mode of ACTS to build a new test suite "from scratch." ACTS also allows you to specify the test generation constraints. You should encode your constraints specified in your model within the interface/language that ACTS provides.
Note, ACTS only explicitly supports enums, booleans, and integers, and the data that we want to work with may include string or more complex data. This is okay. Recall the different ways that we looked at formulating the triangle classification problem. Even complex structures like shapes can be represented using only enums and booleans, although you may have to do more work to translate the results from ACTS into actual test data.
Once you have generated a plan for which tests to run, create an actual test for each planned test frame. You should run these tests on your selected program.
For example, to test a csv2xml program, one approach is to create your tests
within a TestData/ directory with three subdirectories: TestFiles/,
ExpectedOutput/, ExpectedMessages/.
The TestFiles/ directory contains all the input files for your test cases with
one input file per case.
The ExpectedOutput/ directory contains an expected XML output file for every
input file.
The ExpectedMessages/ directory contains an expected message file for every
test file, with the kind of error message expected.
Whether you use that structure or not, you need to generate actual test inputs
and actual expected outputs for them.
You should design the structure of these files to work with a test harness
script/program that you also write.
This test harness should call the program on each input file and produces the
outputs to compare them against the expected results (as computed above).
If you structure things cleanly, you can even just do this with diff.
You must actually construct, automate, and run the tests in your test plan. You should submit your actual testing infrastructure. You may write everything from scratch or generate tests in an appropriate framework of your choice.
Export your ACTS project as an XML file and include it in your submission.
Using the generated test suite, evaluate the program under test and report the results. How many tests did you generate? How many of these tests were successful/passing? How many tests would have been generated if you didn't use pairwise testing? What tradeoffs did you make as a result of pairwise testing? Give concrete examples and explain the implications.
You should document and explain each component of your assignment in an assignment report that you submit along with all components of the project. Note that this report should not simply be a list of bullet points. Write. It should address questions raised in all sections of this document with justified explanations. That doesn't mean that it needs to be long. It just needs to address the questions.
Finally, include a brief reflection on the experience. What was easy? What was difficult? What did you feel would have made your experience easier at a technical level?
The assignment is due 11:59 pm on October 6.
Points on the assignment are allocated as follows:
| Basic Specification | 5 |
| Input Space Modeling | 15 |
| Combinatorial Planning | 10 |
| Test Infrastructure | 10 |
| Report and Analysis | 10 |
| Total | 50 |