Exercise: Unit Testing and UI Testing

Outline

Introduction

In your previous software engineering courses, you should have at least seen and hopefully even used unit testing. You should also understand the important role that testing (and unit testing in particular) plays in agile software development. In this exercise, you will get hands on experience using Google Test for unit testing and using Selenium for UI testing. The experiences using these two frameworks are broken into separate groups of tasks for each framework, and you must turn in both.

Unit Testing with GoogleTest.

In this portion of the exercise, you will see how to integrate unit testing using the Google Test framework into a C++ project built using CMake. You will gain first hand experience writing some simple unit tests, and you will be provided with resources that allow you to use more complex testing patterns within your unit tests.

For outside information, refer to:

If you are new to CMake, you may wish to also look at this CMake exercise. Please bear in mind that you should be using out of source builds. You will ultimately submit only the source code of your project and not any of the build artifacts. You should also not modify any of the provided header files or non-test source code files. They may be replaced during grading.

Submissions that do not compile and run from a clean build directory using the commands

cmake <path/to/submission/>
make
test/runAllTests

will receive 0 points.

The provided files for this portion of exercise illustrate a reasonable way to incorporate Google Test into a project. The project itself implements a small library with a variety of different functionalities. The source and header files for this library reside in the lib/ directory. Your tasks as a part of this exercise will be to test this project using the facilities in the Google Test framework. All of the tests for the project will live in the test/ directory. The test directory also includes the source code of Google Test and Google Mock inside test/lib/. Including the source code of Google Test in the project and compiling it as a part of the project has two key advantages. (1) It ensures that the version of Google Test used to run the test suite is consistent. (2) It avoids some subtle corner cases involved with compiling, linking, and testing native code specifically. In practice, you could instead include a more compact version of Google Test and Google Mock.

All of the files that you need to modify can be found in the test/ directory. NOTE: The files inside lib/ will be replaced during the grading process. Do not modify the files in those directories to create your tests. Remember to follow the instructions carefully, as projects will be graded (mostly) automatically. Specifically, make sure to spell fixture or test group names exactly as specified.

Tasks

To create a set of related test cases, you should create a C++ source file in the test/ directory. Make sure to include gtest/gtest.h and/or gmock/gmock.h as necessary. All of the source files for your tests should then be added to the list of source files for creating the runAllTests program in test/CMakeLists.txt. You can do this by editing the function call to add_executable in that file. Notice that there is also an add_test function call in test/CMakeLists.txt. You do not need to modify this. The libraries for Google Test and Google Mock will be linked in by the CMakeLists.txt configuration already, as you can see on the target_link_libraries lines.

Task 1

Open the files lib/simple/include/Parallelogram.h and lib/simple/Parallelogram.cpp. These files provide the declaration and definition of a simple Parallelogram class. The constructor of Parallelogram takes in the integral lengths of the sides of a Parallelogram and the measure of one interior angle in floating point degrees. By contract, the angle must be between 0 and 90 degrees, excluding 0 and including 90. Note that there are bugs in the getPerimeter(), getArea(), and getKind() methods. You must complete the following tasks for the Parallelogram class using the Google Test framework:

Each of these test cases should be in its own test function, and the group name or test fixture for the tests should be called ParallelogramTests. This name will be used to automatically extract individual tests during grading.

Task 2

Now consider the function checkMatthewsOutcome() declared in Matthews.h and defined in Matthews.cpp. Create a set of related tests such that every statement in checkMatthewsOutcome() is executed by at least one test. The group name or test fixture for the tests should be called MatthewsTests. Make sure the name is correct.

Task 3

Finally, consider the performAwardCeremony function declared in Awards.h and defined in Awards.cpp. This function reads a list of names from a sequence and awards medals to the first three names. Write a test case that makes sure the ceremony runs as intended. You will need to create a stub for RankList and a mock for AwardCeremonyActions. The methods of AwardCeremonyActions should be called exactly once each in the order: playAnthem(), awardBronze(), awardSilver(), awardGold(), and turnOffTheLightsAndGoHome(). The getNext() method of RankList should be called three times, and the names returned should be passed to awardBronze(), awardSilver(), and awardGold() in the same order they are read from the list. I highly recommend that you consult the Google Mock Dummies Guide in order to make sure that you (1) correctly create the test fakes, (2) validate that the methods were called, and (3) validate that they were called in the right order and with the right arguments. The group name or test fixture for the test should be called AwardsTests.

GoogleTest Submission

First double check that you have named your fixtures well by trying these commands from your build directory:

test/runAllTests --gtest_filter=ParallelogramTests.*
test/runAllTests --gtest_filter=MatthewsTests.*
test/runAllTests --gtest_filter=AwardsTests.*

To submit your exercise, create an archive of the directory containing your source (not your build), and submit it via CourSys. This should contain all of the provided project files along with your additions and modifications necessary to run your tests.

NOTE: Your archive should contain the googletest-template/ directory from the project archive as well as its subdirectories. This is necessary for your submission to be graded. Again, this should not include your build artifects. To produce this, run:

tar zcvf e1-googletest.tar.gz googletest-template/

UI Testing with Selenium

In this portion of the exercise, you will make use of Pytest and Selenium to perform UI testing of an incredibly simple web app. Pytest is a commonly used testing framework for Python, and Selenium, as discussed in class, is commonly used for UI automation and testing. Both frameworks are used extensively in industry and have many tutorials online. The concepts, however, follow the same ones that we used in class with GoogleTest and Flutter. The differences are in the exact API calls and syntax.

In this case, you have been provided with a template containing a simple web app in the page directory and a consistent location for tests in the test directory. Note, unlike the previous example, you will be testing something that is not written in the same language that you are using for writing the tests. Because UI testing often involves interacting with other programs or frameworks, it is often driven by tests that act as glue between a high level language for expressing the test logic (Python in this case) and some independent infrastructure for the app itself (HTML+JavaScript here).

For outside information, you might refer to:

The latter two will also help you get up and running to install infrastructure on your personal machine should you decide to do so for your convenience.

To receive points, your tests should run by running

pytest-3 # In many Ubuntu setups

or, depending on your system,

pytest # Specifically in CSIL

from within the test/ directory of the provided template. Note that this will use/require the Python 3 version of pytest.

NOTE: Similar to before, all of the files within page/ will be replaced during grading. You can modify them, but they will not be used for the grading process. Only the contents of your test/ directory will be used for grading. However, even the e1utils.py file in that directory will be replaced. As you can imagine, replacing the contents of page/ allows us to make sure that you can find different bugs. You might take a similar approach when testing whether your tests meet your expectations.

NOTE 2: Your may wish to test your code in CSIL. Sometimes students make a lot of incorrect customizations for getting Selenium working on their home computers. You should not need to modify e1utils.py. It will be replaced during grading. If you want to watch your tests execute, You could comment out the "headless" option.

Working in CSIL

If you must work in CSIL, then you'll want access to additional python packages. I have made them available within a python 3 virtual environment. To use them, run:

source /usr/shared/CMPT/faculty/wsumner/base/env473/bin/activate

before running your tests with only

pytest

to leave the virtual environment, run

deactivate

Task

You will have only one task for this component.

The provided web page has a main page (index.html) that is expected to be the landing page of the application. A user can fill out the form on the page and submit it to be directed to a response page (response.html). Various parts of the user's answers in the form are used within this page, but most notably, if the user enters an appropriate secret code in the code field, the response page presents them with an extra button that will take them to a special page for privileged users. The simulated server can be a bit slow, though, so it takes some time for the special page to load.

If you look at the the provided web pages, you will find that they make use of unique IDs that identify different elements of the page and how they are used. In the provided version of the web page these IDs clearly have a specific meaning corresponding to the web page behavior. You may assume that the values appearing in the identified elements are the expected ones in the provided page, modulo user inputs. For instance, an ID indicating the user name is supposed to be used to hold the user's name. You can make use of these within your tests by using the find_element() method of a Selenium Driver in order to interact with specific elements. Element interaction is trivial and demonstrated in the documentation. For instance, there are click() and send_keys() methods for sending data as well as a text field to extract data. Similarly, Drivers themselves have a current_url.

You must write tests that check for some straightworward user interactions. You should use best practices as described in class and try to keep the tests both efficient and robust. To help you with this, example code has been provided for making your tests run in "headless mode" inside of Chrome. This allows the tests to run without the actual UI being displayed. Note that if you comment out the line configuring headless mode, you will be able to see the tests run, albeit quickly. You do not have a guarantee on how long the secret page may require to load, but if it takes longer than 10 seconds, it is considered a failure.

From a high level perspective, Pytest treats each function starting with test_ as a separate test. It is advisable to create a separate test for each of these scenarios. Because they exercise similar behaviors, you might refactor those behaviors out to common functions that both tests use. You have also been provided with a function that computes the location of the landing page when you run Pytest correctly.

Scenario 1

When a user:

  1. lands on the main page
  2. fills in non-empty values for name and favorite food
  3. fills in a value for the password other than "magic" or "abracadabra"
  4. clicks submit.

Then the app should:

  1. redirect to the response page at response.html
  2. where the response page should include a thank you for the user by name
  3. and it should note that the company also likes the same food as the user
  4. and it should not include a button to the secret page.

Scenario 2

When a user:

  1. lands on the main page
  2. fills in non-empty values for name and favorite food
  3. fills in either the secret code "magic" or "abracadabra"
  4. clicks submit.

Then the app should:

  1. redirect to the response page at response.html
  2. where the response page should include a thank you for the user by name
  3. and it should note that the company also likes the same food as the user
  4. and it should include a button to the secret page
  5. and clicking the button should redirect to the page at secret.html
  6. and the secret page should thank them by name
  7. and the secret page should repeat the specific secret code entered by the user.

Selenium Submission

To submit your exercise, create an archive of the directory containing your source, and submit it via CourSys. This should contain all of the provided project files along with your additions and modifications necessary to run your tests.

NOTE: Your archive should contain the selenium-template/ directory from the project archive as well as its subdirectories. This is necessary for your submission to be graded. To produce this, run:

tar zcvf e1-selenium.tar.gz selenium-template/

Other Frameworks

As noted in class many different UI frameworks have their own UI testing support as well. In addition, there are newer general frameworks for different platforms that have come out with various trade-offs and benefits, e.g. MS Playwright. If you are interested in developing skills for a specific framework or platform, you should investigate the options most relevant to your specializations. There is no one best tool for every job.