Our strength is the automated, model-based generation of test suites using graphical usage models for hardware and software systems. Test suites can be further processed as JSON documents for the development of executable test cases or can be visualized in a user-friendly way.



Agile Travel - Web App from Practical Web Test Automation



By means of the web application Agile Travel , which is the standard example from Zhimin Zhan's book Practical Web Test Automation, we want to show how modern test methods can be advantageously extended in combination with the TestPlayer© Framework.

The app Agile Travel simulates a flight booking system and is started directly via this URL. Afterwards, the user interface presents the following view:

After pressing the Sign In button, flights can be booked:

.

Then, after pressing the Continue button, the passenger's data are entered:

.

The corresponding usage model, which serves as a basis for the automatic generation of test cases, illustrates the various options to interact with the web application:



From the initial start state [ , which is executed when the test case is started, only the usage state Agile Travel can be reached directly, then other states such as Remember me , Heading New Stuff , User Name , Password or Sign in can be selected depending on the intentions of the user. Each test case ends when the final stop state ] is reached.

State transitions between the 29 usage states (including start and stop states) are triggered by generic events e1 to e99 , which were automatically generated by the TestPlayer© in conjunction with associated transition probabilities p1 to p99 during generation of the test suite by means of the TestPlayer© Dashboard (see below).

The TestPlayer© automatically generates test cases for the model-based testing of the web application Agile Travel based on the usage model above. The following test suite contains 19 test cases for the complete coverage of all usage states, which are generated according to the sort criterion sort_l , i.e. the test cases have been sorted out of 500 statistically randomly generated test cases according to their length. Each new test case covers at least one new usage state compared to its predecessor:


  • Test suite state coverage contains 19 test cases (of in total 500 automatically generated)
The test cases are animated (marked by a bright orange coloration) and show the coverage of the usage states (characterised by a slight orange colouring) as well as the transitions between the states. The number after the colon of the respective click event shows how often the respective state transition was executed during the test case.

When generating the test suite, the TestPlayer© creates several diagrams that assist in better understanding the characteristics of the test suite. These include, for example
  • deviation of the statistical frequencies of usage states from the theoretical distribution (if no probability values are specified, this is equivalent to a uniform distribution of all state transitions). Only a few deviations can be observed, i.e. the test suite with 500 test cases reproduces the given usage model very well:



  • the frequency of usage states in the reduced test suite, which covers all states:



  • the frequency of transition events in the reduced test suite, which covers all states:


In addition to the graphical representations, the TestPlayer© also generates text files that provide the test cases of the test suite in a compact JSON format for the test execution. File test_cases_for_state_coverage.json contains all 19 test cases of the test suite state coverage for the test of the web application Agile Travel :

For automatic testing of the web application Agile Travel you need additional drivers to automate the access to the web browser. For this purpose we apply the automation framework SeleniumHQ, that can easily be integrated into an Eclipse -based test environment:

For each usage state, the Python script AgileTravel.py contains a description which actions, i.e. realizations of the abstract test steps e1 to e99 should be executed during the test step. For example, test step



defines what to do when the First name usage state is reached in order to enter a passenger name for the flight booking during test execution. In detail, the following actions must be performed:

  1. Checking pre-condition <Passenger Details> assures that the actions of the test step are executed in the Passenger Details usage state. If an error occurs, the else branch is processed and an error message is displayed during the execution of the assert statement.
  2. Finding an input field named passengerFirstName.
  3. Deleting any texts due to previous inputs.
  4. Entering a random first name (in this case Jeff) in the passengerFirstName input field:



  5. Checking of the post condition <Passenger Details> ensures that the correct state transition to the Passenger Details usage state takes place. In addition, the assert statement monitors the execution of the test step and assigns the test verdict pass , fail respectively error , depending on the outcome of the test.

In Eclipse, Run Configurations is performed to pass call parameters for test execution:



File option -f test_cases/AgileTravel.TestPlayer/500_sort_l/test_cases_for_state_coverage.json.txt loads the automatically generated test suite to cover all usage states during test execution.

Pressing the Run button starts the web browser in automatic mode (recognizable by the robot icon before the address bar), which performs all actions of the test suite on behalf of the user:

During test execution, the individual test cases are logged in the console and their execution times are displayed at the end:

At the end of the test execution, the result is recorded in a detailed test report. The following data is provided for each test case in the test suite:

  • Start Time : timestamp when the test case was started
  • Duration : execution time of the test case
  • Result : test verdict ( pass , fail , error ) and the number of test steps
  • test case # : number of the test case in test suite (1, ... )
  • file name of the test suite
Further details can be set for each test case in the Show area:
  • Summary : short summary of all test steps
  • Failed : detailed presentation of all test steps having test verdict fail
  • All : detailed presentation of all test



The test report indicates that all test cases of the test suite passed the test with test verdict pass .

Very often testers are interested in testing only a part of the application or at the beginning of a test cycle only the so-called happy path.

For this purpose, the test profile can be specifically adapted in the TestPlayer© dashboard, i.e. the transition probabilities are set in such a way that only parts of the usage model are traversed. The following profile will focus the test suite only on test cases with one-way flights ( p27=0 ) and Visa booking ( p74=0 ) without visiting the states Remember me and Heading New Stuff ( p9=1 ). Other adjustments of probability values in the profile ensure that no backward loops are enabled and that the usage model is traversed in a linear order from the start state [ to the end state ] :

A focused test suite can now be generated in the TestPlayer© Dashboard as follows. First, the model AgileTravel and the profile AgileTravel are selected in the Model settings area. In addition, Profile Usage must be set to yes :



In the Generation settings area, the number of test cases to be generated is set to 10 in the Testcase Definitions tab. This small value is sufficient for the focused test suite. In addition, the strategy fast overview is set as the generation strategy, which produces useful diagrams in addition to the test suite:


By means of the Filemanager , the profile can be checked again before the automatic generation of the test suite is started:

When the automatic generation is finished, the Filemanager can be opened to have access to the test cases in the directory testcases/AgileTravel/ . If, for example, subdirectory 10_sort_l_p is selected, test suites are shown that are sorted according to the length of the test cases and generated via a profile. File statistics.txt indicates, among other things, that 5 test cases are sufficient to cover all usage states and state transitions of the focused test suite:


File test_cases_for_state_coverage.json.txt contains the 5 test cases that are needed for the subsequent test execution:

Additional diagrams were generated in directory diagrams/AgileTravel/10/... by means of the generation strategy fast overview . Below is a diagram that demonstrates a variant of the TestPlayer© visualization. The diagram shows hat only selected usage states are checked during the test, while the blue colored states are not considered during the test:

Below the focused test suite containing 5 test cases is visualized once again. Each additional test case covers at least one new usage state compared to its predecessor:

Afterwards, a focused test of the web application Agile Travel can be performed by using these 5 test cases:

In Eclipse , Run Configurations loads the focused test suite for test execution via the file option -f test_cases/AgileTravel.TestPlayer/10_sort_l_p/test_cases_for_state_coverage.json.txt . During the test execution, individual test cases are logged in the console and the execution times are displayed at the end:



At the end of the test execution, the result is documented in a detailed test report :



The test report indicates that the 5 test cases of the test suite passed the test with test verdict pass .

The following video shows the described procedure and all essential steps once again in a compact representation: