Model Sequencers

Model Sequencer is algorithm-based test case (test sequence) generation from the model.

The test case generation is referred to as Generate action.

The generated test cases can be automated to drive Application Under Test (AUT) through Groovy-based Scripting and Plugins. This step is referred to as Execute model action.

The algorithms used by the sequencers differ depend on the type of models: State-based and Combinatorial Data.

For State-based models, Model Sequencer generates the test sequence by traversing the state diagram using graph-based algorithms. A graph path from model's initial state to any of the final state is considered a test case.

Different algorithms achieve different coverage on the model. In combination with stop conditions, you can stop the test generation when the desired coverage is achieved.

For Combinatorial Data models, Model Sequencer generates the permutations for the variable value (domain) sets to achieve the specific combination coverage.

Constraints (rules) can be added to Combinatorial Data models to remove unwanted combinations.

Random sequencer traverses State-based models using the variation from the markov-chain process to construct test paths from iniital state to final states.

The weight of the outgoing transitions determines the probability a transition will be traversed from its source state. The higher the weight, the more often the transition will be traversed, with a caveat: non-traversed transitions are preferred over already-traversed transitions.

Random sequencer is useful for exploratory type of testing by allowing the model to run for an extended period of time to let the algorithm explore “all possible” paths.

Random sequencer is also often used for load and stress testing by running model with many virtual users (VUs) to simulate realistic load of a production systems.

Another good use of Random sequencer is to find memory leak, locking contention, etc. that are caused by running AUT over a long period of time or under heavy load. You can just leave the model running with the elapse time stop condition set to desired duration of time.

The disadvantage of Random sequencer is that the model coverage is typically time-dependent - it may take longer time for it to achieve the desired model coverage.

Optimal sequencer uses the algorithm for Route inspection problem (aka Chinese Postman Problem) to generate the test cases (paths) that covers all transitions (and thus all states) in the State-based models.

The obvious advantage of Optimal sequencer is that it generates the least number of steps to cover all transitions in the model. As the result, this sequencer would work better for larger models or if you just want to run all test cases in the least amount of time.

The side effect is that the test cases generated by Optimal sequencer tends to be longer and fewer test cases with the minimal total number of test steps. This can be viewed as either an advantage or disadvantage depending on your specific use case.

Priority sequencer prioritizes transitions based on their assigned weight and tries to traversed transitions with higher weight first using graph algorithm to achieve desired model coverage.

The test cases generated by Priority sequencer usually are shorter than Optimal sequencer due to the nature of the algorithm, but the number of test steps to achieve the similar model coverage would be larger.

Priority sequencer might be a good choice if you have certain transitions in the model that you wish to cover earlier and/or prefers shorter test cases.

This sequencer supports following parameters to adjust the sensitivity on choosing higher priority transitions and over traversed transitions:

• TraversedTransCost: integer, default 500
• scale: float >= 0.1, default 1.0 = neutral, < 1.0 for less sensitive (less penalty on choosing lower weight transitions), > 1.0 for more sensitive

You may also set above parameters in config.properties for all models:

  Sequencer.TraversedTransCost=integer
  Sequencer.scale=float

Pairwise sequencer explores combinations of incoming and outgoing transitions of each state in the model and generate test paths that cover all incoming - outgoing transition pairs.

Pairwise sequencer offers more extensive model coverage than most of the sequencers ( Optimal and Priority, and others) with a deterministic test sequence.

Often times you may want to only test certain part of the model. Mark Optimal sequencer (and Mark Serial) generates test cases to cover the marked transitions.

Mark Optimal sequencer generates the test cases using the same [https://en.wikipedia.org/wiki/Route_inspection_problem | Route inspection problem (aka Chinese Postman Problem)]] as Optimal sequencer except that only the marked transitions are required to be covered.

Mark Optimal sequencer (and Mark Serial) can be used for Exploratory Testing, e.g. testing a new user stories just added to the model.

Mark Optimal sequencer is only available when running model in IDE.

Mark Serial sequencer is similar to Sequencer - Mark Optimal, it generates test cases to cover the marked transitions in the model.

What distinguishes Mark Serial sequencer from Mark Optimal sequencer is that the test cases generates follows the order the transitions are marked. As the result, the test sequence generated by Mark Serial sequencer is usually longer.

Mark Serlia sequencer (and Mark Optimal) can be used for Exploratory Testing, e.g. testing a new user stories just added to the model.

Mark Serial sequencer allows you to test scenarios that require traversing transitions in a specific order.

Mark Optimal sequencer is only available when running model in IDE.

MCase is a collection of transitions that represents a custom test case. MCase sequencer generates the actual test sequence (test steps) to navigate through the model to cover the transitions included in the MCase.

The main use of MCase sequencer is to generate custom test cases, which can be used to test a bug fix, regression testing of important workflows and smoke testing.

Since MCase is built with scripting, you can dynamically build any number of MCases through REST api. This provides an option to allow ALM process to test bug fixes remotely.

Concurrent sequencer is the only sequencer that supports concurrent system models. Concurrent models allows simultaneous firing of multiple transitions from the state and it's used to simulate processing of event-driven system or concurrent systems.

In the ideal world we would want exhausted testing, but we all know that's not practical and in many cases not feasible.

Combinatorial Algorithms are used to generate test cases for Combinatorial Data models. TestOptimal supports following combinatorial algorithms:

• pairwise
• 3-wise
• 4-wise
• 5-wise
• 6-wise

Pairwise algorithm geneartes the least number of test cases (permutations of variables) while 6-wise generates the most test cases but provides most coverage among the algorithm listed above.

Empirically most of defects are caused by the interaction between two variables, hence pairwise algorithm would provide decent coverage with least amount of test cases for most of the system. This is commonly used especially for larger models as it could potentially cut down the test cases by 99%.