Tutorial: Combinatorial Modeling and Test Generation

• What is Combinatorial Model?
• Create Model
• Generate Test Cases
• Make Changes

Combinatorial Model is an abstraction and representation of a combinatorial problem.

Combinatorial Model consists of a list of variables with each variable having a set of domain values to be tested. The objective is to produce the least number of permutations of variable values to achieve the desired degree of interactions among variables.

Consider three variables with each variable has a domain of 10 possible values. If we were to test all possible permutations, we would have to create 10x10x10 = 1000 test cases. We would rarely do this as full permutations in most cases is overkill, unnecessary, and sometimes impossible.

So if we were to reduce the test cases to test, which of the 1000 test cases can we cut?

According to NIST research showed that most software bugs and failures are caused by one or two parameters (variables), with progressively fewer by three or more.

TestOptimal provides a set of combinatorial algorithms to produce the least number of permutations to cover the desired degree of interactions. After the model is built, we can choose the appropriate algorithms to generate sufficient test cases to meet our testing needs.

In the example above, suppose we want to ensure every pair of variables are tested, we could use Pairwise algorithm to generate the test cases and the result is 99 test cases, which is 10% of 1000 test cases.

To create a combinatorial model, clicki on on the toolbar or clicking on menu File / Open/New Model:

The Menu File List opens:

By default it opens at the root folder model. Navigate to the folder that you want the new model to be stored and click on to create combinatorial model.

Enter the model name at the New Model prompt:

The new model is open in DEFINE tab like this:

The first step is to define all of the variables and set the domain values for each variable. This is done in DEFINE tab:

1. click on
2. select the appropriate data type: int, float, txt or bool
3. enter domain values one value on each line

You should have something like this:

Go ahead and create two more variables and we have the model built:

You can also add constraints to remove certain permutations that are illegal, invalid or unwanted for any reason. This is not covered in this tutorial, please refer to the Data Design IDE / DEFINE for details.

Save the changes and we are ready to generates the test cases from this model.

Test generation from combinatorial model is done in GENERATE tab. Go ahead and select GENERATE tab.

To generate the test cases,

• select an algorithm

• click on ▶ next to the algorithm dropdown

Generated test cases are listed in DataTable as below for pairwise:

You may remove or add additional rows to the DataTable as you see fit, even to update values for any rows. However this must be done with caution as any changes to DataTable will be lost on next time DataTable is re-generated.

Feel free to experiment with different generation algorithms. You must save the changes before you run another generation.

When selecting Mixed algorithm, you will see an additional section displayed below the header (variable names):

Mixed algorithm allows you to selectively apply different interaction strength (pairwise vs three-wise, etc.) for specific subset of variables.

We will not cover Mixed algorithm here, but you can find more details about Mixed algorithm in Data Design IDE / GENERATE.

As you must have figured out that you can easily add additional variables and remove any variables from the model.

Other changes are adding, removing and changing the domain values of each variable.

Upon any changes to the model, you should always re-generate DataTable.