Tutorial: Combined Hybrid Model

Learning Objectives:

• Differences between state model and combinatorial model
• Hybrid model

State model and combinatorial model are two most common types of models used in Model-Based Testing (MBT).

State model is used when you can describe the AUT behavior using a state diagram or finite state machine (FSM). The sequencer is then used to generate test cases (paths from initial state to final states).

Combinatorial model is typically used to generate test cases to test interactions between/among a set of variables. The algorithm is then used to generate variable permutations to achieve the desired degree of interaction strength. Most commonly used algorithm is Pairwise.

State model focuses on system behavior while combinatorial model focuses on static variable interaction.

Often times you may find that in your state model you need to test a set of variable interactions, for example, to test a online web store application, besides testing the navigation between pages, you also need to test all permutations of:

• payment types: visa, master, discover, american express, gift card
• promotion code: no code/blank, 10% discount code, 20% discount code, …
• product warranty offer: accepted (checked), denied (unchecked)
• shipping methods: 2-day, next-day, same-day (rush)

So state model would be good to generate test cases to navigate web pages while combinatorial model would be good to test permutation of various options at check out page.

If you find similar situation, the hybrid model by bringing combinatorial model into your state model would allow you to perform both testing in one model.

You can embed combinatorial model in a state model as data sets in the state model and reference the data sets in the state model:

• create state model
• add data sets and generate data table
• attach data set to transition or
• loop through data set programmatically in scripts

First create a state model with a transition named “CheckOut”, the transition we will use to attach data sets to.

If you need help creating the state model, you may refer to Create State Model tutorial.

You can also use any of your existing state model as well, just select a transition from the model which we will attach the data sets.

Open the state model and select DATASET tab. You should see the following:

As shown above, a new data set has been created for us and assigned a default name “NEW_DS_01”.

Let's rename the data set to “PaymentOptions”: hover mouse over the data set name:

click on the pencil icon to open dataset rename dialog:

Enter new data set name “PaymentOptions”:

The data set created has no variables, let's proceed to add variables.

As stated earlier, the variables we would like to test all combinations of are:

• PaymentType: VISA, MASTER, DISCCOVER, AMERICAN, GIFTCARD
• CardNum: 4111-1111-1111-1111 (visa), 5500-0000-0000-0004 (master), 6011-0000-0000-0004(Discover), 3400-0000-0000-009 (American Express), 2123-222-2123 (gift card)
• PromoCode: blank, 10%OFF, 20%OFF, $5OFF, 15$OFF
• Warranty: true, false
• Shipping: 2 (2-day), 1 (next-day), 0 (same-day)

To add a variable click on “+” button:

and you should see a variable is added:

Go ahead and enter first variable name “PaymentType” in “Name” field and values to be tested in “Domain” field - one value per line as shown below:

Note that the default data type is “text”, which is good for “PaymentType”.

Proceed to add the rest of the variables as below - be sure to set the data type accordingly:

Click on the orange disk button to save the changes before proceeding to the next step.

To generate the combinations of the variables, select an algorithm from the “Overall Strength” dropdown list on the upper-right corner of the screen:

Select “Pairwise” for now and click the generate button next to it. You should see the DataTable populated with rows of data:

The system generated 25 combinations (rows). Since Pairwise was used to generate the combinations, all pairs of any two variables should be represented by at least one row in the DataTable.

You may have noticed that “PaymentType” and “CardNum” are messed up, for example, we are seeing payment type of VISA has the non-VISA card number, which is not desired.

We need to make “CardNum” variable to match variable “PaymentType”. To fix this issue, on variable “CardNumselect select “PaymentType” from the “Coupling” dropdown list as show below and re-generate DataTable:

Now the payment types and credit card numbers are matching correctly.

Feel free to experiment with other algorithms and compare the # of rows generated in DataTable by different algorithms.

Next step is to attach this data set to the “CheckOut” transition on the model.

In this step, we will attach the data set “PaymentOptions” we have created to the transition “CheckOut” in our model.

Open the transition properties by double-clicking on the transition label or through transition context menu and select “PaymentOptions” from the “Data Set” dropdown list:

When a data set is attached to the transition, the sequencer that generates test cases from the model will ensure the test cases generates will cover the transition the number of times to loop through all of the rows in the data set.

Save the changes. We are now ready to generate the test cases.

Select “Optimal” sequencer (default) and run the model. Our expectation is that at least 25 test cases would be generated from the state model with each test case covering “CheckOut” transition to consume a row from the data set.

The best way to show this is through Test Case Graph in RESULT tab.

Notice that in Test Case Graph, there are 25 test cases and each test case covers the transition “CheckOut” matching 25 rows in the data set.

To confirm that each row in data set is used, let's print out the data row as model runs - the test cases generated and executed.

To write a line of script in TRIGGER for transition “CheckOut”, select script context menu for transition “CheckOut”:

IDE should navigate to SCRIPT tab. Since the TRIGGER for transition “CheckOut” has not been created, you will be prompted to create it:

Click “OK” to create the skeleton TRIGGER script for the transition:

Alternatively you can add the skeleton TRIGGER script for the transition by pressing Ctrl-I while your cursor is placed on a blank line in SCRIPT tab and select transition “CheckOut”:

TRIGGER script is just a groovy function, go ahead and add two lines of scripts as show below:

The “log” scripts above prints the current row of the data set and “Payment Type” and “CardNum” to Script Log.

Save the changes and run the model again, and examine Script Log by clicking on the script log button in RESULT tab:

Below is the first few lines from the script log:

 C:\eclipse-workspace\TestOptimal_Dev\model\Demo_DataSet.fsm\temp\script.log
 
 >>> Date: 2020-12-30 
 20:51:25.915 data row [0]: [PaymentType:VISA, CardNum:4111-1111-1111-1111, Warranty:false, Shipping:0, PromoCode:Blank, $$hashKey:object:444]
 20:51:25.916 PaymentType: VISA, CardNum: 4111-1111-1111-1111
 20:51:25.917 data row [1]: [PaymentType:VISA, CardNum:4111-1111-1111-1111, Warranty:true, Shipping:1, PromoCode:10%OFF, $$hashKey:object:445]
 20:51:25.918 PaymentType: VISA, CardNum: 4111-1111-1111-1111
 20:51:25.918 data row [2]: [PaymentType:VISA, CardNum:4111-1111-1111-1111, Warranty:false, Shipping:2, PromoCode:20%OFF, $$hashKey:object:446]
 20:51:25.918 PaymentType: VISA, CardNum: 4111-1111-1111-1111
 20:51:25.919 data row [3]: [PaymentType:VISA, CardNum:4111-1111-1111-1111, Warranty:false, Shipping:0, PromoCode:$5OFF, $$hashKey:object:447]
 20:51:25.919 PaymentType: VISA, CardNum: 4111-1111-1111-1111
 20:51:25.919 data row [4]: [PaymentType:VISA, CardNum:4111-1111-1111-1111, Warranty:true, Shipping:0, PromoCode:15$OFF, $$hashKey:object:448]
 20:51:25.919 PaymentType: VISA, CardNum: 4111-1111-1111-1111
 20:51:25.919 data row [5]: [PaymentType:MASTER, CardNum:5500-0000-0000-0004 , Warranty:true, Shipping:1, PromoCode:Blank, $$hashKey:object:449]
 20:51:25.919 PaymentType: MASTER, CardNum: 5500-0000-0000-0004 
 20:51:25.920 data row [6]: [PaymentType:MASTER, CardNum:5500-0000-0000-0004 , Warranty:false, Shipping:2, PromoCode:10%OFF, $$hashKey:object:450]
 20:51:25.920 PaymentType: MASTER, CardNum: 5500-0000-0000-0004 
 20:51:25.920 data row [7]: [PaymentType:MASTER, CardNum:5500-0000-0000-0004 , Warranty:true, Shipping:0, PromoCode:20%OFF, $$hashKey:object:451]
 20:51:25.920 PaymentType: MASTER, CardNum: 5500-0000-0000-0004 
 20:51:25.921 data row [8]: [PaymentType:MASTER, CardNum:5500-0000-0000-0004 , Warranty:false, Shipping:1, PromoCode:$5OFF, $$hashKey:object:452]
 20:51:25.921 PaymentType: MASTER, CardNum: 5500-0000-0000-0004 
 20:51:25.921 data row [9]: [PaymentType:MASTER, CardNum:5500-0000-0000-0004 , Warranty:false, Shipping:0, PromoCode:15$OFF, $$hashKey:object:453]
 20:51:25.921 PaymentType: MASTER, CardNum: 5500-0000-0000-0004 
 20:51:25.922 data row [10]: [PaymentType:DISCOVER, CardNum:6011-0000-0000-0004, Warranty:true, Shipping:0, PromoCode:Blank, $$hashKey:object:454]
 20:51:25.922 PaymentType: DISCOVER, CardNum: 6011-0000-0000-0004