Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
tutorial:comb_data_modeling [2020/06/10 03:45]
admin 		tutorial:comb_data_modeling [2021/01/05 03:00] (current)
Line 1: Line 1:
 ====== Tutorial: Combinatorial Modeling and Test Generation ====== ====== Tutorial: Combinatorial Modeling and Test Generation ======
  
-Learning Objectives:+====Learning Objectives ====
   * [[#What is Combinatorial Model?]]   * [[#What is Combinatorial Model?]]
-  * [[#Build Model]]+  * [[#Create Model]]
   * [[#Generate Test Cases]]   * [[#Generate Test Cases]]
   * [[#Make Changes]]   * [[#Make Changes]]
Line 10: Line 10:
 Combinatorial Model is an abstraction and representation of a combinatorial problem. Combinatorial Model is an abstraction and representation of a combinatorial problem.
  
-Combinatorial Model consists of a list of variables with each variable having a set of domain values to be tested. The objective of creating the combinatorial model is to produce a list of variable interactions to ensure all possible interactions among all variables are covered.+Combinatorial Model consists of a list of variables with each variable having a set of domain values to be tested. The objective is to produce the least number of permutations of variable values to achieve the desired degree of interactions among variables. 
 + 
 +Consider three variables with each variable has a domain of 10 possible values.  If we were to test all possible permutations, we would have to create 10x10x10 = 1000 test cases.  We would rarely do this as full permutations in most cases is overkill, unnecessary, and sometimes impossible.  
 + 
 +So if we were to reduce the test cases to test, which of the 1000 test cases can we cut? 
 + 
 +According to {{https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software | NIST research}} showed that most software bugs and failures are caused by one or two parameters (variables), with progressively fewer by three or more. 
 + 
 +//TestOptimal// provides a set of combinatorial algorithms to produce the least number of permutations to cover the desired degree of interactions.  After the model is built, we can choose the appropriate algorithms to generate sufficient test cases to meet our testing needs. 
 + 
 +In the example above, suppose we want to ensure every pair of variables are tested, we could use Pairwise algorithm to generate the test cases and the result is 99 test cases, which is 10% of 1000 test cases. 
 + 
 +  
 + 
 +===== Create Model ===== 
 +To create a combinatorial model, clicki on {{wiki:overview:ide_btn_open.png?16}} on the toolbar or clicking on menu //File / Open/New Model//: 
 + 
 +{{wiki:overview:ide_menu_file.png?200}} 
 + 
 +The //Menu File List// opens: 
 + 
 +{{wiki:overview:tut_new_model.png?200}} 
 + 
 +By default it opens at the root folder //model//.  Navigate to the folder that you want the new model to be stored and click on {{wiki:overview:tut_comb_btn_new.png?20}} to create combinatorial model. 
 + 
 +Enter the model name at the New Model prompt: 
 + 
 +{{wiki:overview:ide_newmodel_prompt.png?300}} 
 + 
 +The new model is open in //DEFINE// tab like this: 
 + 
 +{{wiki:overview:tut_comb_new_model.png?600}} 
 + 
 + 
 +The first step is to define all of the variables and set the domain values for each variable. This is done in //DEFINE// tab: 
 +  - click on {{wiki:overview:tut_comb_btn_plus.png?20}} 
 +  - select the appropriate data type: int, float, txt or bool 
 +  - enter domain values one value on each line 
 + 
 +You should have something like this: 
 + 
 +{{wiki:overview:tut_comb_new_model_var1.png?150}} 
 + 
 +Go ahead and create two more variables and we have the model built: 
 + 
 +{{wiki:overview:tut_comb_new_model_var3.png?500}} 
 + 
 +You can also add constraints to remove certain permutations that are illegal, invalid or unwanted for any reason.  This is not covered in this tutorial, please refer to the [[../data_design_ide#define | Data Design IDE / DEFINE]] for details. 
 + 
 +Save the changes and we are ready to generates the test cases from this model. 
  
-===== Build Model ===== 
  
  
 ===== Generate Test Cases ===== ===== Generate Test Cases =====
 +Test generation from combinatorial model is done in [[../data_design_ide#generate | GENERATE]] tab.  Go ahead and select //GENERATE// tab.
 +
 +{{wiki:overview:tut_comb_new_model_gen.png?600}}
 +
 +To generate the test cases,
 +  * select an algorithm {{wiki:overview:tut_comb_algorithms.png?150}}
 +
 +  * click on ▶ next to the algorithm dropdown
 +
 +Generated test cases are listed in //DataTable// as below for //pairwise//:
 +
 +{{wiki:overview:tut_comb_new_model_pairwise_gen.png?600}}
 +
 +You may remove or add additional rows to the //DataTable// as you see fit, even to update values for any rows. However this must be done with caution as any changes to //DataTable// will be lost on next time //DataTable// is re-generated.
 +
 +Feel free to experiment with different generation algorithms.  You must save the changes before you run another generation.
 +
 +When selecting //Mixed// algorithm, you will see an additional section displayed below the header (variable names):
 +
 +{{wiki:overview:tut_comb_new_model_gen_mixed.png?600}}
 +
 +//Mixed// algorithm allows you to selectively apply different interaction strength (pairwise vs three-wise, etc.) for specific subset of variables.
 +
 +We will not cover //Mixed// algorithm here, but you can find more details about //Mixed// algorithm in [[../data_design_ide#generate | Data Design IDE / GENERATE]].
 + 
 +
 +
  
  
 ===== Make Changes ===== ===== Make Changes =====
 +
 +As you must have figured out that you can easily add additional variables and remove any variables from the model.
 +
 +Other changes are adding, removing and changing the domain values of each variable.
 +
 +Upon any changes to the model, you should always re-generate //DataTable//.
  
  
  
-===== =====