Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
sequencers [2021/02/17 18:03] – [Types of Sequencers] adminsequencers [2024/01/03 21:51] (current) – [Sequencer - AllPairs] admin
Line 5: Line 5:
 The test case generation is referred to as //Generate// action. The test case generation is referred to as //Generate// action.
  
-The generated test cases can be automated to drive Application Under Test (AUT) through //Groovy-based Scripting// and [[Plugins]].  This step is referred to as //Execute// model action.+The generated test cases can be automated to drive Application Under Test (AUT) through //Groovy-based Scripting//.  This step is referred to as //Execute// model action.
  
  
Line 11: Line 11:
  
 =====Types of Sequencers===== =====Types of Sequencers=====
-The algorithms used by the sequencers differ depend on the type of models: //State-based// and //Combinatorial Data//. +Sequencers generate the test sequence by traversing the state diagram using graph-based algorithms.  A graph path from model's initial state to any of the final state is considered a test case.
- +
-For //State-based// models, //Model Sequencer// generates the test sequence by traversing the state diagram using graph-based algorithms.  A graph path from model's initial state to any of the final state is considered a test case.+
  
 Different algorithms achieve different coverage on the model. In combination with //stop conditions//, you can stop the test generation when the desired coverage is achieved. Different algorithms achieve different coverage on the model. In combination with //stop conditions//, you can stop the test generation when the desired coverage is achieved.
- 
-For //Combinatorial Data// models, //Model Sequencer// generates the permutations for the variable value (domain) sets to achieve the specific combination coverage. 
- 
-Constraints (rules) can be added to //Combinatorial Data// models to remove unwanted combinations. 
  
  
Line 28: Line 22:
 The weight of the outgoing transitions determines the probability a transition will be traversed from its source state.  The higher the weight, the more often the transition will be traversed, with a caveat: non-traversed transitions are preferred over already-traversed transitions. The weight of the outgoing transitions determines the probability a transition will be traversed from its source state.  The higher the weight, the more often the transition will be traversed, with a caveat: non-traversed transitions are preferred over already-traversed transitions.
  
-//Random// sequencer is useful for exploratory type of testing by allowing the model to run for an extended period of time to let the algorithm traverse "all possible" paths.+//Random// sequencer is useful for exploratory type of testing by allowing the model to run for an extended period of time to let the algorithm explore "all possible" paths.
  
-//Random// sequencer is also often used for load and stress testing by running model with may //virtual users (VU)// to simulate realistic load of a production systems.+//Random// sequencer is also often used for load and stress testing by running model with many //virtual users (VUs)// to simulate realistic load of a production systems.
  
-Another good use of //Random// sequencer is to find memory leak, locking contention, etc. that are caused by running //AUT// over a long period of time. You can just leave the model running with the elapse time stop condition set to desired duration of time.+Another good use of //Random// sequencer is to find memory leak, locking contention, etc. that are caused by running //AUT// over a long period of time or under heavy load. You can just leave the model running with the elapse time stop condition set to desired duration of time.
  
 The disadvantage of //Random// sequencer is that the model coverage is typically time-dependent - it may take longer time for it to achieve the desired model coverage. The disadvantage of //Random// sequencer is that the model coverage is typically time-dependent - it may take longer time for it to achieve the desired model coverage.
Line 40: Line 34:
  
 =====Sequencer - Optimal===== =====Sequencer - Optimal=====
-//Optimal// sequencer uses uses algorithms for [[https://en.wikipedia.org/wiki/Route_inspection_problem | Route inspection problem (aka Chinese Postman Problem)]] to generate the test cases (paths) that covers all transitions (and thus all states) in the //State-based// models.+//Optimal// sequencer uses the algorithm for [[https://en.wikipedia.org/wiki/Route_inspection_problem | Route inspection problem (aka Chinese Postman Problem)]] to generate the test cases (paths) that covers all transitions (and thus all states) in the //State-based// models.
  
-The obvious advantage of //Optimal// sequencer is that it generates the least number of steps to cover every transitions in the model.  As the result, this sequencer would work better for larger models or if you just want to run all test cases in the least amount of time.+The obvious advantage of //Optimal// sequencer is that it generates the least number of steps to cover all transitions in the model.  As the result, this sequencer would work better for larger models or if you just want to run all test cases in the least amount of time.
  
 The side effect is that the test cases generated by //Optimal// sequencer tends to be longer and fewer test cases with the minimal total number of test steps.  This can be viewed as either an advantage or disadvantage depending on your specific use case. The side effect is that the test cases generated by //Optimal// sequencer tends to be longer and fewer test cases with the minimal total number of test steps.  This can be viewed as either an advantage or disadvantage depending on your specific use case.
Line 51: Line 45:
 //Priority// sequencer prioritizes transitions based on their assigned weight and tries to traversed transitions with higher weight first using graph algorithm to achieve desired model coverage. //Priority// sequencer prioritizes transitions based on their assigned weight and tries to traversed transitions with higher weight first using graph algorithm to achieve desired model coverage.
  
-The test cases generated by //Priority// sequencer usually are shorter than //Optimal// sequencer due to the nature of the algorithm, but the number of test steps to achieve the similar model coverage would be larger.+The test cases generated by //Priority// sequencer usually are shorter than //Optimal// sequencer due to the nature of the algorithm, but the number of test steps to achieve the similar model coverage would be larger compared to //Optimal// sequencer.
  
 //Priority// sequencer might be a good choice if you have certain transitions in the model that you wish to cover earlier and/or prefers shorter test cases. //Priority// sequencer might be a good choice if you have certain transitions in the model that you wish to cover earlier and/or prefers shorter test cases.
Line 65: Line 59:
  
  
-=====Sequencer - Pairwise===== +=====Sequencer - AllPairs===== 
-//Pairwise// sequencer explores combinations of incoming and outgoing transitions of each state in the model and generate test paths that cover all incoming - outgoing transition pairs.  +//AllPairs// sequencer explores combinations of incoming and outgoing transitions of each state in the model and generate test paths that cover all incoming - outgoing transition pairs.  
  
-//Pairwise// sequencer offers more extensive model coverage than most of the sequencers ([[#Sequencer - Optimal | Optimal]] and [[#Sequencer - Priority | Priority]], and others) with a deterministic test sequence.  +//AllPairs// sequencer offers more extensive model coverage than most of the sequencers ([[#Sequencer - Optimal | Optimal]] and [[#Sequencer - Priority | Priority]], and others) with a deterministic test sequence. 
-=====Sequencer - Mark Optimal===== +
-Often times you may want to only test certain part of the model.  //Mark Optimal// sequencer (and [[#Sequencer - Mark Serial| Mark Serial]]) generates test cases to cover the marked transitions.+
  
-//Mark Optimal// sequencer generates the test cases using the same [https://en.wikipedia.org/wiki/Route_inspection_problem | Route inspection problem (aka Chinese Postman Problem)]] as [[#Sequencer - Optimal | Optimal]] sequencer except that only the marked transitions are required to be covered. +=====Sequencer - Marking States/Transitions===== 
- +With IDE, you can mark (highlight) states and transitions and generate test sequence to cover partial of the model.  With ProMBT, you can also submit model execution with REST api.
-//Mark Optimal// sequencer (and [[#Sequencer - Mark Serial| Mark Serial]]) can be used for [[https://en.wikipedia.org/wiki/Exploratory_testing | Exploratory Testing]], e.g. testing a new user stories just added to the model. +
- +
-//Mark Optimal// sequencer is only available when running model in IDE. +
-=====Sequencer - Mark Serial===== +
-//Mark Serial// sequencer is similar to [[#Sequencer - Mark Optimal]], it generates test cases to cover the marked transitions in the model. +
- +
-What distinguishes //Mark Serial// sequencer from //Mark Optimal// sequencer is that the test cases generates follows the order the transitions are marked.  As the result, the test sequence generated by //Mark Serial// sequencer is usually longer. +
- +
-//Mark Serlia// sequencer (and [[#Sequencer - Mark Optimal| Mark Optimal]]) can be used for [[https://en.wikipedia.org/wiki/Exploratory_testing | Exploratory Testing]], e.g. testing a new user stories just added to the model. +
- +
-//Mark Serial// sequencer allows you to test scenarios that require traversing transitions in a specific order. +
- +
- +
-//Mark Optimal// sequencer is only available when running model in IDE.+
  
 =====Sequencer - MCase===== =====Sequencer - MCase=====
-//MCase// is a collection of transitions that represents a custom test case.  //MCase// sequencer generates the actual test sequence (test steps) to navigate through the model to cover the transitions included in the //MCase//. +//MCase// is a collection of transitions that represents a custom test case.  //MCase// sequencer generates the actual test sequence (test steps) to navigate through the model to cover the transitions included in the //MCase//.  This works very similarly as //Marked Serial// sequencer except that the transition collections are defined in scripts and //MCases// can be executed in IDE as well as remotely.
  
 The main use of //MCase// sequencer is to generate custom test cases, which can be used to test a bug fix, regression testing of important workflows and smoke testing. The main use of //MCase// sequencer is to generate custom test cases, which can be used to test a bug fix, regression testing of important workflows and smoke testing.
  
 Since //MCase// is built with scripting, you can dynamically build any number of //MCases// through //REST// api.  This provides an option to allow ALM process to test bug fixes remotely. Since //MCase// is built with scripting, you can dynamically build any number of //MCases// through //REST// api.  This provides an option to allow ALM process to test bug fixes remotely.
- 
- 
-=====Sequencer - Concurrent===== 
-//Concurrent// sequencer is the only sequencer that supports concurrent system models.  //Concurrent// models allows simultaneous firing of multiple transitions from the state and it's used to simulate processing of event-driven system or concurrent systems. 
- 
-===== Combinatorial Algorithms ===== 
-In the ideal world we would want exhausted testing, but we all know that's not practical and in many cases not feasible. 
- 
-//Combinatorial Algorithms// are used to generate test cases for //Combinatorial Data// models. //TestOptimal// supports following combinatorial algorithms: 
- 
-  * pairwise 
-  * 3-wise 
-  * 4-wise 
-  * 5-wise 
-  * 6-wise 
- 
- 
-Pairwise algorithm geneartes the least number of test cases (permutations of variables) while 6-wise generates the most test cases but provides most coverage among the algorithm listed above. 
- 
-Empirically most of defects are caused by the interaction between two variables, hence pairwise algorithm would provide decent coverage with least amount of test cases for most of the system.  This is commonly used especially for larger models as it could potentially cut down the test cases by 99%. 
- 
-