Data Design IDE

Data Design IDE is the development and execution (automation) of combinatorial test design models. This is typically used to generate a set of minimal permutations of variables to achieve the desired coverage using Combinatorial Testing.

Most of the software fault are caused by the interactions of 2-3 variables ( NIST).

By using the appropriate combinatorial algorithm, you could cut down the number of test cases by 90% and still achieve sufficient test coverage.

There are 4 tabs in this IDE that matches the development steps:

• DEFINE
• GENERATE
• SCRIPT
• EXECUTE

In DEFINE, you define the testing variables as well as the expected result variables.

• Variable Name - variables must have unique names and preferably legal java variable names
• Data Type - appropriate data type must be selected especially if the variables are referenced by Derived variables
• Domain Values - a list of discrete values to be used to generate (populate) GENERATE or groovy/java expression if Derived is checked
• Derived - check this box if the value of the variable depends on values of other variables. Derived variable usually have static value or expression that evaluates to a value specified in Domain attribute.
• Coupling - used to couple / tie this variable to another variable. For example CreditCardType and CreditCardNumber should be coupled together as CreditCardNumber for VISA must match CreditCardType of VISA. Variables coupled together must have the same number of Domain values.

Domain values for non-Derived variables must be static values.

For Derived variables, Domain must have exactly one value, which can be static value or groovy expression. Below is a simple example to calculate the insurance premium credit of $100 given to good student age < 20:

 (Age < 20 && GoodStudentStatus)?100:0
 

You may use plugin reference in Domain value/expression which will be evaluated during EXECUTE. For example:

 $VAR.adminUsername              $VAR.adminPassword
 $VAR.normalUsername             $VAR.normalPassword
 $VAR.supervisorUsername       $VAR.supervisorPassword
 

For more complex scenario:

 $UTIL.toJSON($VAR.order1Obj)

where order1Obj is a groovy/java object that is defined/created in DATA SCRIPT.

Use [blank] to encode blank string and [null] for null string/object.

When designing test data using combinatorial testing techniques, we often need to remove or avoid certain combinations as they are either unfeasible or unnecessary for whatever reason. You can remove these unwanted combinations manually after the test combinations have been generated, but that's tedious and prone to errors. Depending on number of test combinations generated, manually removing unwanted combinations may even be impossible due to large volume.

The solution is to create a set of constraints or rules to filter out certain permutations.

Constraints are expressed as a simple expression:

 IF condition THEN assertion

The condition and assertion are simple boolean expressions (NOTE: this is not groovy expression):

 [variable name] [operator] [value or [variable name] [&& or ||] ... 

For example:

 IF (OS = "Windows") THEN (Browser = "IE" || Browser = "FireFox" || Browser = "Netscape")

The operator can be:

• > : greater than, numeric fields only
• < : less than, numeric fields only
• = : equals
• != : not equals
• >= : greater than or equal, numeric fields only
• <= : less than or equal, numeric fields only

Arithmetic operations are also supported:

 IF DEPOSIT_AMOUNT - WITHDRAW_AMOUNT <= 0 THEN REJECT_TRANSATION = true
 

Only numeric variables can participate in the arithmetic operations. Following arithmetic operations are supported:

• + : addition
• - : substration
• * : multiplication
• / : division
• % : mod operation

When comparing with text, the text must be enclosed with double quotes (“).

Numeric constants may be used in the comparison, for example:

 IF AGE >= 19 THEN ADULT = true
 

Parenthesis can be used to help organize complex expression, for example:

 IF (WINTER=true && TEMPERATURE > 30) || (SUMMER=true && TEMPERATURE < 90) THEN GOOD_WEATHER = true

Static values specified in comparison must be a valid value for the variable or it will result in errors during generation.

To generate DataTable, select an algorithm from Overall Strength drop-down list and click start button next to it.

• pairwise
• 3-wise
• 4-wise
• 5-wise
• 6-wise
• Mixed

You may customize /edit DataTable under exceptional cases.

• remove rows - invalid permutations (rows)
• add rows - additional required permutations
• edit values - change value, usually done for derived variables

Be aware that any changes to DataTable will affect the coverage, and the changes made will be lost next time DataTable is re-generated. In most cases, the customization can be accomplished with Constraints and Derived as described in DEFINE above.

By default, Overall Strength applies to all variables excluding Derived variables. In cases you wish to apply more rigorous testing than other variables. You can do so by selecting Mixed in Overall Strength and and apply specific level of strength for each subset of the variables.

After selecting Mixed strength, section Variable Interaction (Mixed Overall Strength only) appears. Click on ”+“ to add a custom interaction/strength set, select the desired strength/algorithm and select the variables respectively.

Script to be executed to process rows in DataTable. As shown above, the script just loops through all rows in $DATASET which represents the DataTable and perform REST calls to test REST API on every row in DataTable and logs the results.

$DATASET.notifyClient(row) sends the result back to IDE. If the execution was started with REST api, the result will be written to the output file specified in the REST request.

Refer to Script Editor for more information about this script editor.

There are two types of scripts: MAIN and DATA.

MAIN script is the main program which executes the test cases.

DATA script is used to instantiate and create objects (groovy/java) to be used in test case execution. For example, you may create a purchase order object to be posted to REST api:

 $VAR.order1 = [product: 'cookie', price: 2.00, quantity: 20]

The order object “order1” is assigned to $VAR. It can then be used in Domain expression as:

 $UTIL.toJSON($VAR.order1)

INIT script is for initialization of variables, e.g. environment, user name, password, etc., the variables that may need to be changed from execution to execution.

As script executes, IDE receives and displays the execution result one row at a time in the tabular format.

The header shows the execution progress. The execution results can also be exported to tab delimited file - click on the export button on the header.

The executions currently are not counted in Dashboard stats.

Executions can be archived/saved for future access.

Historical executions can be accessed by clicking on Past link.