Table of Contents

Data Design IDE

Data Design IDE is the development and execution (automation) of combinatorial test design models. This is typically used to generate a set of minimal permutations of variables to achieve the desired coverage using Combinatorial Testing.

Most of the software fault are caused by the interactions of 2-3 variables ( NIST).

By using the appropriate combinatorial algorithm, you could cut down the number of test cases by 90% and still achieve sufficient test coverage.

There are 4 tabs in this IDE that matches the development steps:

DEFINE

Variables

In DEFINE, you define the testing variables as well as the expected result variables.

Domain Expression

Domain values for non-Derived variables must be static values.

For Derived variables, Domain must have exactly one value, which can be static value or groovy expression. Below is a simple example to calculate the insurance premium credit of $100 given to good student age < 20: 

 (Age < 20 && GoodStudentStatus)?100:0

You may use plugin reference in Domain value/expression which will be evaluated during EXECUTE. For example: 

 $VAR.adminUsername              $VAR.adminPassword
 $VAR.normalUsername             $VAR.normalPassword
 $VAR.supervisorUsername       $VAR.supervisorPassword

For more complex scenario: 

 $UTIL.toJSON($VAR.order1Obj)

where order1Obj is a groovy/java object that is defined/created in DATA SCRIPT.

Use [blank] to encode blank string and [null] for null string/object.

Constraints / Rules

When designing test data using combinatorial testing techniques, we often need to remove or avoid certain combinations as they are either unfeasible or unnecessary for whatever reason. You can remove these unwanted combinations manually after the test combinations have been generated, but that's tedious and prone to errors. Depending on number of test combinations generated, manually removing unwanted combinations may even be impossible due to large volume.

The solution is to create a set of constraints or rules to filter out certain permutations.

Constraints are expressed as a simple expression: 

 IF condition THEN assertion

The condition and assertion are simple boolean expressions (NOTE: this is not groovy expression): 

 [variable name] [operator] [value or [variable name] [&& or ||] ...

For example: 

 IF (OS = "Windows") THEN (Browser = "IE" || Browser = "FireFox" || Browser = "Netscape")

The operator can be:

Arithmetic operations are also supported: 

 IF DEPOSIT_AMOUNT - WITHDRAW_AMOUNT <= 0 THEN REJECT_TRANSATION = true

Only numeric variables can participate in the arithmetic operations. Following arithmetic operations are supported:

When comparing with text, the text must be enclosed with double quotes (“).

Numeric constants may be used in the comparison, for example: 

 IF AGE >= 19 THEN ADULT = true

Parenthesis can be used to help organize complex expression, for example: 

 IF (WINTER=true && TEMPERATURE > 30) || (SUMMER=true && TEMPERATURE < 90) THEN GOOD_WEATHER = true

Static values specified in comparison must be a valid value for the variable or it will result in errors during generation.

GENERATE

To generate DataTable, select an algorithm from Overall Strength drop-down list and click start button next to it.

You may customize /edit DataTable under exceptional cases.

Be aware that any changes to DataTable will affect the coverage, and the changes made will be lost next time DataTable is re-generated. In most cases, the customization can be accomplished with Constraints and Derived as described in DEFINE above.

Mixed Strengths

By default, Overall Strength applies to all variables excluding Derived variables. In cases you wish to apply more rigorous testing than other variables. You can do so by selecting Mixed in Overall Strength and and apply specific level of strength for each subset of the variables.

After selecting Mixed strength, section Variable Interaction (Mixed Overall Strength only) appears. Click on ”+“ to add a custom interaction/strength set, select the desired strength/algorithm and select the variables respectively.

SCRIPT

Script to be executed to process rows in DataTable. As shown above, the script just loops through all rows in $DATASET which represents the DataTable and perform REST calls to test REST API on every row in DataTable and logs the results.

$DATASET.notifyClient(row) sends the result back to IDE. If the execution was started with REST api, the result will be written to the output file specified in the REST request.

Refer to Script Editor for more information about this script editor.

There are two types of scripts: MAIN and DATA.

MAIN script is the main program which executes the test cases.

DATA script is used to instantiate and create objects (groovy/java) to be used in test case execution. For example, you may create a purchase order object to be posted to REST api: 

 $VAR.order1 = [product: 'cookie', price: 2.00, quantity: 20]

The order object “order1” is assigned to $VAR. It can then be used in Domain expression as: 

 $UTIL.toJSON($VAR.order1)

INIT script is for initialization of variables, e.g. environment, user name, password, etc., the variables that may need to be changed from execution to execution.

EXECUTE

As script executes, IDE receives and displays the execution result one row at a time in the tabular format.

The header shows the execution progress. The execution results can also be exported to tab delimited file - click on the export button on the header.

The executions currently are not counted in Dashboard stats.

Executions can be archived/saved for future access.

Historical executions can be accessed by clicking on Past link.