Data Design IDE is the development and execution (automation) of combinatorial test design models. This is typically used to generate a set of minimal permutations of variables to achieve the desired coverage using Combinatorial Testing.

Most of the software fault are caused by the interactions of 2-3 variables ( NIST).

By using the appropriate combinatorial algorithm, you could cut down the number of test cases by 90% and still achieve sufficient test coverage.

There are 4 tabs in this IDE that matches the development steps:

• DEFINE
• GENERATE
• SCRIPT
• EXECUTE

In DEFINE, you define the testing variables as well as the expected result variables.

• Variable Name - variables must have unique names and preferably legal java variable names
• Data Type - appropriate data type must be selected especially if the variables are referenced by Derived variables
• Domain Values - a list of discrete values to be used to generate (populate) DataTable or groovy/java expression if Derived is checked
• Derived - check this box to exclude the variable from participating in the data generation. For example the variables you will use to validate result or variables that can be derived from other variables. If you
• Coupling - used to couple / tie this variable to another variable. For example CreditCardType and CreditCardNumber should be coupled together as CreditCardNumber for VISA must match CreditCardType of VISA. Variables coupled together must have the same number of Domain values.

When designing test data using combinatorial testing techniques, we often need to remove or avoid certain combinations as they are either unfeasible or unnecessary for whatever reason. You can certainly remove them manually after the test combinations have been generated, but that's tedious and prone to errors. Depending on number of test combinations generated manually removing unwanted combinations may even be impossible due to large volume.

The solution is to create a set of constraints or rules to filter out certain permutations.

Constraints are expressed as a simple expression:

 IF condition THEN assertion

The condition and assertion are simple boolean expressions:

 [variable name] [operator] [value or [variable name] [&& or ||] ... 

For example:

 IF (OS = "Windows") THEN (Browser = "IE" || Browser = "FireFox" || Browser = "Netscape")

The operator can be:

• > : greater than, numeric fields only
• < : less than, numeric fields only
• = : equals
• != : not equals
• >= : greater than or equal, numeric fields only
• <= : less than or equal, numeric fields only

Arithmetic operations are also supported:

 IF DEPOSIT_AMOUNT - WITHDRAW_AMOUNT <= 0 THEN REJECT_TRANSATION = true
 

Only numeric variables can participate in the arithmetic operations. Following arithmetic operations are supported:

• + : addition
• - : substration
• * : multiplication
• / : division
• % : mod operation

When comparing with text, the text must be enclosed with double quotes (“).

Numeric constants may be used in the comparison, for example:

 IF AGE >= 19 THEN ADULT = true
 

Parenthesis can be used to help organize complex expression, for example:

 IF (WINTER=true && TEMPERATURE > 30) || (SUMMER=true && TEMPERATURE < 90) THEN GOOD_WEATHER = true

—-

To generate DataTable, select an algorithm from Overall Strength drop-down list and click start button next to it.

• pairwise
• 3-wise
• 4-wise
• 5-wise
• 6-wise
• Mixed

You may customize /edit DataTable under exceptional cases.

• remove rows - invalid permutations (rows)
• add rows - additional required permutations
• edit values - change value, usually done for derived variables

Be aware that any changes to DataTable will affect the coverage, and the changes made will be lost next time DataTable is re-generated. In most cases, the customization can be accomplished with Constraints and Derived as described in DEFINE above.

By default, Overall Strength applies to all variables excluding Derived variables. In cases you wish to apply more rigorous testing than other variables. You can do so by selecting Mixed in Overall Strength and and apply specific level of strength for each subset of the variables.

Script to be executed to process rows in DataTable. As shown above, the script just loops through all rows in $DATASET which represents the DataTable and perform REST calls to test REST API on every row in DataTable and logs the results.

$DATASET.notifyClient(row) sends the result back to IDE. If the execution was started with REST api, the result will be written to the output file specified in the REST request.

Refer to Script Editor for more information about this script editor.

As script executes, IDE receives and displays the execution result one row at a time in the tabular format.

The header shows the execution progress. The execution results can also be exported to tab delimited file - click on the export button on the header.

The executions currently are not counted in Dashboard stats.