TestOptimal Tutorials

Dashboard - Requirements, KPI and Reports

Dashboard

This tutorial will cover:

  • Overview
  • Dashboard Charts
  • Reports
  • Create & Publish Projects
  • Create & Publish KPIs
  • Aggregate Setup (ETL)

Dashboard - Overview

  • TestOptimal Dashboard provides an aggregated view of MBT executions from all ProMBT, RuntimeMBT, Enterprise TestOptimal servers.
  • It provides a configurable Key Performance Indicators (KPI)
  • Various charts show test case execution, requirements coverage and defects raised by different categories.
  • Historical trend graphs clearly show the progress and state over time on several key performance measurements.
  • Various reports provide indepth details of MBT executions, how and where in the model each requirement is covered and where failures have been detected.

press down arrow for feature details

Dashboard Charts

press down arrow for feature details

Dashboard Charts - KPI

Shows selected (published) Key Performance Indicators:

  • KPI value
  • KPI Status
  • KPI Ranges (Hover Popup)
  • Multile KPIs
  • KPI Configurable

press down arrow for feature details

Dashboard Charts - Requirements Summary

  • Requirement Status on All Projects
  • Click on Category Code to Add/Remove Category from Chart
  • Bar Chart - Select Grouped or Stacked Chart

press down arrow for feature details

Dashboard Charts - Test Case Summary

  • Pie Chart - Test Case Execution Status on All Projects
  • Bar Chart - Test Case Execution Status by Projects

press down arrow for feature details

Dashboard Charts - Defects Summary

  • Pie Chart - Defects Breakdown by Severity on All Projects
  • Bar Chart - Defects Breakdown by Projects

press down arrow for feature details

Dashboard Charts - Historical Trends

  • Shows Trends based on past executions
  • Time Axis is by Day
  • Each Series (line) Represents a status or one Project
  • Time Period is Configurable
  • Three Chart Modes:
    • Stacked
    • Stream - individual line chart
    • Expanded - convert to %
  • press down arrow for feature details

Dashboard Charts - Requirement Trend

  • Left Chart - Trend of Requirements Execution By Requirement Status
  • Right Chart - Trend of Requirement Execution by Project

press down arrow for feature details

Dashboard Charts - Test Case Trend

  • Left Chart - Trend of Test Case Execution By Test Case Status
  • Right Chart - Trend of Test Case Execution by Project

press down arrow for feature details

Dashboard Charts - Defect Trend

  • Left Chart - Trend of Defect Raised By Defect Severity Code
  • Right Chart - Trend of Defect Raised by Project

Dashboard Reports

  • Reports are by Project
  • Currently 5 Reports
  • Click on Project to Set Report Context
  • Click on Report Buttons to Show Report
  • Click on "Refresh" to reload project list

press down arrow for feature details

Dashboard Reports - Requirement Traceability

Requirement Traceability by Model:

  • Shows requirement coverage by model
  • MScript location where assert is placed
  • Number of asserts on the requirement
  • Number of asserts failed

press down arrow for feature details

Dashboard Reports - Requirement Traceability

Requirement Traceability for selected project:

  • Shows requirement coverage by project
  • Number of asserts in all models for selected project

press down arrow for feature details

Dashboard Reports - Model Failed Requirements

Failed Requirements by Model:

  • Shows failed requirements by model
  • MScript location where assert is placed
  • Date/time and test case that exercised assert

press down arrow for feature details

Dashboard Reports - Failed Requirements

Failed Requirements for selected project:

  • Shows failed requirements for the selected project
  • MScript location where assert is placed
  • Date/time and test case that exercised assert

press down arrow for feature details

Dashboard Reports - Test Case Stats

Execution stats on test cases:

  • Shows generatl execution stats on test cases for selected project
  • Avg/Min/Max Step columns show test case lengths

Dashboard - Projects

  • Projects are collections of models
  • Model may be placed in more than one projects
  • Projects must be created and published to be included in Dashboard Chart

press down arrow for feature details

Dashboard - Projects

Create New Project:

  • Add project - click on
  • New project displayed in the right pane
  • Enter project name - tab out of Project field to save changes
  • Click on to add model to or remove model from project
  • Use filter above model name to search or filter model list

press down arrow for feature details

Dashboard - Project

Publish Project:

  • Projects must be explicitly published to be included in Dashboard Charts & Reports
  • Green check next to project name indicates
    the project is published
  • Click on to toggle project publish indicator

press down arrow for feature details

Dashboard - Project

Edit / Delete Projects:

  • Click on the project name to select project for edit
  • Delete projects:
    • Select projects to be deleted by checking the checkbox before project name
    • Click toolbar button
  • Use filter field to search or filter project list

Dashboard - Create & Publish KPI

  • Key Performance Indicators (KPI) are displayed on the upper-right corner of the application.
  • KPI must be first created and published.
  • Choose from over 20 performance measurements as KPIs.
  • Dashboard Edition - published KPIs can also be subscribed by ProMBT and EnterpriseMBT editions

press down arrow for feature details

Dashboard - Create & Publish KPI

press down arrow for feature details

Dashboard - Create & Publish KPI

Create KPI:

  • Create KPI - click on
  • New KPI is displayed in the right pane
  • Enter KPI name - tab out of field to save changes
  • Sequence Number - display order KPIs on Dashboard
  • Select a measurement from KPI Element dropdown list
  • Check the checkbox to publish KPI to Dashboard
  • You may also click on in KPI List to publish/un-publish KPI

press down arrow for feature details

Dashboard - Create & Publish KPI

Create KPI - add ranges:

  • Ranges are used to determine color state of KPI
  • Create a range - click on in "KPI Range" toolbar
  • Enter a color name or HTML color code
  • Enter min and max fields - leave blank for unbound range
  • Ranges must not overlap
  • Ranges must be consecutive leaving no gap
  • Delete KPI Ranges:
    • Select ranges by checking the checkbox
    • Click on in "KPI Range" toolbar

press down arrow for feature details

Dashboard - Create & Publish KPI

Edit/Delete KPI:

  • Click on KPI name in KPI List to select KPI for edit
  • Click on to toggle KPI publish flag
  • Delete KPI:
    • Select KPIs to delete by checking the checkbox
    • Click on in "KPI" toolbar
  • Use filter field to search or filter KPI list
  • Click on to refresh KPI List

Dashboard - Aggregate Setup (ETL)

  • Each TestOptimal Server/Edition stores MBT execution stats in its own database
  • Dashboard Edition provides an automatic ETL facility to aggregate MBT executions from all ProMBT, EnterpriseMBT and RuntimeMBT servers
  • MBT executions from various TestOptimal servers are merged and aggregated within Dashboard
  • Project groups models for charting and reporting

press down arrow for feature details

Dashboard - Aggregate Setup (ETL)

  • Dashboard trend graphs are generated using model executions within the date intervals set by Dashboard Trend Duration
  • System performs daily cleanup to purge old execution stats
  • Cleanup criterial is set in Keep Execution Stats
  • Keep Execution Stats interval must be larger or equal to Dashboard Trend Duration
  • Enter date interval using suffix "D" for days, "W" for weeks and "Y" for years. e.g. 2W for 2 weeks.

press down arrow for feature details

Dashboard - Aggregate Setup (ETL)

Create ETL Process:

  • ETL (Extract-Transform-Load) Process retrieves MBT executions from a specific TestOptimal server
  • ETL Process is scheduled to run in background at specified interval and time
  • Create ETL Process - click on in toolbar
  • Set Server and Port# fields pointing to TestOptimal server
  • Set Sched Time and Every fields to the desired value.
  • Check Active checkbox to activate ETL Process
  • By default, ETL Process is created with Active checkbox unchecked.

press down arrow for feature details

Dashboard - Aggregate Setup (ETL)

Monitor & Manage ETL Processes:

  • ETL Processes are automatically scheduled to run at the specified time and interval
  • Next Run Date shows when the next run will be for the ETL Process - valid only if Active is checked
  • Last Run Date/Time and Last Run Status show when the last execution of ETL Process and the status of that run

press down arrow for feature details

Dashboard - Aggregate Setup (ETL)

Monitor & Manage ETL Processes:

  • Click on in toolbar for more details of the ETL Process execution
  • Click on in toolbar to hide the extra columns
  • Uncheck Active checkbox to stop it from being executed by ETL scheduler

press down arrow for feature details

Dashboard - Aggregate Setup (ETL)

Delete ETL Process:

  • Select ETL Process to delete by checking the checkbox
  • Click on in toolbar

press down arrow for feature details

Dashboard - Aggregate Setup (ETL)

Trouble-shooting - Test Connection:

  • Select ETL Process to test the connection by checking the checkbox
  • Click on in toolbar to start connection test
  • An alert dialog will be displayed to show the connection test results

press down arrow for feature details

Dashboard - Aggregate Setup (ETL)

Trouble-shooting - Manual Execution:

  • Select ETL Process for manual execution by checking the checkbox
  • Click on in toolbar
  • Selected ETL Processes will be executed immediately
  • Click on in toolbar to check for execution status
  • ETL Process that failed at last run is highlighted in orange

THE END

Visit other tutorials Here

© TestOptimal LLC.