Run a Test Suite and Resolve Missing Model Coverage

This example uses:

This example shows how to run a test suite, collect coverage, and generate test cases to satisfy missing model coverage.

Open and Examine Project

In this example, you test the target speed selection algorithm, which calculates the vehicle target speed in the adaptive cruise control of an autonomous vehicle. Your tests must ensure that your development team implemented the algorithm according to the requirements.

Open the project that this example uses. You can open the project by navigating to the project folder, RequirementsBasedTestingProject and then double-clicking RequirementsBasedTestingProject.prj or entering:

openProject("RequirementsBasedTestingProject");

Refreshing Simulink customizations

In this example, you use the test suite in this project, which contains several test cases linked to requirements. You use Simulink® Test™ to:

Execute the test suite
Measure test coverage
Generate tests to analyze missing model test coverage
Resolve missing test coverage

Model test coverage, or model coverage, is a measure of how much of your design has been tested through simulation. Having full model coverage raises confidence that you have thoroughly tested the behavior of your model. Missing model coverage may be the result of dead logic, incomplete testing, unexpected functionality, or incomplete/missing requirements.

In this example, you use:

Requirements Toolbox™ to manage requirements in the Simulink environment
Simulink® Test™ to develop, manage, and execute simulation-based tests
Simulink® Coverage™ to measure test coverage for Simulink models and generated code
Simulink® Design Verifier™ to detect design errors, generate tests to achieve test coverage goals, and formally verify requirements using static analysis

Open Test Suite in Test Manager

Opening the test file in the Test Manager.

sltest.testmanager.clear % clears test files from Test Manager
sltest.testmanager.load('CruiseControl_TestSuite.mldatx');
sltest.testmanager.view

The test suite consists of many test cases, each of which tests a single requirement.

Enable Coverage Analysis

Next, enable model coverage analysis for the test suite. Collecting model coverage for multiple tests of the same model and adding the results into a single coverage summary is called coverage aggregation because model coverage is collected for each test case and then aggregated across all test cases in the test suite.

To enable model coverage in the Test Manager:

In the Test Browser pane, click the CruiseControl_TestSuite test suite.
Open the Coverage Settings section.
Select Record coverage for system under test.

In the Coverage Metrics section, Decision, Condition, and MCDC are selected. To learn more about these coverage metrics, see Types of Model Coverage.

Execute Test Suite

To execute the test suite:

In the Test Browser pane, select the test suite CruiseControl_TestSuite.
Click Run.

The Test Manager switches to the Results and Artifacts tab to show the test execution progress. This test suite has 15 test cases.

View Model Coverage Results

To view model coverage results, click the most recent test run in the Results and Artifacts tab, then expand Aggregated Coverage Results.

The tests report almost full coverage. To confirm that the coverage received is based on the correct requirements, select Scope coverage results to linked requirements.

The tests now scope the coverage results based on requirements:

The coverage results decrease when you scope them to requirements. This decrease is due to incidental coverage, which is when a test case exercises a part of the model that is not associated with the requirement that the test case is meant to test. In requirements-scoped coverage, an objective is only fully covered if it is fully exercised by the test case linked to the requirement for that objective.

Open the coverage report to investigate further. Click the report icon in Aggregated Coverage Results.

In the Summary section, click SF:ON.

The coverage report displays the summary for the coverage results that are scoped to linked requirements. Missing coverage is colored red, and most of this missing coverage is due to incidental coverage, which you removed by scoping the results to linked requirements.

Most of the coverage objectives have an associated test case, even though they are not satisfied. In addition, there are no requirements links for the "ON" state. For an example of a fully covered coverage objective, look at the Transition from Junction #0 to Junction #1.

The Stateflow transition is fully exercised by test case T3, which is the test case associated with the parent requirement for the transition. In this case, the Stateflow transition has no incidental coverage.

Resolve Incidental Coverage

To resolve the incidental coverage for the "ON" state, look at the requirements to determine whether the project has a missing requirement or if the "ON" state implements an existing requirement.

Open the requirements set in the Requirements Editor:

slreq.clear
reqSet = 'CruiseControl_TestSuite.slreqx';
slreq.open(reqSet);
rmi.navigate('linktype_rmi_slreq',reqSet,'17')
clear reqSet

Requirement 2.4 is the correct requirement to associate with this part of the design.

Create a link from requirement 2.4 to the state by selecting the requirement and clicking Add Link in the Requirements Editor, then Select for Linking with Requirement.

Investigate Incomplete Coverage By Generating Test Vectors

Before you scoped the coverage to the requirements, the project coverage was incomplete. You can confirm this by opening the Test Manager and clearing Scope coverage results to linked requirements.

This missing coverage indicates that there are parts of the model which were not exercised during simulation.

Click the model name in the Aggregated Coverage Results section to open the model with coverage results highlighting.

In the Coverage Details pane you can see coverage results for model elements by clicking them and viewing the details. You can also open the Coverage Details pane using the coverage details badge in the lower-left corner of the Simulink canvas.

Causes for Missing Model Coverage

There are many possible root causes for incomplete model coverage, such as dead logic, incomplete requirements-based tests, or missing or incomplete requirements. In some cases, incomplete model coverage may be difficult to analyze due to design complexity. In these cases, you can use Simulink Design Verifier to create automatically generated tests that can help you understand the required simulation inputs to reach a given point in your design. You can also use these generated tests to supplement existing requirements-based tests. These generated tests are not requirements-based because they do not include expected results, which you must define.

The two most important questions for this example are:

Is the design fully testable? In other words, is there dead logic? If a test cannot be generated to reach certain parts of the model, there might be dead logic.
If the design is fully testable, what does a test case look like? You can think of a test case as a language that can help you understand why you are missing test coverage in your requirements-based tests.

Generate Test Vectors to Analyze Missing Model Coverage

In the Results and Artifacts pane in the Test Manager, click the most recent test run. Under the Aggregated Coverage Results section, select the most recent set of coverage results and click Add Tests for Missing Coverage.

In the Add Tests for Missing Coverage window, choose these options:

Harness: <Create a new harness>
Source: Signal Editor
Test Case: <Create a new test case>
Test Type: Simulation Test
Test File: CruiseControl_TestSuite
Click OK.

Simulink® Design Verifier™ launches with these settings.

The Results window indicates that Simulink Design Verifier accounted for the coverage achieved from the existing test cases and generated test vectors to achieve coverage for four additional coverage objectives.

The four missing coverage objectives are satisfied, which indicates that they are not dead logic. It is recommended to perform dead logic analysis before using Simulink Design Verifier to generate test cases. For more information, see How to Use Simulink Design Verifier to Automatically Detect Design Errors in Your Simulink Models.

When Simulink Design Verifier generates the tests, the new test case appears in the Test Manager.

The three test vectors are iterations within that test case.

Generate Coverage Report for Full Test Suite with Automatically Generated Test Vectors

To understand how these test vectors help achieve the missing coverage objectives for the test suite, look at the coverage report.

Move this new test case into the existing test suite by right-clicking on the test case and selecting Cut, then right-click CruiseControl_TestSuite and click Paste. Then right-click New Test Case 1 and select Rename to rename the test case Generated test vectors.

In the Inputs section of the Generated test vectors test case, select Include input data in test result.

Run CruiseControl_TestSuite with the new test case.

Look at the updated test results.

The summary of the test suite shows 18 tests passed. The generated test vectors satisfy the missing coverage, but Simulink Design Verifier does not analyze what the expected output of the model is for these input vectors. If you want to use these test cases in your testing, you must determine what the model output should be when given the generated input, and update or create requirements or tests as needed.

Generate a new coverage report by clicking the arrow icon in the Report column in the Aggregated Coverage Results section.

In the coverage report, in the Summary section, click SF: ON to open the Details section of that state.

Recall that you are looking for a test case that would exit the ON state from the Accel substate. It looks like Test Case T18 did that. This was one of the tests generated by Simulink Design Verifier.

To see the test case that analyzes the case where the model exits the ON state from the Accel substate, return to the coverage report section State "ON" and view the State "Accel" row. Test case T18, which is one of the tests cases that Simulink Design Verifier generated, satisfies this objective.

Click T18 to scroll to the list of tests.

T18 is TestCase_3. Click TestCase_3 to view the test case in the Test Manager.

Visualize the test case by clicking Visualize > Show Simulation Output Plots.

You can view the inputs by adding them using the Signal Editor Inputs drop-down menu, and you add subplots using the Subplots button.

You can also zoom in on specific times and pan through the test case using your cursor.

Based on your analysis of the input vectors that Simulink Design Verifier generated, you must take action to resolve the issue in your model. You can:

Write a new requirement
Modify an existing requirement
Append to or modify an existing test
Justify the missing coverage

The simplest option is to append to or modify an existing test. For this example, append additional steps to the Disengage upon braking event test case.

Click Test Browser to view test cases.
Click Disengage upon braking event.
In the System under test section, under Test Harness, click the arrow next to the test harness name to open the test harness.

Using the test harness, you can modify the test case to satisfy the coverage objectives that the generated test case satisfies.