Justify Missing Coverage for MATLAB Source Code

Since R2024b

You can improve your coverage metrics by using justifications to filter blocks of MATLAB^® source code covered by methods other than automated testing, for instance, by interactive testing. Justifying missing coverage for a block of code is useful if it is not feasible to satisfy all coverage outcomes for that block in automated tests or if you do not intend for your automated tests to exercise the unsatisfied outcomes. Justifications allow you to focus on parts of source code that can and should be tested through automated tests.

This example shows how to justify missing coverage for a block of code in a MATLAB project. First, you run the tests in the project and generate an interactive code coverage report. Then, you justify missing coverage for a block of code by selecting its entry point in the report and creating a justification. When you save the justification, the testing framework treats the entire block of code as covered and updates the coverage metrics as if all coverage outcomes for that block were satisfied.

Generate Code Coverage Report

To justify the missing coverage for your code, first generate an interactive HTML code coverage report that includes information at the decision-coverage level or above. For more information about coverage types, see Types of Code Coverage for MATLAB Source Code.

Open the ShortestPath project, which includes a function named shortest_path in its src folder and tests for the function in its tests folder. The shortest_path function returns the length of the shortest path between two nodes in a graph.

openExample("matlabtest/ShortestPathExample")

Using a test runner that provides programmatic access to information on all possible coverage types, run the tests in the project and collect code coverage information for the source code in the src folder. The coverage result indicates that the tests in the project achieve full function coverage. However, the tests do not achieve full coverage for the remaining coverage types.

import matlab.unittest.plugins.CodeCoveragePlugin
import matlab.unittest.plugins.codecoverage.CoverageResult
 
runner = testrunner("minimal");
format = CoverageResult;
plugin = CodeCoveragePlugin.forFolder("src", ...
    Producing=format,MetricLevel="mcdc");
addPlugin(runner,plugin)
 
suite = testsuite;
testResults = run(runner,suite);
 
coverageResult = format.Result

coverageResult = 

  Result with properties:

        Filename: "C:\work\Examples\matlabtest\ShortestPathExample\MATLABShortestPath\src\shortest_path.m"
    CreationDate: 02-Jul-2024 10:06:55
          Filter: [0×0 matlabtest.coverage.Justification]

Coverage summary (HTML report):
   Function: 3/3 (100%)
   Statement: 44/50 (88%)
   Decision: 29/36 (80.55%)
   Condition: 23/30 (76.66%)
   MC/DC: 8/15 (53.33%)

Use coverageSummary to retrieve information from the coverage results.

Generate an interactive code coverage report from the coverage result. The report displays the collected metrics and uses different colors to highlight the covered, missed, or partially covered executables.

generateHTMLReport(coverageResult)

Create Justification to Filter Code Block

You can filter blocks of code that did not receive full coverage by creating justifications in an interactive code coverage report. The framework automatically creates a matlabtest.coverage.Justification object for each justification that you create in the report. The framework also saves the Justification objects to a MAT file for future use. For example, suppose that you create an interactive code coverage report in a project folder, justify the missing coverage, and then close the report. If you reopen the report, the report includes the justifications that you previously created.

Creating justifications in a code coverage report is subject to these requirements:

• You can create a justification only if Function, Statement, or Decision is selected from the Currently viewing list on the code coverage report. Creating a justification in the condition coverage or MC/DC view is not supported.

• To create a justification for a block of code, you must select the highlighted entry point to that block. Examples of code block entry points include:

  • An if or switch statement

  • A for or while statement

  • A try statement

  • A function definition

This example creates a justification in the generated code coverage report for illustrative purposes. To create a justification, select Decision from the Currently viewing list on the report. Then, in the Source Details section, click the highlighted code if(isscalar(node) && isnumeric(node) && ~isinf(node) && floor(node) == node), which is the entry point to an if-else code block. The framework displays the Create Coverage Justification dialog box that prompts you to specify the reason for coverage filtering. Enter a reason for the justification, and then click OK. The framework creates a Justification object and saves it as the Filters variable to a MAT file under the project root folder. If you create additional justifications, the framework saves them to the same MAT file.

Note

The framework creates the MAT file at a predetermined location only in MATLAB projects. If you are not using a project, you must specify a file path using your system file browser.

Once you justify missing coverage for a block of code, the framework updates the code coverage report as if all the coverage outcomes for that block were satisfied. The Overall Coverage Summary and Breakdown by Source sections of the report include the updated coverage metrics and represent the percentages of justified outcomes using gray bars. In the Source Details section, code highlighting for the affected executables depends on the executable type. The framework highlights the missed executables as justified (in gray) and the partially covered executables as covered (in green).

Interact with Justification in Report

You can programmatically access the justifications saved to a MAT file by loading the corresponding Justification array into the workspace. Alternatively, you can interact with justifications directly in the Source Details section of the coverage report:

• To view the reason for a justification, point to the Justification Summary button for a justified outcome or directly point to the highlighted code.

• To view the summary of a justification or to delete the justification, click Justification Summary for a justified outcome or directly click the highlighted code.

For example, with MC/DC selected from the Currently viewing list on the code coverage report, click Justification Summary in the Statement column for the justified outcome out = false; of the filtered code block. The framework displays a summary of the justification in the Justification Applied dialog box. The summary includes the name of the file containing the code block, the entry point to the code block, and the justification reason. The dialog box contains a button for deleting the justification and another button for closing the dialog box. For this example, close the dialog box by clicking the Cancel button.

Apply Justification to Coverage Result

After justifying the missing coverage for your code, you can create new coverage results by using the applyFilter and resetFilter methods. For example, create a coverage result that incorporates the justified outcomes by applying the Justification object to the original coverage result. In this example, which is based on a MATLAB project, the MAT file containing the justification has a path predetermined by the framework. If you are not using a project to organize your work, load the MAT file by specifying the path you used when creating your justifications.

S = load(fullfile(pwd,"derived","codecoverage","Filter.mat"));
newResult = coverageResult.applyFilter(S.Filters)

newResult = 

  Result with properties:

        Filename: "C:\work\Examples\matlabtest\ShortestPathExample\MATLABShortestPath\src\shortest_path.m"
    CreationDate: 02-Jul-2024 10:06:55
          Filter: [1×1 matlabtest.coverage.Justification]

Coverage summary (HTML report):
   Function: 3/3 (100%)
   Statement: (44+1)/50 (90%)
   Decision: (29+1)/36 (83.33%)
   Condition: (23+4)/30 (90%)
   MC/DC: (8+4)/15 (80%)

Use coverageSummary to retrieve information from the coverage results.

Generate a standalone code coverage report from newResult. When you generate a standalone code coverage report or code coverage results in Cobertura XML format, the framework represents the justified outcomes as covered.

generateStandaloneReport(newResult)

Generating standalone report. Please wait.
    Preparing content for the standalone report.
    Adding content to the standalone report.
    Writing standalone report to file.

Reuse Justification in Fresh Analysis

If you rerun your tests to collect fresh code coverage information, you can reuse the justifications previously created for the source code under test. For instance, run the tests in the project to create a new interactive code coverage report in the report subfolder of the project root folder. The framework automatically applies the existing justification in the project to the generated report.

import matlab.unittest.plugins.codecoverage.CoverageReport
 
r = testrunner("minimal");
f = CoverageReport("report");
p = CodeCoveragePlugin.forFolder("src",Producing=f,MetricLevel="mcdc");
addPlugin(r,p)
results = run(r,suite);

MATLAB code coverage report has been saved to:
 C:\work\Examples\matlabtest\ShortestPathExample\MATLABShortestPath\report\index.html

Note

If you are not using a MATLAB project, reusing justifications requires you to specify the Filter name-value argument when creating a plugin using one of the static methods of the matlab.unittest.plugins.CodeCoveragePlugin class.