Create Advanced Parameterized Test

This example shows how to create a test that is parameterized in the TestClassSetup, TestMethodSetup, and Test methods blocks. The example test class tests the random number generator.

Test Overview
Create TestRand Test Class
Define properties Blocks
Define Test Class and Test Method Setup Methods
Define Sequential Parameterized Test Methods
Define Pairwise Parameterized Test Methods
Define Exhaustive Parameterized Test Methods
TestRand Class Definition Summary
Create Suite from All Tests
Run Suite from Class Using Selector
Run Suite from Method Using Selector
Run All Double Precision Tests

Test Overview

The TestRand test class is parameterized at three different levels.

Parameterization Level	Method Attribute	Property Attribute
Test level	`Test`	`TestParameter`
Method setup level	`TestMethodSetup`	`MethodSetupParameter`
Class setup level	`TestClassSetup`	`ClassSetupParameter`

At each test level, you can use the ParameterCombination method attribute to specify the test parameterization.

`ParameterCombination` Attribute	Method Invocation
`'exhaustive'` (default)	Methods are invoked for all combinations of parameters. The test framework uses this default combination if you do not specify the `ParameterCombination` attribute.
`'sequential'`	Methods are invoked with corresponding values from each parameter. Each parameter must contain the same number of values.
`'pairwise'`	Methods are invoked for every pair of parameter values at least once. While the test framework guarantees that tests are created for every pair of values at least once, you should not rely on that size, ordering, or specific set of test suite elements.

For example, use the combined methods attribute TestMethodSetup, ParameterCombination='sequential' to specify sequential combination of the method setup-level parameters defined in the MethodSetupParameter properties block.

For this example, class setup-level parameterization defines the type of random number generator. The method setup-level parameterization defines the seed for the random number generator, and the test-level parameterization defines the data type and size of the random number output.

Create TestRand Test Class

In a file in your working folder, create a class that inherits from matlab.unittest.TestCase. This class tests various aspects of random number generation.

classdef TestRand < matlab.unittest.TestCase

Define properties Blocks

Define the properties used for parameterized testing. Each properties block corresponds to parameterization at a particular level.

    properties (ClassSetupParameter)
        generator = {'twister','combRecursive','multFibonacci'};
    end
    
    properties (MethodSetupParameter)
        seed = {0, 123, 4294967295};
    end
    
    properties (TestParameter)
        dim1 = struct('small', 1,'medium', 2, 'large', 3);
        dim2 = struct('small', 2,'medium', 3, 'large', 4);
        dim3 = struct('small', 3,'medium', 4, 'large', 5);
        type = {'single','double'};
    end

Define Test Class and Test Method Setup Methods

Define the setup methods at the test class and test method level. These methods register the initial random number generator state. After the framework runs the tests, the methods restore the original state. The ClassSetup method defines the type of random number generator, and the TestMethodSetup seeds the generator.

    methods (TestClassSetup)
        function ClassSetup(testCase, generator)
            orig = rng;
            testCase.addTeardown(@rng, orig)
            rng(0, generator)
        end
    end
    
    methods (TestMethodSetup)
        function MethodSetup(testCase, seed)
            orig = rng;
            testCase.addTeardown(@rng, orig)
            rng(seed)
        end
    end

Define Sequential Parameterized Test Methods

Define a methods block with the Test and ParameterCombination='sequential' attributes. The test framework invokes these methods once for each corresponding property value.

    methods (Test, ParameterCombination='sequential')
        function testSize(testCase,dim1,dim2,dim3)
            testCase.verifySize(rand(dim1,dim2,dim3),[dim1 dim2 dim3])
        end 
    end

The method tests the size of the output for each corresponding parameter in dim1, dim2, and dim3. For example, to test all the 'medium' values use: testCase.verifySize(rand(2,3,4),[2 3 4]);. For a given TestClassSetup and TestMethodSetup parameterization, the framework calls the testSize method three times—once each for the 'small', 'medium', and 'large' values.

Define Pairwise Parameterized Test Methods

Define a methods block with the Test and ParameterCombination='pairwise' attributes. The test framework invokes these methods at least once for every pair of property values.

    methods (Test, ParameterCombination='pairwise')
        function testRepeatable(testCase,dim1,dim2,dim3)
            state = rng;
            firstRun = rand(dim1,dim2,dim3);
            rng(state)
            secondRun = rand(dim1,dim2,dim3);
            testCase.verifyEqual(firstRun,secondRun)
        end
    end

The test method verifies that the random number generator results are repeatable. For a given TestClassSetup and TestMethodSetup parameterization, the framework calls the testRepeatble method 10 times to ensure testing of each pair of dim1, dim2, and dim3. However, if the parameter combination attribute is exhaustive, the framework calls the method 3^3=27 times.

Define Exhaustive Parameterized Test Methods

Define a methods block with the Test attribute or no defined parameter combination. The parameter combination is exhaustive by default. The test framework invokes these methods once for every combination of property values.

    methods (Test)
        function testClass(testCase,dim1,dim2,type)
            testCase.verifyClass(rand(dim1,dim2,type), type)
        end
    end

The test method verifies that the class of the output from rand is the same as the expected class. For a given TestClassSetup and TestMethodSetup parameterization, the framework calls the testClass method 3*3*2=18 times to ensure testing of each combination of dim1, dim2, and type.

TestRand Class Definition Summary

classdef TestRand < matlab.unittest.TestCase
    properties (ClassSetupParameter)
        generator = {'twister','combRecursive','multFibonacci'};
    end
    
    properties (MethodSetupParameter)
        seed = {0, 123, 4294967295};
    end
    
    properties (TestParameter)
        dim1 = struct('small', 1,'medium', 2, 'large', 3);
        dim2 = struct('small', 2,'medium', 3, 'large', 4);
        dim3 = struct('small', 3,'medium', 4, 'large', 5);
        type = {'single','double'};
    end
    
    methods (TestClassSetup)
        function ClassSetup(testCase, generator)
            orig = rng;
            testCase.addTeardown(@rng, orig)
            rng(0, generator)
        end
    end
    
    methods (TestMethodSetup)
        function MethodSetup(testCase, seed)
            orig = rng;
            testCase.addTeardown(@rng, orig)
            rng(seed)
        end
    end
    
    methods (Test, ParameterCombination='sequential')
        function testSize(testCase,dim1,dim2,dim3)
            testCase.verifySize(rand(dim1,dim2,dim3),[dim1 dim2 dim3])
        end 
    end
    
    methods (Test, ParameterCombination='pairwise')
        function testRepeatable(testCase,dim1,dim2,dim3)
            state = rng;
            firstRun = rand(dim1,dim2,dim3);
            rng(state)
            secondRun = rand(dim1,dim2,dim3);
            testCase.verifyEqual(firstRun,secondRun);
        end
    end
    
    methods (Test)
        function testClass(testCase,dim1,dim2,type)
            testCase.verifyClass(rand(dim1,dim2,type), type)
        end
    end
end

Create Suite from All Tests

At the command prompt, create a suite from TestRand.m class.

suite = matlab.unittest.TestSuite.fromClass(?TestRand)

suite = 

  1x279 Test array with properties:

    Name
    BaseFolder
    Parameterization
    SharedTestFixtures
    Tags

Tests Include:
   17 Unique Parameterizations, 0 Shared Test Fixture Classes, 0 Tags.

The test suite contains 279 test elements. For a given TestClassSetup and TestMethodSetup parameterization, the framework creates 3+10+18=31 test elements. These 31 elements are called three times—once for each TestMethodSetup parameterization resulting in 3*31=93 test elements for each TestClassSetup parameterization. There are three TestClassSetup parameterizations resulting in a total of 3*93=279 test elements.

Examine the names of the first test element.

suite(1).Name

ans = 

TestRand[generator=twister]/[seed=value1]testClass(dim1=small,dim2=small,type=single)

The name of each element is constructed from the combination of the following:

Test class: TestRand
Class setup property and property name: [generator=twister]
Method setup property and property name: [seed=value1]
Test method name: testClass
Test method properties and property names: (dim1=small,dim2=small,type=single)

The name for the seed property isn't particularly meaningful (value1). The testing framework provided this name because the seed property values are numbers. For a more meaningful name, define the seed property as a struct with more descriptive field names.

Run Suite from Class Using Selector

At the command prompt, create a selector to select test elements that test the 'twister' generator for 'single' precision. Omit test elements that use properties with the 'large' name.

import matlab.unittest.selectors.HasParameter
s = HasParameter('Property','generator', 'Name','twister') & ...
    HasParameter('Property','type', 'Name','single') & ...
    ~HasParameter('Name','large');

suite2 = matlab.unittest.TestSuite.fromClass(?TestRand,s)

suite2 = 

  1x12 Test array with properties:

    Name
    BaseFolder
    Parameterization
    SharedTestFixtures
    Tags

Tests Include:
   9 Unique Parameterizations, 0 Shared Test Fixture Classes, 0 Tags.

If you first generate the full suite, construct the same test suite as above using the selectIf method.

suite = matlab.unittest.TestSuite.fromClass(?TestRand);
suite2 = selectIf(suite,s);

Run the test suite.

suite2.run;

Running TestRand
..........
..
Done TestRand
__________

Run Suite from Method Using Selector

At the command prompt, create a selector that omits test elements that use properties with the 'large' or 'medium' name. Limit results to test elements from the testRepeatable method.

import matlab.unittest.selectors.HasParameter
s =  ~(HasParameter('Name','large') | HasParameter('Name','medium'));

suite3 = matlab.unittest.TestSuite.fromMethod(?TestRand,'testRepeatable',s);
{suite3.Name}'

ans = 

    'TestRand[generator=twister]/[seed=value1]testRepeatable(dim1=small,dim2=small,dim3=small)'
    'TestRand[generator=twister]/[seed=value2]testRepeatable(dim1=small,dim2=small,dim3=small)'
    'TestRand[generator=twister]/[seed=value3]testRepeatable(dim1=small,dim2=small,dim3=small)'
    'TestRand[generator=combRecursive]/[seed=value1]testRepeatable(dim1=small,dim2=small,dim3=small)'
    'TestRand[generator=combRecursive]/[seed=value2]testRepeatable(dim1=small,dim2=small,dim3=small)'
    'TestRand[generator=combRecursive]/[seed=value3]testRepeatable(dim1=small,dim2=small,dim3=small)'
    'TestRand[generator=multFibonacci]/[seed=value1]testRepeatable(dim1=small,dim2=small,dim3=small)'
    'TestRand[generator=multFibonacci]/[seed=value2]testRepeatable(dim1=small,dim2=small,dim3=small)'
    'TestRand[generator=multFibonacci]/[seed=value3]testRepeatable(dim1=small,dim2=small,dim3=small)'

Run the test suite.

suite3.run;

Running TestRand
.........
Done TestRand
__________

Run All Double Precision Tests

At the command prompt, run all the test elements from TestRand.m that use the parameter name 'double'.

runtests('TestRand','ParameterName','double');

Running TestRand
..........
..........
..........
..........
..........
..........
..........
..........
.
Done TestRand
__________

Documentation