runperf

Run set of tests for performance measurement

collapse all in page

Syntax

  • results = runperf
  • results = runperf(tests)
    example
  • results = runperf(tests,Name,Value)
    example

Description

results = runperf runs all the tests in your current folder for performance measurements and returns an array of MeasurementResult objects. Each element in results corresponds to an element in the test suite.

The performance test framework runs the tests using a variable number of measurements to reach a sample mean with a 0.05 relative margin of error within a 0.95 confidence level. It runs the tests four times to warm up the code, and then between 4 and 32 times to collect measurements that meet the statistical objectives. If the sample mean does not meet the 0.05 relative margin of error within a 0.95 confidence level after 32 test runs, the performance test framework stops running the test and displays a warning. In this case, the MeasurementResult object contains information for the 4 warm-up runs and 32 measurement runs.

The runperf function provides a simple way to run a collection of tests as a performance experiment.

example

results = runperf(tests) runs a specified set of tests.

example

results = runperf(tests,Name,Value) runs a set of tests with additional options specified by one or more Name,Value pair arguments.

Examples

collapse all

In your current working folder, create a script-based test, onesTest.m, that uses three different methods to initialize a 1000x1500 matrix of ones.

rows = 1000;
cols = 1500;

%% Ones Function
X = ones(rows,cols);

%% Loop Assignment Without Preallocation
for r = 1:rows
    for c = 1:cols
        X(r,c) = 1;
    end
end

%% Loop Assignment With Preallocation
X = zeros(rows,cols);
for r = 1:rows
    for c = 1:cols
        X(r,c) = 1;
    end
end

Run the script as a performance test. Your results might vary.

results = runperf('onesTest');
Running onesTest
..........
..........
..........
......Warning: The target Relative Margin of Error was not met after running the MaxSamples for onesTest/OnesFunction.
....
..........
..........
..........
....
Done onesTest
__________

In the example output, the performance testing framework ran OnesFunction test the maximum number of times, but did not achieve a 0.05 relative margin of error with a 0.95 confidence level.

Display the results. The results variable is a 1x3 MeasurementResult array. Each element in the array corresponds to one of the tests defined in the code section in onesTest.m.

results
results = 

  1x3 MeasurementResult array with properties:

    Name
    Valid
    Samples
    TestActivity

Totals:
   3 Valid, 0 Invalid.

Display the measurement results for the second test, which loops the assignment without preallocation.

results(2)
ans = 

  MeasurementResult with properties:

            Name: 'onesTest/LoopAssignmentWithoutPreallocation'
           Valid: 1
         Samples: [4x7 table]
    TestActivity: [8x12 table]

Totals:
   1 Valid, 0 Invalid.

Display the complete table of test measurements.

results(2).TestActivity
ans = 

                       Name                        Passed    Failed    Incomplete    MeasuredTime    Objective         Timestamp             Host        Platform           Version                      TestResult                          RunIdentifier            
    ___________________________________________    ______    ______    __________    ____________    _________    ____________________    ___________    ________    _____________________    ________________________________    ____________________________________

    onesTest/LoopAssignmentWithoutPreallocation    true      false     false         0.42843         warmup       29-Dec-2015 15:26:46    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    [1x1 matlab.unittest.TestResult]    03232d6b-d0b7-4ab5-b4b1-5aa5061336b4
    onesTest/LoopAssignmentWithoutPreallocation    true      false     false         0.38467         warmup       29-Dec-2015 15:26:46    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    [1x1 matlab.unittest.TestResult]    03232d6b-d0b7-4ab5-b4b1-5aa5061336b4
    onesTest/LoopAssignmentWithoutPreallocation    true      false     false         0.34094         warmup       29-Dec-2015 15:26:46    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    [1x1 matlab.unittest.TestResult]    03232d6b-d0b7-4ab5-b4b1-5aa5061336b4
    onesTest/LoopAssignmentWithoutPreallocation    true      false     false         0.34076         warmup       29-Dec-2015 15:26:47    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    [1x1 matlab.unittest.TestResult]    03232d6b-d0b7-4ab5-b4b1-5aa5061336b4
    onesTest/LoopAssignmentWithoutPreallocation    true      false     false         0.36286         sample       29-Dec-2015 15:26:47    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    [1x1 matlab.unittest.TestResult]    03232d6b-d0b7-4ab5-b4b1-5aa5061336b4
    onesTest/LoopAssignmentWithoutPreallocation    true      false     false         0.39701         sample       29-Dec-2015 15:26:48    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    [1x1 matlab.unittest.TestResult]    03232d6b-d0b7-4ab5-b4b1-5aa5061336b4
    onesTest/LoopAssignmentWithoutPreallocation    true      false     false         0.38368         sample       29-Dec-2015 15:26:48    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    [1x1 matlab.unittest.TestResult]    03232d6b-d0b7-4ab5-b4b1-5aa5061336b4
    onesTest/LoopAssignmentWithoutPreallocation    true      false     false         0.39277         sample       29-Dec-2015 15:26:48    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    [1x1 matlab.unittest.TestResult]    03232d6b-d0b7-4ab5-b4b1-5aa5061336b4

The performance testing framework ran four warm-up runs, followed by four measurements runs (indicated as sample in the Objective column).

Display the mean measured time for the second test. To exclude data collected in the warm-up runs, use the values in the Samples field.

mean(results(2).Samples.MeasuredTime)
ans =

    0.3841

To compare the different initialization methods in the script, display the mean measured time for all the tests. Concatenate the values from the Samples field across the three elements in the results array. Then use varfun to group the table entries by name and compute the mean.

fullTable = vertcat(results.Samples);
varfun(@mean,fullTable,'InputVariables','MeasuredTime','GroupingVariables','Name')
ans = 

                       Name                        GroupCount    mean_MeasuredTime
    ___________________________________________    __________    _________________

    onesTest/OnesFunction                          32            0.0031614        
    onesTest/LoopAssignmentWithoutPreallocation     4              0.38408        
    onesTest/LoopAssignmentWithPreallocation       26             0.016025

In the example output, the ones function was the fastest way to initialize the matrix to ones. The performance testing framework made 32 measurement runs for this test. Your results might vary.

Open Script

In your current working folder, create a class-based test, preallocationTest.m, that compares different methods of preallocation.

classdef preallocationTest < matlab.perftest.TestCase
    methods(Test)
        function testOnes(testCase)
            x = ones(1,1e7);
        end
        
        function testIndexingWithVariable(testCase)
            id = 1:1e7;
            x(id) = 1;
        end
        
        function testIndexingOnLHS(testCase)
            x(1:1e7) = 1;
        end
        
        function testForLoop(testCase)
            for i=1:1e7
                x(i) = 1;
            end
        end
        
    end
end

The measurement boundary for the preallocationTest class is the test method. The time measurement for each test method includes all the code in method. For information on designating measurement boundaries, see the startMeasuring and stopMeasuring methods of matlab.perftest.TestCase.

Run performance tests for all the elements that contain 'Indexing' in the name. Your results might vary, and you might see a warning if runperf doesn't meet statistical objectives.

results = runperf('preallocationTest','Name','*Indexing*')

Running preallocationTest
..........
..........
..........
..........
..
Done preallocationTest
__________


results = 

  1x2 MeasurementResult array with properties:

    Name
    Valid
    Samples
    TestActivity

Totals:
   2 Valid, 0 Invalid.

Display the mean measured time for each of the tests. Concatenate the values from the Samples field across the two elements in the results array. Then use varfun to group the table entries by name and compute the mean.

fullTable = vertcat(results.Samples);
varfun(@mean,fullTable,'InputVariables','MeasuredTime','GroupingVariables','Name')

ans = 

                       Name                       GroupCount    mean_MeasuredTime
    __________________________________________    __________    _________________

    preallocationTest/testIndexingWithVariable     4             0.16637         
    preallocationTest/testIndexingOnLHS           30            0.076792

Input Arguments

collapse all

Suite of tests specified as a character vector or cell array of character vectors. Each character vector in the cell array can contain the name of a test file, a test class, a test suite element name, a package containing your test classes, or a folder containing your test files.

Example: runperf('ATestFile.m')

Example: runperf('ATestFile/aTest')

Example: runperf('mypackage.MyTestClass')

Example: runperf(pwd)

Example: runperf({'mypackage.MyTestClass','ATestFile.m',pwd,'mypackage.subpackage'})

Name-Value Pair Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside single quotes (' '). You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

Example: runperf(tests,'Name','productA_*') runs test elements with a name that starts with 'productA_'.

collapse all

Name of the suite element, specified as a character vector. This argument filters TestSuite array elements. For the testing framework to run a test, the Name property of the test element must match the specified name. Use the wildcard character, *, to match any number of characters. Use the question mark character, ?, to match a single character.

Indicator to run tests in subfolders, specified as false or true (0 or 1). By default the framework runs tests in the specified folders, but not in their subfolders.

Data Types: logical

Indicator to run tests in subpackages, specified as false or true (0 or 1). By default the framework runs tests in the specified packages, but not in their subpackages.

Data Types: logical

Name of a parameterization property used by the test suite element, specified as a character vector. This argument filters TestSuite array elements. Use the wildcard character * to match any number of characters. Use the question mark character ? to match to a single character.

Name of a parameter used by the test suite element, specified as a character vector. This argument filters TestSuite array elements. Use the wildcard character * to match any number of characters. Use the question mark character ? to match a single character.

Name of the base folder that contains the file defining the test class, function, or script, specified as a character vector. This argument filters TestSuite array elements. For a test element to be included in the suite, the test element must be contained in the specified base folder. Use the wildcard character * to match any number of characters. Use the question mark character ? to match a single character. For test files defined in packages, the base folder is the parent of the top-level package folder.

Name of test element tag, specified as a character vector. This argument filters TestSuite array elements. Use the wildcard character * to match any number of characters. Use the question mark character ? to match a single character.

Alternatives

To create a test suite explicitly, you can use the testsuite function or the matlab.unittest.TestSuite methods to create a suite. Then, you can run your performance test with the run method of your specified TimeExperiment.

More About

collapse all

Tips

  • To customize the statistical objectives of the performance test, use the TimeExperiment class to construct and run the performance test.

See Also

| |

Introduced in R2016a

Was this topic helpful?
MATLAB Documentation
Other Documentation
Support