Test Performance Using Scripts or Functions

This example shows how to create a script or function-based performance test that times the preallocation of a vector using four different approaches.

Write Performance Test

Create a performance test in a file, preallocationTest.m, in your current working folder. In this example, you can choose to use either the following script-based test or the function-based test. The output in this example is for the function-based test. If you use the script-based test, then your test names will be different.

Script-Based Performance TestFunction-Based Performance Test
vectorSize = 1e7;

%% Ones Function
x = ones(1,vectorSize);

%% Indexing With Variable
id = 1:vectorSize;
x(id) = 1;

%% Indexing On LHS
x(1:vectorSize) = 1;

%% For Loop
for i=1:vectorSize
    x(i) = 1;
end
function tests = preallocationTest
tests = functiontests(localfunctions);
end

function testOnes(testCase)
vectorSize = getSize();
x = ones(1,vectorSize());
end

function testIndexingWithVariable(testCase)
vectorSize = getSize();
id = 1:vectorSize;
x(id) = 1;
end

function testIndexingOnLHS(testCase)
vectorSize = getSize();
x(1:vectorSize) = 1;
end

function testForLoop(testCase)
vectorSize = getSize();
for i=1:vectorSize
    x(i) = 1;
end
end

function vectorSize = getSize()
vectorSize = 1e7;
end

Run Performance Test

Run the performance test. Depending on your system, the warnings you see might vary. In this example output, the performance testing framework ran the preallocationTest/testOnes test the maximum number of times, but it did not achieve a 0.05 relative margin of error with a 0.95 confidence level.

results = runperf('preallocationTest.m')
Running preallocationTest
..........
..........
..........
......Warning: The target Relative Margin of Error was not met after running the MaxSamples for preallocationTest/testOnes.
....
..........
..........
..........
..........
.....
Done preallocationTest
__________


results = 

  1x4 MeasurementResult array with properties:

    Name
    Valid
    Samples
    TestActivity

Totals:
   4 Valid, 0 Invalid.

The results variable is a 1x4 MeasurementResult array. Each element in the array corresponds to one of the tests defined in the code section in preallocationTest.m.

Display Test Results

Display the measurement results for the second test. Your results might vary.

results(2)
ans = 

  MeasurementResult with properties:

            Name: 'preallocationTest/testIndexingWithVariable'
           Valid: 1
         Samples: [17x7 table]
    TestActivity: [21x12 table]

Totals:
   1 Valid, 0 Invalid.

As indicated by the size of the TestActivity property, the performance testing framework collected 21 measurements. This number of measurements includes four measurements to warm up the code. The Samples property excludes warm-up measurements.

Display the sample measurements for the second test.

results(2).Samples
ans = 

                       Name                       MeasuredTime         Timestamp             Host        Platform           Version                      RunIdentifier            
    __________________________________________    ____________    ____________________    ___________    ________    _____________________    ____________________________________

    preallocationTest/testIndexingWithVariable    0.12496         31-Dec-2015 06:29:38    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.16411         31-Dec-2015 06:29:39    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.13467         31-Dec-2015 06:29:39    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.14919         31-Dec-2015 06:29:39    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.13663         31-Dec-2015 06:29:39    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.12597         31-Dec-2015 06:29:39    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.13036         31-Dec-2015 06:29:39    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.17423         31-Dec-2015 06:29:40    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.13087         31-Dec-2015 06:29:40    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.13951         31-Dec-2015 06:29:40    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.12493         31-Dec-2015 06:29:40    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.12613         31-Dec-2015 06:29:40    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.15276         31-Dec-2015 06:29:40    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.16414         31-Dec-2015 06:29:41    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.13791         31-Dec-2015 06:29:41    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.12533         31-Dec-2015 06:29:41    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551
    preallocationTest/testIndexingWithVariable    0.12339         31-Dec-2015 06:29:41    MY-HOSTNAME    win64       9.0.0.320924 (R2016a)    e0f6e9a4-c41d-409b-a480-93fc30d88551

Compute Statistics for Single Test Element

Display the mean measured time for the second test. To exclude data collected in the warm-up runs, use the values in the Samples field.

sampleTimes = results(2).Samples.MeasuredTime;
meanTest2 = mean(sampleTimes)
meanTest2 =

    0.1391

The performance testing framework collected 17 sample measurements for the second test. The test took an average of 0.1391 second.

Compute Statistics for All Test Elements

Determine the average time for all the test elements. The preallocationTest test includes four different methods for allocating a vector of ones. Compare the time for each method (test element).

Since the performance testing framework returns a Samples table for each test element, concatenate all these tables into one table. Then group the rows by test element Name, and compute the mean MeasuredTime for each group.

fullTable = vertcat(results.Samples);
summaryStats = varfun(@mean,fullTable,...
    'InputVariables','MeasuredTime','GroupingVariables','Name')
summaryStats = 

                       Name                       GroupCount    mean_MeasuredTime
    __________________________________________    __________    _________________

    preallocationTest/testOnes                    32            0.031445         
    preallocationTest/testIndexingWithVariable    17             0.13912         
    preallocationTest/testIndexingOnLHS           23            0.071286         
    preallocationTest/testForLoop                  4             0.80677

Recall that the performance testing framework issued a warning stating that the measurements for the preallocationTest/testOnes test did not meet the statistical objectives. The testing framework collected the maximum number of samples, which is 32, and then it stopped the test. By contrast, the measurements for the preallocationTest/testForLoop test met statistical objectives in the minimum number of samples, which is four.

Change Statistical Objectives and Rerun Tests

Change the statistical objectives defined by the runperf function by constructing and running a time experiment. Construct a time experiment with measurements that reach a sample mean with an 8% relative margin of error within a 97% confidence level.

Construct an explicit test suite.

suite = testsuite('preallocationTest');

Construct a time experiment with a variable number of sample measurements, and run the tests.

import matlab.perftest.TimeExperiment
experiment = TimeExperiment.limitingSamplingError('NumWarmups',2,...
    'RelativeMarginOfError',0.08, 'ConfidenceLevel', 0.97);
resultsTE = run(experiment,suite);

Running preallocationTest
..........
..........
.......
Done preallocationTest
__________

Compute the statistics for all the test elements.

fullTableTE = vertcat(resultsTE.Samples);
summaryStatsTE = varfun(@mean,fullTableTE,...
    'InputVariables','MeasuredTime','GroupingVariables','Name')
summaryStatsTE = 

                       Name                       GroupCount    mean_MeasuredTime
    __________________________________________    __________    _________________

    preallocationTest/testOnes                    4             0.025568         
    preallocationTest/testIndexingWithVariable    6              0.12898         
    preallocationTest/testIndexingOnLHS           5             0.066603         
    preallocationTest/testForLoop                 4              0.78484

See Also

| | |

Was this topic helpful?
MATLAB Documentation
Support