This example shows how to create a performance
test and regression test for the fprintf function.
Consider the following unit (regression) test. You can run this
test as a performance test using runperf('fprintfTest') instead
of runtests('fprintfTest').
classdef fprintfTest < matlab.unittest.TestCase properties file fid end methods(TestMethodSetup) function openFile(testCase) testCase.file = tempname; testCase.fid = fopen(testCase.file,'w'); testCase.assertNotEqual(testCase.fid,-1,'IO Problem') testCase.addTeardown(@delete,testCase.file); testCase.addTeardown(@fclose,testCase.fid); end end methods(Test) function testPrintingToFile(testCase) textToWrite = repmat('abcdef',1,5000000); fprintf(testCase.fid,'%s',textToWrite); testCase.verifyEqual(fileread(testCase.file),textToWrite) end function testBytesToFile(testCase) textToWrite = repmat('tests_',1,5000000); nbytes = fprintf(testCase.fid,'%s',textToWrite); testCase.verifyEqual(nbytes,length(textToWrite)) end end end
The measured time does not include the time to open and close
the file or the assertion because these activities take place inside
a TestMethodSetup block, and not inside a Test block.
However, the measured time includes the time to perform the verifications.
Best practice is to measure a more accurate performance boundary.
Create a performance test in a file, fprintfTest.m,
in your current working folder. This test is similar to the regression
test with the following modifications:
The test inherits from matlab.perftest.TestCase instead
of matlab.unittest.TestCase.
The test calls the startMeasuring and stopMeasuring methods
to create a boundary around the fprintf function
call.
classdef fprintfTest < matlab.perftest.TestCase properties file fid end methods(TestMethodSetup) function openFile(testCase) testCase.file = tempname; testCase.fid = fopen(testCase.file,'w'); testCase.assertNotEqual(testCase.fid,-1,'IO Problem') testCase.addTeardown(@delete,testCase.file); testCase.addTeardown(@fclose,testCase.fid); end end methods(Test) function testPrintingToFile(testCase) textToWrite = repmat('abcdef',1,5000000); testCase.startMeasuring(); fprintf(testCase.fid,'%s',textToWrite); testCase.stopMeasuring(); testCase.verifyEqual(fileread(testCase.file),textToWrite) end function testBytesToFile(testCase) textToWrite = repmat('tests_',1,5000000); testCase.startMeasuring(); nbytes = fprintf(testCase.fid,'%s',textToWrite); testCase.stopMeasuring(); testCase.verifyEqual(nbytes,length(textToWrite)) end end end
The measured time for this performance test includes only the
call to fprintf, and the testing framework still
evaluates the qualifications.
Run the performance test. Depending on your system, you might see warnings that the performance testing framework ran the test the maximum number of times, but did not achieve a 0.05 relative margin of error with a 0.95 confidence level.
results = runperf('fprintfTest');Running fprintfTest
..........
..........
..........
..........
.....
Done fprintfTest
__________
results =
1x2 MeasurementResult array with properties:
Name
Valid
Samples
TestActivity
Totals:
2 Valid, 0 Invalid.The results variable is a 1x2 MeasurementResult array.
Each element in the array corresponds to one of the tests defined
in the test file.
Display the measurement results for the first test. Your results might vary.
results(1)
ans =
MeasurementResult with properties:
Name: 'fprintfTest/testPrintingToFile'
Valid: 1
Samples: [10x7 table]
TestActivity: [14x12 table]
Totals:
1 Valid, 0 Invalid.As indicated by the size of the TestActivity property,
the performance testing framework collected 14 measurements. This
number includes 4 measurements to warm up the code. The Samples property
excludes warm-up measurements.
Display the sample measurements for the first test.
results(1).Samples
ans =
Name MeasuredTime Timestamp Host Platform Version RunIdentifier
______________________________ ____________ ____________________ ___________ ________ _____________________ ____________________________________
fprintfTest/testPrintingToFile 0.067772 02-Jan-2016 18:24:52 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
fprintfTest/testPrintingToFile 0.085359 02-Jan-2016 18:24:53 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
fprintfTest/testPrintingToFile 0.075863 02-Jan-2016 18:24:53 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
fprintfTest/testPrintingToFile 0.068161 02-Jan-2016 18:24:53 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
fprintfTest/testPrintingToFile 0.067606 02-Jan-2016 18:24:53 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
fprintfTest/testPrintingToFile 0.073692 02-Jan-2016 18:24:54 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
fprintfTest/testPrintingToFile 0.070815 02-Jan-2016 18:24:54 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
fprintfTest/testPrintingToFile 0.067791 02-Jan-2016 18:24:54 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
fprintfTest/testPrintingToFile 0.077599 02-Jan-2016 18:24:54 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
fprintfTest/testPrintingToFile 0.07438 02-Jan-2016 18:24:55 MY-HOSTNAME win64 9.0.0.320924 (R2016a) 9b6a0d5c-5fe7-4d26-8479-222792127ebc
Display the mean measured time for the first test. To exclude
data collected in the warm-up runs, use the values in the Samples field.
sampleTimes = results(1).Samples.MeasuredTime; meanTest = mean(sampleTimes)
meanTest =
0.0729Determine the average time for all the test elements. The fprintfTest test
includes two different methods. Compare the time for each method (test
element).
Since the performance testing framework returns a Samples table
for each test element, concatenate all these tables into one table.
Then group the rows by test element Name, and compute
the mean MeasuredTime for each group.
fullTable = vertcat(results.Samples); summaryStats = varfun(@mean,fullTable,... 'InputVariables','MeasuredTime','GroupingVariables','Name')
summaryStats =
Name GroupCount mean_MeasuredTime
______________________________ __________ _________________
fprintfTest/testPrintingToFile 10 0.072904
fprintfTest/testBytesToFile 27 0.079338 Both test methods write the same amount of data to a file. Therefore,
some of the difference between the mean values is attributed to calling
the fprintf function with an output argument.
Change the statistical objectives defined by the runperf function
by constructing and running a time experiment. Construct a time experiment
with measurements that reach a sample mean with a 3% relative margin
of error within a 97% confidence level. Collect eight warm-up measurements.
Construct an explicit test suite.
suite = testsuite('fprintfTest');Construct a time experiment with a variable number of sample measurements, and run the tests.
import matlab.perftest.TimeExperiment experiment = TimeExperiment.limitingSamplingError('NumWarmups',8,... 'RelativeMarginOfError',0.03, 'ConfidenceLevel', 0.97); resultsTE = run(experiment,suite);
Running fprintfTest .......... .......... .......... ..........Warning: The target Relative Margin of Error was not met after running the MaxSamples for fprintfTest/testPrintingToFile. .......... .......... .......... ..........Warning: The target Relative Margin of Error was not met after running the MaxSamples for fprintfTest/testBytesToFile. Done fprintfTest __________
In this example output, the performance testing framework is not able to meet the stricter statistical objectives with the default number of maximum samples. Your results might vary.
Compute the statistics for all the test elements.
fullTableTE = vertcat(resultsTE.Samples); summaryStatsTE = varfun(@mean,fullTableTE,... 'InputVariables','MeasuredTime','GroupingVariables','Name')
summaryStatsTE =
Name GroupCount mean_MeasuredTime
______________________________ __________ _________________
fprintfTest/testPrintingToFile 32 0.081782
fprintfTest/testBytesToFile 32 0.076378 Increase the maximum number of samples to 100 and rerun the time experiment.
experiment = TimeExperiment.limitingSamplingError('NumWarmups',2,... 'RelativeMarginOfError',0.03,'ConfidenceLevel',0.97,'MaxSamples',100); resultsTE = run(experiment,suite);
Running fprintfTest .......... .......... .......... .......... .......... .......... .......... .......... .......... .......... .......... .. Done fprintfTest __________
Compute the statistics for all the test elements.
fullTableTE = vertcat(resultsTE.Samples); summaryStatsTE = varfun(@mean,fullTableTE,... 'InputVariables','MeasuredTime','GroupingVariables','Name')
summaryStatsTE =
Name GroupCount mean_MeasuredTime
______________________________ __________ _________________
fprintfTest/testPrintingToFile 55 0.07783
fprintfTest/testBytesToFile 53 0.079008 The testing framework achieves the statistical objectives for both tests in approximately 50 samples.
Start a new MATLAB® session. A new session ensures that MATLAB has not run the code contained in your tests.
Measure the first-time cost of your code by creating and running a fixed time experiment with zero warm-up measurements and one sample measurement.
Construct an explicit test suite. Since you are measuring the first-time cost of the function, run a single test. To run multiple tests, save the results and start a new MATLAB session between tests.
suite = testsuite('fprintfTest/testPrintingToFile');Construct and run the time experiment.
import matlab.perftest.TimeExperiment
experiment = TimeExperiment.withFixedSampleSize(1);
results = run(experiment,suite);Running fprintfTest . Done fprintfTest __________
Display the results. Observe the TestActivity table
to ensure there are no warm-up samples.
fullTable = results.TestActivity
fullTable =
Name Passed Failed Incomplete MeasuredTime Objective Timestamp Host Platform Version TestResult RunIdentifier
______________________________ ______ ______ __________ ____________ _________ ____________________ ___________ ________ _____________________ ________________________________ ____________________________________
fprintfTest/testPrintingToFile true false false 0.065501 sample 06-Jan-2016 09:53:44 MY-HOSTNAME win64 9.0.0.323070 (R2016a) [1x1 matlab.unittest.TestResult] e0fe27aa-3224-46d1-a05a-542e9b8d9edb
The performance testing framework collects one sample is collected for each test.
matlab.perftest.TestCase | matlab.perftest.TimeExperiment | matlab.unittest.measurement.MeasurementResult | runperf | testsuite