Documentation

Custom Plugins Overview

TestRunnerPlugin methods have three levels: Test Suite, Test Class, and Test. At each level, you implement methods to extend the creation, setup, run, and teardown of tests or test fixtures. The TestRunner runs these methods as shown in the figure.

Additionally, the reportFinalizedResult method enables the test runner to report finalized test results. A test result is finalized when no remaining test content can modify the results. The test runner determines if it invokes the reportFinalizedResult method at each level.

The creation methods are the only set of TestRunnerPlugin methods with an output argument. Typically, you extend the creation methods to listen for various events originating from the test content at the corresponding level. Since both TestCase and Fixture instances inherit from the handle class, you add these listeners using the addlistener method. The methods that set up, run and tear down test content extend the way the TestRunner evaluates the test content.

Extending Test Level Plugin Methods

The TestRunnerPlugin methods at the test level extend the creation, setup, run, and teardown of a single test suite element. A single test element consists of one test method or, if the test is parameterized, one instance of the test's parameterization.

At this level, the createTestMethodInstance method is the only plugin method with an output argument. It returns the TestCase instances created for each Test element. The test framework passes each of these instances into corresponding Test methods, and into any methods with the TestMethodSetup or TestMethodTeardown attribute.

The test framework evaluates methods at the test level within the scope of the runTest method. Provided the test framework completes all TestMethodSetup work, it invokes the plugin methods in this level a single time per test element.

Extending Test Class Level Plugin Methods

The TestRunnerPlugin methods at the test class level extend the creation, setup, run, and teardown of test suite elements that belong to the same test class or the same function-based test. These methods apply to a subset of the full TestSuite that the TestRunner runs.

At this level, the createTestClassInstance method is the only plugin method with an output argument. It returns the TestCase instances created at the class level. For each class, the test framework passes the instance into any methods with the TestClassSetup or TestClassTeardown attribute.

A test class setup is parameterized if it contains properties with the ClassSetupParameter attribute. In this case, the test framework evaluates the setupTestClass and teardownTestClass methods as many times as the class setup parameterization dictates.

The run method at this level, runTest, extends the running of a single TestSuite element, and incorporates the functionality described for the test level plugin methods.

The test framework evaluates methods at the test class level within the scope of the runTestClass method. If TestClassSetup completes successfully, it invokes the runTest method one time for each element in the Test array. Each TestClassSetup parameterization invokes the creation, setup, and teardown methods a single time.

Extending Test Suite Level Plugin Methods

The TestRunnerPlugin methods at the test suite level extend the creation, setup, run, and teardown of shared test fixtures. These methods fall within the scope of runTestSuite.

At this level, the createSharedTestFixture method is the only plugin method with an output argument. It returns the Fixture instances for each shared fixture required by a test class. These fixture instances are available to the test through the getSharedTestFixtures method of TestCase.

The run method at this level, runTestClass, extends the running of tests that belong to the same test class or the same function-based test, and incorporates the functionality described for the test class level plugin methods.