Run Parallel Simulations

Overview of Calling sim from Within parfor

The parfor command allows you to run parallel (simultaneous) Simulink® simulations of your model (design). In this context, parallel runs mean multiple model simulations at the same time on different workers. Calling sim from within a parfor loop often helps for performing multiple simulation runs of the same model for different inputs or for different parameter settings. For example, you can save simulation time performing parameter sweeps and Monte Carlo analyses by running them in parallel. Note that running parallel simulations using parfor does not currently support decomposing your model into smaller connected pieces and running the individual pieces simultaneously on multiple workers.

Normal, Accelerator, and Rapid Accelerator simulation modes are supported by sim in parfor. (See Choosing a Simulation Mode for details on selecting a simulation mode and Design Your Model for Effective Acceleration for optimizing simulation run times.) For other simulation modes, you need to address any workspace access issues and data concurrency issues to produce useful results. Specifically, the simulations need to create separately named output files and workspace variables. Otherwise, each simulation overwrites the same workspace variables and files, or can have collisions trying to write variables and files simultaneously.

For information on code regeneration and parameter handling in Rapid Accelerator mode, see Parameter Tuning in Rapid Accelerator Mode.

Also, see parfor.

    Note:   If you open models inside a parfor statement, close them again using bdclose all to avoid leaving temporary files behind.

The following examples show how to use sim in parfor using the sldemo_suspn_3dof model. To access these examples, from Simulink Documentation Center, click Examples > Modeling Features and navigate to Simulation Performance:

  • Parallel Simulations Using Parfor: Test-Case Sweep

  • Parallel Simulations Using Parfor: Parameter Sweep in Normal Mode

  • Parallel Simulations Using Parfor: Parameter Sweep in Rapid Accelerator Mode

Computationally Intensive Simulations

To further improve the performance of multiple simulations of the same model, you can use:

  • Multiple designated workers (MATLAB® computational engines) using the Parallel Computing Toolbox™ software

  • Multiple computer clusters, clouds, and grids using the MATLAB Distributed Computing Server™ software

To take advantage of these environments, you can:

  1. Simulate your model on a single computer.

  2. When satisfied with single simulation, run multiple simulations in parallel on multiple designated workers on a local multicore desktop.

  3. When satisfied with multiple simulations on local multicore desktop, run simulations in parallel remotely on computer clusters, clouds, and grids.

Consider the following:

ActionRequired Software*
MATLABSimulinkParallel Computing ToolboxMATLAB Distributed Computing Server

Run a single simulation.

   

Run multiple simulations simultaneously on your computer.

  

Run multiple simulations remotely on a server.

* And other required and optional software

Simulink and Parallel Computing Toolbox Software

Before you run simulations in the Parallel Computing Toolbox environment, see Parallel Computing Toolbox. You can use the Parallel Computing Toolbox software to run simulations on a local multicore computer or on multiple remote computers.

  1. Have a Parallel Computing Toolbox license.

  2. Run sim in parfor for your model.

For an example of using sim in parfor using Normal mode, see sim in parfor with Normal Mode.

For an example of using sim in parfor using Rapid Accelerator, see sim in parfor with Rapid Accelerator Mode.

    Note:   Code generation operations for the same model overwrite each other. If you want each worker to generate its own copy of code, attach and distribute the folder to each worker.

You can also use Parallel Computing Toolbox software to run simulations on multiple remote computers or in a nonhomogeneous environment. For these cases, you can use the Parallel Computing Toolbox software with the MATLAB Distributed Computing Server software.

Simulink and MATLAB Distributed Computing Server Software

Before you run simulations in the MATLAB Distributed Computing Server environment, see MATLAB Distributed Computing Server.

  1. Have Parallel Computing Toolbox and MATLAB Distributed Computing Server licenses.

  2. Select and configure your cluster configuration.

  3. Use the Parallel Computing Toolbox software to create, import, and select a default parallel cluster profile.

  4. Use the Parallel Computing Toolbox software to run sim in parfor for your model.

For an example of sim in parfor with Normal mode using MATLAB Distributed Computing Server software, see sim in parfor with Normal Mode and MATLAB Distributed Computing Server Software.

You can also use sim in parfor with Rapid Accelerator and MATLAB Distributed Computing Server software (see sim in parfor with Rapid Accelerator Mode). In that example, you build the code once and distribute that generated code to the other local workers. You can adapt sim in parfor with Rapid Accelerator Mode example for a remote cluster environment by:

  • Calling parpool with a cluster name as input

  • Attaching required files

For an example of how to adapt to a remote cluster environment, see sim in parfor with Normal Mode and MATLAB Distributed Computing Server Software.

sim in parfor with Normal Mode

This code fragment shows how you can use sim and parfor in Normal mode. Save changes to your model before simulating in parfor. The saved copy of your model is distributed to parallel workers when simulating in parfor.

% 1) Load model and initialize the pool.
model = 'sldemo_suspn_3dof';
load_system(model);
parpool;

% 2) Set up the iterations that we want to compute.
Cf                  = evalin('base', 'Cf');
Cf_sweep            = Cf*(0.05:0.1:0.95);
iterations          = length(Cf_sweep);
simout(iterations)  = Simulink.SimulationOutput;

% 3) Need to switch all workers to a separate tempdir in case 
% any code is generated for instance for StateFlow, or any other 
% file artifacts are  created by the model.
spmd
    % Setup tempdir and cd into it
    currDir = pwd;
    addpath(currDir);
    tmpDir = tempname;
    mkdir(tmpDir);
    cd(tmpDir);
    % Load the model on the worker
    load_system(model);
end

% 4) Loop over the number of iterations and perform the
% computation for different parameter values.
parfor idx=1:iterations   
    set_param([model '/Road-Suspension Interaction'],'MaskValues',...
        {'Kf',num2str(Cf_sweep(idx)),'Kr','Cr'});
    simout(idx) = sim(model, 'SimulationMode', 'normal');
end

% 5) Switch all of the workers back to their original folder.
spmd
    cd(currDir);
    rmdir(tmpDir,'s');
    rmpath(currDir);
    close_system(model, 0);
end

close_system(model, 0);
delete(gcp('nocreate'));

sim in parfor with Normal Mode and MATLAB Distributed Computing Server Software

This code fragment is identical to the one in sim in parfor with Normal Mode

. Modify it as follows for using sim and parfor in Normal mode:

  • In item 1, modify the parpool command as follows to create an object and use it to call a cluster name.

    p = parpool('clusterProfile');
% 'clusterProfile' is the name of the distributed cluster

  • In item 1, find files on which the model depends and attach those files to the model for distribution to cluster workers on remote machines.

    files = dependencies.fileDependencyAnalysis(modelName);
p.addAttachedFiles(files);

  • If you do not have a MATLAB Distributed Computing Server cluster, use your local cluster. For more information, see Clusters and Cluster Profiles.

Start your cluster before running the code.

% 1) Load model and initialize the pool.
model = 'sldemo_suspn_3dof';
load_system(model);
parpool;

% 2) Set up the iterations that we want to compute.
Cf                  = evalin('base', 'Cf');
Cf_sweep            = Cf*(0.05:0.1:0.95);
iterations          = length(Cf_sweep);
simout(iterations)  = Simulink.SimulationOutput;

% 3) Need to switch all workers to a separate tempdir in case 
% any code is generated for instance for StateFlow, or any other 
% file artifacts are  created by the model.
spmd
    % Setup tempdir and cd into it
    addpath(pwd);
    currDir = pwd;
    addpath(currDir);
    tmpDir = tempname;
    mkdir(tmpDir);
    cd(tmpDir);
    % Load the model on the worker
    load_system(model);
end

% 4) Loop over the number of iterations and perform the
% computation for different parameter values.
parfor idx=1:iterations   
    set_param([model '/Road-Suspension Interaction'],'MaskValues',...
        {'Kf',num2str(Cf_sweep(idx)),'Kr','Cr'});
    simout(idx) = sim(model, 'SimulationMode', 'normal');
end

% 5) Switch all of the workers back to their original folder.
spmd
    cd(currDir);
    rmdir(tmpDir,'s');
    rmpath(currDir);
    close_system(model, 0);
end

close_system(model, 0);
delete(gcp('nocreate'));

sim in parfor with Rapid Accelerator Mode

Running Rapid Accelerator simulations in parfor combines speed with automatic distribution of a prebuilt executable to the parfor workers. As a result, this mode eliminates duplication of the update diagram phase.

To run parallel simulations in Rapid Accelerator simulation mode using the sim and parfor commands:

  • Configure the model to run in Rapid Accelerator simulation mode.

  • Save changes to your model before simulating in parfor. The saved copy of your model is distributed to parallel workers when simulating in parfor.

  • Ensure that the Rapid Accelerator target is already built and up to date.

  • Disable the Rapid Accelerator target up-to-date check by setting the sim command option RapidAcceleratorUpToDateCheck to 'off'.

To satisfy the second condition, you can change parameters only between simulations that do not require a model rebuild. In other words, the structural checksum of the model must remain the same. Hence, you can change only tunable block diagram parameters and tunable run-time block parameters between simulations. For a discussion on tunable parameters that do not require a rebuild subsequent to their modifications, see Determine If the Simulation Will Rebuild.

To disable the Rapid Accelerator target up-to-date check, use the sim command, as shown in this sample.

parpool;
% Load the model and set parameters
model = 'vdp';
load_system(model);
% Build the Rapid Accelerator target
rtp = Simulink.BlockDiagram.buildRapidAcceleratorTarget(model);
% Run parallel simulations
parfor i=1:4
   simOut{i} = sim(model,'SimulationMode', 'rapid',...
               'RapidAcceleratorUpToDateCheck', 'off',...
               'SaveTime', 'on',...
               'StopTime', num2str(10*i));
   close_system(model, 0);
end

close_system(model, 0);
delete(gcp('nocreate'));

In this example, the call to the buildRapidAcceleratorTarget function generates code once. Subsequent calls to sim with the RapidAcceleratorUpToDateCheck option off guarantees that code is not regenerated. Data concurrency issues are thus resolved.

When you set RapidAcceleratorUpToDateCheck to 'off', changes that you make to block parameter values in the model (for example, by using block dialog boxes, by using the set_param function, or by changing the values of MATLAB variables) do not affect the simulation. Instead, use RapidAcceleratorParameterSets to pass new parameter values directly to the simulation.

For a detailed example of this method of running parallel simulations, refer to the Rapid Accelerator Simulations Using PARFOR.

Workspace Access Issues

Workspace Access for MATLAB worker sessions

By default, to run sim in parfor, a parallel pool opens automatically, enabling the code to run in parallel. Alternatively, you can also first open MATLAB workers using the parpool command. The parfor command then runs the code within the parfor loop in these MATLAB worker sessions. The MATLAB workers, however, do not have access to the workspace of the MATLAB client session where the model and its associated workspace variables have been loaded. Hence, if you load a model and define its associated workspace variables outside of and before a parfor loop, then neither is the model loaded, nor are the workspace variables defined in the MATLAB worker sessions where the parfor iterations are executed. This is typically the case when you define model parameters or external inputs in the base workspace of the client session. These scenarios constitute workspace access issues.

Transparency Violation

When you run sim in parfor with srcWorkspace set to current, Simulink uses the parfor workspace, which is a transparent workspace. Simulink then displays an error for transparency violation. For more information on transparent workspaces, see Specify Variables in parfor-Loops.

Resolving Workspace Access Issues

When a Simulink model is loaded into memory in a MATLAB client session, it is only visible and accessible in that MATLAB session; it is not accessible in the memory of the MATLAB worker sessions. Similarly, the workspace variables associated with a model that are defined in a MATLAB client session (such as parameters and external inputs) are not automatically available in the worker sessions. You must therefore ensure that the model is loaded and that the workspace variables referenced in the model are defined in the MATLAB worker session by using the following two methods.

  • In the parfor loop, use the sim command to load the model and to set parameters that change with each iteration. (Alternative: load the model and then use the g(s)et_param command(s) to set the parameters in the parfor loop)

  • In the parfor loop, use the MATLAB evalin and assignin commands to assign data values to variables.

Alternatively, you can simplify the management of workspace variables by defining them in the model workspace. These variables will then be automatically loaded when the model is loaded into the worker sessions. There are, however, limitations to this method. For example, you cannot store parameter objects that use a storage class other than Auto in a model workspace. For a detailed discussion on the model workspace, see Model Workspaces.

Specifying Parameter Values Using the sim Command

Use the sim command in the parfor loop to set parameters that change with each iteration.

%Specifying Parameter Values Using the sim Command
    
model = 'vdp';
load_system(model)
    
%Specifying parameter values. 
paramName = 'StopTime';
paramValue = {'10', '20', '30', '40'};
    
% Run parallel simulations
parfor i=1:4
    simOut{i} = sim(model, ...
                    paramName, paramValue{i}, ...
                    'SaveTime', 'on'); %#ok
end
    
close_system(model, 0);

An equivalent method is to load the model and then use the set_param command to set the paramName in the parfor loop.

Specifying Variable Values Using the assignin Command

You can pass the values of model or simulation variables to the MATLAB workers by using the assignin or the evalin command. The following example illustrates how to use this method to load variable values into the appropriate workspace of the MATLAB workers.

parfor i = 1:4
    assignin('base', 'extInp', paramValue{i})%#ok
    % 'extInp' is the name of the variable in the base 
    % workspace which contains the External Input data
    simOut{i} = sim(model, 'ExternalInput', 'extInp'); %#ok
end

For further details, see the Rapid Accelerator Simulations Using PARFOR example.

Data Concurrency Issues

Data concurrency issues refer to scenarios for which software makes simultaneous attempts to access the same file for data input or output. In Simulink, they primarily occur as a result of the nonsequential nature of the parfor loop during simultaneous execution of Simulink models. The most common incidences arise when code is generated or updated for a simulation target of a Stateflow®, Model block or MATLAB Function block during parallel computing. The cause, in this case, is that Simulink tries to concurrently access target data from multiple worker sessions. Similarly, To File blocks may simultaneously attempt to log data to the same files during parallel simulations and thus cause I/O errors. Or a third-party blockset or user-written S-function may cause a data concurrency issue while simultaneously generating code or files.

A secondary cause of data concurrency is due to the unprotected access of network ports. This type of error occurs, for example, when a Simulink product provides blocks that communicate via TCP/IP with other applications during simulation. One such product is the HDL Verifier™ for use with the Mentor Graphics® ModelSim® HDL simulator.

Resolving Data Concurrency Issues

The core requirement of parfor is the independence of the different iterations of the parfor body. This restriction is not compatible with the core requirement of simulation via incremental code generation, for which the simulation target from a prior simulation is reused or updated for the current simulation. Hence during the parallel simulation of a model that involves code generation (such as Accelerator mode simulation), Simulink makes concurrent attempts to access (update) the simulation target . However, you can avoid such data concurrency issues by creating a temporary folder within the parfor loop and then adding several lines of MATLAB code to the loop to perform the following steps:

  1. Change the current folder to the temporary, writable folder.

  2. In the temporary folder, load the model, set parameters and input vectors, and simulate the model.

  3. Return to the original, current folder.

  4. Remove the temporary folder and temporary path.

In this manner, you avoid concurrency issues by loading and simulating the model within a separate temporary folder. Following are examples that use this method to resolve common concurrency issues.

A Model with Stateflow, MATLAB Function Block, or Model Block

In this example, either the model is configured to simulate in Accelerator mode or it contains a Stateflow, a MATLAB Function block, or a Model block (for example, sf_bounce, sldemo_autotrans, or sldemo_mdlref_basic). For these cases, Simulink generates code during the initialization phase of simulation. Simulating such a model in parfor would cause code to be generated to the same files, while the initialization phase is running on the worker sessions. As illustrated below, you can avoid such data concurrency issues by running each iteration of the parfor body in a different temporary folder.

parfor i=1:4
   cwd = pwd;
   addpath(cwd)
   tmpdir = tempname;
   mkdir(tmpdir)
   cd(tmpdir)
   load_system(model)
   % set the block parameters, e.g., filename of To File block
   set_param(someBlkInMdl, blkParamName, blkParamValue{i})
   % set the model parameters by passing them to the sim command
   out{i} = sim(model, mdlParamName, mdlParamValue{i});
   close_system(model,0);
   cd(cwd)
   rmdir(tmpdir,'s')
   rmpath(cwd)
end

Note the following:

  • You can also avoid other concurrency issues due to file I/O errors by using a temporary folder for each iteration of the parfor body.

  • On Windows® platforms, consider inserting the evalin('base', 'clear mex'); command before rmdir(tmpdir, 's'). This sequence closes MEX-files first before calling rmdir to remove tmpdir.

evalin('base', 'clear mex');
rmdir(tmpdir, 's')

A Model with To File Blocks

If you simulate a model with To File blocks from inside of a parfor loop, the nonsequential nature of the loop may cause file I/O errors. To avoid such errors during parallel simulations, you can either use the temporary folder idea above or use the sim command in Rapid Accelerator mode with the option to append a suffix to the file names specified in the model To File blocks. By providing a unique suffix for each iteration of the parfor body, you can avoid the concurrency issue. 

rtp = Simulink.BlockDiagram.buildRapidAcceleratorTarget(model); 
       parfor idx=1:4 
       sim(model, ... 
           'ConcurrencyResolvingToFileSuffix', num2str(idx),... 
           'SimulationMode', 'rapid',... 
           'RapidAcceleratorUpToDateCheck', 'off'); 
        end

Related Examples

Was this topic helpful?
Simulink Documentation
Other Documentation
Support