Load Big Data for Simulations

Simulating models with many time steps and signals can use and create data that is too large to fit into working memory on your computer. When your simulation input data does not fit into memory, you can choose one of several strategies to use that data as simulation input with root-level Inport blocks. These strategies work for loading data stored in Dataset format in a Version 7.3 MAT-file, including data logged from another simulation.

1. When individual input signals are too large to fit into memory, you can use a matlab.io.datastore.SimulationDatastore object to access the signal data. The data from the SimulationDatastore object loads into the simulation incrementally in chunks that fit into memory.

2. When your simulation inputs are specified by a Simulink.SimulationData.Dataset object in a file that is too large to load into memory, you can stream the entire contents of the Dataset object into your model using a Simulink.SimulationData.DatasetRef object.

3. When the signals fit into memory and are stored in a file that is too large to load into memory, you can load individual signals from the file into memory using a Simulink.SimulationData.DatasetRef object.

Note

When you want to use data logged in one simulation as input for another, you can also stream data into the model using a matlab.io.datastore.sdidatastore object. The sdidatastore object references data in the Simulation Data Inspector repository on disk, so you do not have to save the logged data to a file. Consider using a sdidatastore object as simulation input for iterative workflows.

All big data loading strategies are for the special case when your data does not fit into memory and can require extra steps. These examples use data that fits fully into memory to illustrate the steps required for big data loading. When your simulation inputs fit into memory, consider using other loading techniques.

Stream Individual Signals Using SimulationDatastore Objects

When individual signals in your input data are too large to fit into memory, you can create matlab.io.datastore.SimulationDatastore objects for those signals and stream them into your model. To create a SimulationDatastore object for a signal you want to stream into your model, first create a Simulink.SimulationData.DatasetRef object to reference the Dataset object that contains your signal of interest. For example, create a DatasetRef for logged data from a simulation of the slexAircraftExample model.

logsout_DSR = Simulink.SimulationData.DatasetRef('aircraftData.mat','logsout');

You can create a SimulationDatastore object for your desired signal by indexing into the DatasetRef object with curly braces or using the getAsDatastore method for the DatasetRef object. In both cases, the SimulationDatastore object exists in the Values property of the returned Simulink.SimulationData.Signal object.

When you know the index of the signal within the Dataset object, you can index into the DatasetRef object with curly braces to create a SimulationDatastore for your signal.

alphaRad_ds = logsout_DSR{4}

alphaRad_ds = 
  Simulink.SimulationData.Signal
  Package: Simulink.SimulationData

  Properties:
              Name: 'alpha, rad'
    PropagatedName: ''
         BlockPath: [1x1 Simulink.SimulationData.BlockPath]
          PortType: 'outport'
         PortIndex: 4
            Values: [1x1 matlab.io.datastore.SimulationDatastore]

To create a SimulationDatastore object for a signal using the signal index, name, or block path, use the getAsDatastore method. For example, create a SimulationDatastore object for the Stick signal.

stick_ds = logsout_DSR.getAsDatastore('Stick')

stick_ds = 
  Simulink.SimulationData.Signal
  Package: Simulink.SimulationData

  Properties:
              Name: 'Stick'
    PropagatedName: ''
         BlockPath: [1x1 Simulink.SimulationData.BlockPath]
          PortType: 'outport'
         PortIndex: 1
            Values: [1x1 matlab.io.datastore.SimulationDatastore]

Because the Values properties of the stick_ds and alphaRad_ds Simulink.SimulationData.Signal objects are SimulationDatastores, the signal data streams into your model. You can include a SimulationDatastore backed Signal object as an element in a Dataset object or as an item in the Input parameter comma-separated list.

Stream Entire Dataset Using DatasetRef Object

When your simulation inputs are saved in a Dataset object in a file that is too large to load into memory, you can create a Simulink.SimulationData.DatasetRef object to stream your simulation inputs into your model. When you specify a DatasetRef object as the value for the Input parameter on the Data Import/Export pane of the model Configuration Parameters, all the signals in the Dataset object used to create the DatasetRef stream into your model during simulation. Use another big data loading technique to load or stream individual signals from a Dataset object.

When the file that contains the Dataset object with your simulation inputs also contains other data, you can use the Simulink.SimulationData.DatasetRef.getDatasetVariableNames function to view a list of the Dataset objects contained in the file. Previewing the names of variables in the file that contain Dataset objects is particularly useful when the file contents do not fit into memory.

dsNames = Simulink.SimulationData.DatasetRef.getDatasetVariableNames('aircraftData.mat')

dsNames = 1x3 cell
    {'logsout'}    {'xout'}    {'yout'}

Create a DatasetRef object that references the data in the Dataset object logsout.

logsout_DSR = Simulink.SimulationData.DatasetRef('aircraftData.mat','logsout');

You can load logsout_DSR using the Input parameter the same way you would load a Simulink.SimulationData.Dataset object. Each signal in the Dataset object used to create the DatasetRef streams into the model in chunks that fit into memory.

Load Individual Signals from a DatasetRef Object

When your simulation input signals individually fit into memory and are stored in a Dataset object in a file that does not fit into memory, use a Simulink.SimulationData.DatasetRef object to load each signal of interest into memory. Then, you can load the signals as simulation inputs for your model.

First, create the DatasetRef object to reference the Dataset object in the file that contains the signals you want to load. For example, create a DatasetRef object for data logged to file from a simulation of the slexAircraftExample model.

logsout_DSR = Simulink.SimulationData.DatasetRef('aircraftData.mat','logsout');

You can use the get or getElement methods to load individual signals into memory with the DatasetRef object. Both methods load the specified element into memory, using the same syntax. You can specify the signal you want to load into memory using its index within the Dataset object or its name. If you don't know the name of the signal you want to load, use the getElementNames method to see the names of the elements in the Dataset object referenced by the DatasetRef object.

elNames = logsout_DSR.getElementNames

elNames = 15x1 cell
    {0x0 char    }
    {0x0 char    }
    {0x0 char    }
    {'alpha, rad'}
    {'q, rad/sec'}
    {0x0 char    }
    {'qGust'     }
    {'wGust'     }
    {0x0 char    }
    {0x0 char    }
    {0x0 char    }
    {'Stick'     }
    {0x0 char    }
    {0x0 char    }
    {0x0 char    }

Load the qGust signal into memory using its name.

qGust = logsout_DSR.getElement('qGust')

qGust = 
  Simulink.SimulationData.Signal
  Package: Simulink.SimulationData

  Properties:
              Name: 'qGust'
    PropagatedName: ''
         BlockPath: [1x1 Simulink.SimulationData.BlockPath]
          PortType: 'outport'
         PortIndex: 2
            Values: [1x1 timeseries]

You can add the qGust signal to a Dataset object of simulation input signals to load to the root-level Inport blocks in your model, or you can specify qGust as an item in the Input parameter comma-separated list.