DSP System Toolbox

Creating New Kinds of System Objects for File Input and Output

This example shows how to create and use four different System objects to facilitate the streaming of data in and out of MATLAB: dspdemo.TextFileReader, dspdemo.TextFileWriter, dspdemo.BinaryFilereader, and dspdemo.BinaryFileWriter.

The objects discussed in this example address a number of realistic use cases, and they can be customized to achieve advanced and more specialized tasks.

Introduction

This example shows how to create and use new types of System objects for file reading and writing. Internally, these System objects use standard low-level file I/O functions available in MATLAB (e.g. fscanf, fprintf, fread, fwrite). By abstracting away most usage details of those functions, they aim to make the task of reading and writing streamed data simpler and more efficient.

This example includes the use a number of advanced System object authoring constructs. For a more basic introduction to authoring System objects, please refer to the examples in the section Define New System ObjectsDefine New System Objects, of the DSP System Toolbox documentation.

The System object interface

System objects are MATLAB classes that derive from matlab.System. As a result, System objects all inherit a common public interface, which includes the following standard methods:

  • setup - to initialize the object, typically at the beginning of a simulation

  • reset - to clear the internal state of the object, bringing it back to its default post-initialization status

  • step - to execute the core functionality of the object, optionally accepting some input and/or returning some output

  • release - to release any resources (e.g. memory, hardware, or OS-specific) used internally by the object

When you create new kinds of System objects, you provide specific implementations for all the preceding methods to determine its behavior.

In this example we discuss the internal structure and the use of the following four System objects:

Definition of the class dspdemo.TextFileReader

All System objects in the previous list share a common structure. For example dspdemo.TextFileReaderdspdemo.TextFileReader includes the following sections

1. A class definition statement, which implies this class is derived from both matlab.System and matlab.system.mixin.FiniteSource.

classdef TextFileReader < matlab.System & matlab.system.mixin.FiniteSource
  • matlab.System is required, and is the base class for all System objects

  • matlab.system.mixin.FiniteSource indicates this class is a signal source with a finite number of data samples. This implies that, in addition to the usual interface, the System object will also expose the method isDone. When isDone returns true the object reached the end of the available data.

2. A number of public properties. In this case two are nontunable (they cannot be changed after the first call to step) and all have a default value. Default values are assigned to the corresponding properties when nothing else is specified by the user. Public properties can be changed by the user to adjust the behaviour of the object to his or her particular application.

   properties (Nontunable)
       Filename   = 'tempfile.txt'
       HeaderLines = 4
   end
   properties
       DataFormat = '%g'
       Delimiter = ','
       SamplesPerFrame = 1024
       PlayCount = 1
   end

3. A number of private properties. These are not visible to the user, and can serve a number of purposes, including

  • To keep hold of values computed only occasionally (e.g. at initialization time, when setup is called or when step is called for the first time) and then consumed by subsequent calls to step. This can save re-computing them at runtime, and hence improve the performance of the core functionality

  • To define the internal state of the object. For example pNumEofReached stores the number of times that the end-of-file indicator was reached

   properties(Access = private)
       pFID = -1
       pNumChannels
       pLineFormat
       pNumEofReached = 0
   end

4. A constructor. This is called when a new instance of dspdemo.TextDataReader is created by the user. Calling the method setProperties within the constructor allows users to set the properties of the object by providing name-value pairs at construction.

   methods
       function obj = TextFileReader(varargin)
           setProperties(obj, nargin, varargin{:});
       end
   end

5. A number of overridden methods from the matlab.System base class. The public methods common to all System objects each have corresponding protected methods that they call internally. The names of these protected methods all include an Impl postfix. They can be implemented when defining the class to program the specific behaviour of the particular System object.

For more information on the correspondence between the standard public methods and their internal implementations, please refer to Methods TimingMethods Timing.

For example, the particular implementation methods that are overridden for dspdemo.TextFileReaderdspdemo.TextFileReader are

  • setupImpl

  • resetImpl

  • stepImpl

  • releaseImpl

  • isDoneImpl

  • processTunedPropertiesImpl

  • loadObjectImpl

  • saveObjectImpl

6. A number of private methods. These methods are only accessible from within other methods of the same class. They can be used to make the rest of the code more readable. They can also improve code reusability, by grouping under separate routines code that is used multiple times in different parts of the class.

Write and read data - Introduction to the example

The code that follows gives a simple demonstration of how these new objects could be used. The following tasks are shown

  • Create a text file containing the samples of two different sinusoidal signals using dspdemo.TextFileWriter

  • Read from the text file using dspdemo.TextFileReader and write to a second file in binary form, this time using dspdemo.BinaryFileWriter

  • Read the signal samples cyclically from the new binary file using dspdemo.BinaryFileReader, and analyze the results graphically.

Create a simple text file containing the desired data

To start, a new text file is created to store two sinusoidal signals with frequencies 50 Hz and 60 Hz, respectively. For each signal, the data stored will be composed of 800 samples at a sampling rate of 8 kHz.

The following prepares the data

% Create data samples
fs = 8000;
tmax = 0.1;
t = (0:1/fs:tmax-1/fs)';
N = length(t);
f = [50,60];
data = sin(2*pi*t*f);

% Optionally, form a header string to describe the data in a readable way
% for future use
fileheader = sprintf(['The following contains %d samples of two ',...
    'sinusoids,\nwith frequencies %d Hz and %d Hz and a sample rate of',...
    ' %d kHz\n\n'], N, f(1),f(2),fs/1000);

To store the signal to a text file, create an instance of a text file writer. The constructor of dspdemo.TextFileWriter needs the name of the target file and some optional parameters, which can be passed in as name-value pairs.

TxtWriter = dspdemo.TextFileWriter('Filename','sinewaves.txt',...
    'Header',fileheader) %#ok<NOPTS>
TxtWriter = 

  System: dspdemo.TextFileWriter 

  Properties:
      Filename: 'sinewaves.txt'
        Header: [1x114 char]   
    DataFormat: '%.18g'        
     Delimiter: ','            
                               

dspdemo.TextFileWriter writes data to delimiter-separated ASCII files. Its public properties include the following

  • Filename: the name of the file to be written. If a file with this name already exists, it is overwritten. When operations start, the object begins writing to the file immediately following the header - it then appends new data at each subsequent call to step, until it is released. Calling reset resumes writing from the beginning of the file.

  • Header: a character string, often composed of multiple lines and terminated by a newline character (\n). This is specified by the user and can be modified to embed human-readable information that describes the actual data.

  • DataFormat: the format used to store each data sample. This can take any value assignable as Conversion Specifier within the formatSpecformatSpec string used by the built-in MATLAB function fwritefwrite. DataFormat applies to all channels written to the file. The default for this property is '%.18g', which allows saving double precision floating point data in full precision.

  • Delimiter: the character used to separate samples from different channels at the same time instant. Every line of the written file maps to a time instant, and it includes as many samples as the number of channels provided as input (i.e. the number of columns in the matrix input passed to step).

To write all the available data to the file, a single call to step can be used as follows

% Write to file by calling step on FileWriter with data as input
step(TxtWriter, data)

% Release control of file
release(TxtWriter)

The data is now stored in the new file. To inspect the file visually type edit('sinewaves.txt')edit('sinewaves.txt'). Because of the header, note that the data starts on line 4, following the 3 lines of the header.

In this simple case, the length of the whole signal is small and it fits comfortably on system memory. Therefore the data can be created all at once and written to a file in a single step as just shown.

There are cases when this approach is not possible or practical. For example the data may be too large to fit into a single MATLAB variable (i.e. too large to fit on system memory), or it may be created cyclically in a loop, or streamed into MATLAB from an external source. In all these cases it can be convenient to stream the data into the file by using an approach similar to the following

% Use a streamed sine wave generator to create a frame of data per step
frameLength = 32;
SineWave = dsp.SineWave('Frequency',[50,60], 'SampleRate', fs, ...
    'SamplesPerFrame', frameLength);

% Run the desired number of iterations to create the data and store it into
% the file
numCycles = N/frameLength;
for k = 1:numCycles
    dataFrame = step(SineWave);
    step(TxtWriter,dataFrame)
end

% Release control of file and sine wave generator
release(TxtWriter)
release(SineWave)

The two previous approaches yield the exact same results, which can be verified by inspecting sinewaves.txtsinewaves.txt.

Read from existing text file and write to new binary file

The next step consists in reading the data from the newly-created file, and writing it into a new binary file.

To read from the text file, create an instance of dspdemo.TextFileReader.

% Create a text file reader
TxtReader = dspdemo.TextFileReader('Filename','sinewaves.txt',...
    'HeaderLines',3,'SamplesPerFrame',frameLength) %#ok<NOPTS>
TxtReader = 

  System: dspdemo.TextFileReader 

  Properties:
           Filename: 'sinewaves.txt'
        HeaderLines: 3              
         DataFormat: '%g'           
          Delimiter: ','            
    SamplesPerFrame: 32             
          PlayCount: 1              
                                    

dspdemo.TextFileReader reads numeric data from delimiter-separated ASCII files. It has a similar set of properties as dspdemo.TextFileWriter. Some differences follow

  • HeaderLines captures the number of lines used by the header within the file specified in Filename. The first call to step starts reading from line number HeaderLines+1. Subsequent calls to step keep reading from the line immediately following the previously read line. Calling reset will resume reading from line HeaderLines+1

  • Delimiter is again the character used to separate samples from different channels at the same time instant. In this case it is also used to determine the number of data channels stored in the file: when step is first called, the object counts the number of Delimiter characters at line HeaderLines+1, say numDel; it then assumes for every time instant it needs to read numChan = numDel+1 numeric values with format DataFormat. The matrix returned by step has size SamplesPerFrame x numChan

  • SamplesPerFrame is the number of lines read per step, i.e. the number of rows of the matrix returned as output. When the last available data rows are reached, it can happen that they are fewer than the required SamplesPerFrame. In that case, the available data are padded with zeros to obtain a matrix of size SamplesPerFrame x numChan. Once all the data are read, step simply returns zeros(SamplesPerFrame,numChan) until reset or release is called.

  • PlayCount is the number of times the data in the file is read cyclically. If the object reaches the end of the file, and the file has not yet been read a number of times equal to PlayCount, reading resumes from the beginning of the data (i.e. line HeaderLines+1). If the last lines of the file do not provide enough samples to form a complete output matrix of size SamplesPerFrame x numChan, then the frame is completed using the initial data. Once the file is read PlayCount times, the output matrix returned by step is filled with zeros, and all calls to isDone return true unless reset or release is called. To loop through the available data indefinitely, PlayCount can be set to Inf.

To write to a new binary file, create an instance of dspdemo.BinaryFileWriter

BinWriter = dspdemo.BinaryFileWriter('Filename','sinewaves.bin',...
    'Header',fileheader) %#ok<NOPTS>
BinWriter = 

  System: dspdemo.BinaryFileWriter 

  Properties:
    Filename: 'sinewaves.bin'
      Header: [1x114 char]   
                             

dspdemo.BinaryFileWriter has similar scope to dspdemo.TextFileWriter, and the property names are mostly self-explanatory. The following differences in behaviour should be noted

  • The space used by Header is equal to its size in Bytes

  • The data passed as input to step is stored using its own data type. For example a double-precision floating-point input will use 8 Bytes per sample.

  • Data samples are stored in time order and multiple channels are interleaved. As a consequence, the input matrices passed to step are appended to the file using a row-major approach (i.e. the first row is stored first, left to right, then the second row, and so on).

To transfer the data from the text file into the binary file, the more general streamed approach is used. This is also relevant to dealing with very large data files.

% Write binary data using single-precision floating point
% Preallocate a data frame with frameLength rows and 2 columns
dataFrame = zeros(frameLength,2,'single');

% Read from the text file and write to the binary file, whilst data is
% present in the source text file. Notice how the method isDone is used
% to control the execution of the while loop
while(~isDone(TxtReader))
    dataFrame(:) = step(TxtReader);
    step(BinWriter, dataFrame)
end

% Release control of both files
release(TxtReader)
release(BinWriter)

Read content of binary file cyclically

sinewaves.bin holds a whole number of periods for both sine waves, i.e. 5 periods at 50 Hz and 6 at 60 Hz. Such signals can be read cyclically and used to generate sine waves of arbitrary length. In the last part of this demonstration dspdemo.BinaryFileReader is used to do exactly this. As data are read, the two sine waves are visualized in the time domain and their product is analyzed in the frequency domain.

Because the file is now read cyclically, a frame length greater than the actual number of samples stored in the file can be used.

frameLength = 1024;

% Create an instance of a binary file reader
BinReader = dspdemo.BinaryFileReader('Filename','sinewaves.bin',...
    'HeaderBytes',numel(fileheader),'NumChannels',2,'DataType','single',...
    'SamplesPerFrame',frameLength,'PlayCount',inf) %#ok<NOPTS>
BinReader = 

  System: dspdemo.BinaryFileReader 

  Properties:
           Filename: 'sinewaves.bin'
        HeaderBytes: 114            
           DataType: 'single'       
        NumChannels: 2              
    SamplesPerFrame: 1024           
          PlayCount: Inf            
                                    

The interface of dspdemo.BinaryFileReader is also mostly self-explanatory. The following are worth noticing

  • HeaderBytes defines the number of Bytes used by the header at the beginning of the file. Notice how the number provided, in Bytes, matches the number of elements in the variable fileHeader, previously assigned to BinWriter.Header

  • NumChannels specifies how many interleaved samples are expected for each time instant. It also determines the number of columns of the output matrix returned by step. step returns a matrix of size SamplesPerFrame x NumChannels, stored in the file in a row-major fashion

  • PlayCount is the number of times the data in the file is read cyclically. Setting this to inf in this case forces the object to wrap around read operations indefinitely. It also implies isDone always returns false, so it cannot be used to control the termination of a while streaming loop.

The following creates components that help with the visual analysis of the streamed signals, both in the time and in the frequency domain

For time-domain visualization, an instance of dsp.TimeScope is created. This is used to plot all data frames for both sine waves, as they are read from the file.

TimeScope = dsp.TimeScope('SampleRate',fs,'TimeSpan',frameLength/fs,...
    'ShowGrid',true);

For frequency domain visualization, we create an instance of dsp.SpectrumAnalyzer. This is used to analyze the spectrum of the product between the two sine waves, expecting two tonal components at (60-50)Hz and (60+50)Hz, respectively.

Because the ratio between the frequencies in the signal and the sample rate is very low, the signals are first decimated by a factor of 16 with an instance of dsp.SampleRateConverter. This brings the sample rate down to 500 Hz and makes the frequency components of the target signal more easily identifiable using a standard spectral analysis.

RateConverter = dsp.SampleRateConverter('InputSampleRate', 8000, ...
    'OutputSampleRate', 500, 'Bandwidth', 100);

% Note how a single spectral snapshot with the following settings
% requires 16384 input samples, far more than the 800 actually stored in
% the file
SpectAnalyzer = dsp.SpectrumAnalyzer(...
    'FrequencyResolutionMethod', 'WindowLength', 'WindowLength', 1024,...
    'FFTLengthSource', 'Property', 'FFTLength', 2048, 'SampleRate',500,...
    'PlotAsTwoSidedSpectrum', false, 'SpectralAverages', 16);

The following while loop reads the data from the file and visualizes the signals as they are read. It runs for 10 minutes worth of simulation time, regardless of the actual number of samples in the file. The plots in the two streamed visualizations match the expected behaviour.

simtime = 0;
% Run the loop until the simulation time is less than 10*60 seconds
while(simtime < 600)
    % Read from binary file, 1024 samples per frame
    dataFrame = step(BinReader);
    % Visualize a single frame of both channels in the time domain
    step(TimeScope, dataFrame)
    % Decimate the two channels down to a new sample rate of 500 Hz,
    % resulting in a new frame length of 64 samples
    dataDecimated = step(RateConverter, dataFrame);
    % Analyze the product of the two sine waves in the frequency domain, by
    % accumulating multiple data frames internally and updating the
    % visualization when ready
    step(SpectAnalyzer, prod(dataDecimated,2))

    % Update value of simulation time elapsed
    simtime = simtime + frameLength/fs;
end

% Release control of files and scopes
release(BinReader)
release(RateConverter)
release(TimeScope)
release(SpectAnalyzer)

Summary

This example authored and used System objects to read from and write to numeric data files. All objects used (i.e. dspdemo.TextFileReader, dspdemo.TextFileWriter, dspdemo.BinaryFileReader, and dspdemo.BinaryFileWriter) can be edited to perform special-purpose file reading and writing operations.

For more information on authoring System objects for custom algorithms, see Define New System ObjectsDefine New System Objects.