fitcsvm

Train support vector machine (SVM) classifier for one-class and binary classification

Syntax

Mdl = fitcsvm(Tbl,ResponseVarName)

Mdl = fitcsvm(Tbl,formula)

Mdl = fitcsvm(Tbl,Y)

Mdl = fitcsvm(X,Y)

Mdl = fitcsvm(___,Name,Value)

Description

fitcsvm trains or cross-validates a support vector machine (SVM) model for one-class and two-class (binary) classification on a low-dimensional or moderate-dimensional predictor data set. fitcsvm supports mapping the predictor data using kernel functions, and supports sequential minimal optimization (SMO), iterative single data algorithm (ISDA), or L1 soft-margin minimization via quadratic programming for objective-function minimization.

To train a linear SVM model for binary classification on a high-dimensional data set, that is, a data set that includes many predictor variables, use fitclinear instead.

For multiclass learning with combined binary SVM models, use error-correcting output codes (ECOC). For more details, see fitcecoc.

To train an SVM regression model, see fitrsvm for low-dimensional and moderate-dimensional predictor data sets, or fitrlinear for high-dimensional data sets.

Mdl = fitcsvm(Tbl,ResponseVarName) returns a support vector machine (SVM) classifier Mdl trained using the sample data contained in the table Tbl. ResponseVarName is the name of the variable in Tbl that contains the class labels for one-class or two-class classification.

Mdl = fitcsvm(Tbl,formula) returns an SVM classifier trained using the sample data contained in the table Tbl. formula is an explanatory model of the response and a subset of the predictor variables in Tbl used to fit Mdl.

Mdl = fitcsvm(Tbl,Y) returns an SVM classifier trained using the predictor variables in the table Tbl and the class labels in vector Y.

Mdl = fitcsvm(X,Y) returns an SVM classifier trained using the predictors in the matrix X and the class labels in vector Y for one-class or two-class classification.

Mdl = fitcsvm(___,Name,Value) specifies options using one or more name-value pair arguments in addition to the input arguments in previous syntaxes. For example, you can specify the type of cross-validation, the cost for misclassification, and the type of score transformation function.

Examples

Train SVM Classifier

Load Fisher's iris data set. Remove the sepal lengths and widths and all observed setosa irises.

load fisheriris
inds = ~strcmp(species,'setosa');
X = meas(inds,3:4);
y = species(inds);

Train an SVM classifier using the processed data set.

SVMModel = fitcsvm(X,y)

SVMModel = 
  ClassificationSVM
             ResponseName: 'Y'
    CategoricalPredictors: []
               ClassNames: {'versicolor'  'virginica'}
           ScoreTransform: 'none'
          NumObservations: 100
                    Alpha: [24x1 double]
                     Bias: -14.4149
         KernelParameters: [1x1 struct]
           BoxConstraints: [100x1 double]
          ConvergenceInfo: [1x1 struct]
          IsSupportVector: [100x1 logical]
                   Solver: 'SMO'


  Properties, Methods

SVMModel is a trained ClassificationSVM classifier. Display the properties of SVMModel. For example, to determine the class order, use dot notation.

classOrder = SVMModel.ClassNames

classOrder = 2x1 cell
    {'versicolor'}
    {'virginica' }

The first class ('versicolor') is the negative class, and the second ('virginica') is the positive class. You can change the class order during training by using the 'ClassNames' name-value pair argument.

Plot a scatter diagram of the data and circle the support vectors.

sv = SVMModel.SupportVectors;
figure
gscatter(X(:,1),X(:,2),y)
hold on
plot(sv(:,1),sv(:,2),'ko','MarkerSize',10)
legend('versicolor','virginica','Support Vector')
hold off

The support vectors are observations that occur on or beyond their estimated class boundaries.

You can adjust the boundaries (and, therefore, the number of support vectors) by setting a box constraint during training using the 'BoxConstraint' name-value pair argument.

Train and Cross-Validate SVM Classifier

Load the ionosphere data set.

load ionosphere
rng(1); % For reproducibility

Train an SVM classifier using the radial basis kernel. Let the software find a scale value for the kernel function. Standardize the predictors.

SVMModel = fitcsvm(X,Y,'Standardize',true,'KernelFunction','RBF',...
    'KernelScale','auto');

SVMModel is a trained ClassificationSVM classifier.

Cross-validate the SVM classifier. By default, the software uses 10-fold cross-validation.

CVSVMModel = crossval(SVMModel);

CVSVMModel is a ClassificationPartitionedModel cross-validated classifier.

Estimate the out-of-sample misclassification rate.

classLoss = kfoldLoss(CVSVMModel)

classLoss = 0.0484

The generalization rate is approximately 5%.

Detect Outliers Using SVM and One-Class Learning

Modify Fisher's iris data set by assigning all the irises to the same class. Detect outliers in the modified data set, and confirm the expected proportion of the observations that are outliers.

Load Fisher's iris data set. Remove the petal lengths and widths. Treat all irises as coming from the same class.

load fisheriris
X = meas(:,1:2);
y = ones(size(X,1),1);

Train an SVM classifier using the modified data set. Assume that 5% of the observations are outliers. Standardize the predictors.

rng(1);
SVMModel = fitcsvm(X,y,'KernelScale','auto','Standardize',true,...
    'OutlierFraction',0.05);

SVMModel is a trained ClassificationSVM classifier. By default, the software uses the Gaussian kernel for one-class learning.

Plot the observations and the decision boundary. Flag the support vectors and potential outliers.

svInd = SVMModel.IsSupportVector;
h = 0.02; % Mesh grid step size
[X1,X2] = meshgrid(min(X(:,1)):h:max(X(:,1)),...
    min(X(:,2)):h:max(X(:,2)));
[~,score] = predict(SVMModel,[X1(:),X2(:)]);
scoreGrid = reshape(score,size(X1,1),size(X2,2));

figure
plot(X(:,1),X(:,2),'k.')
hold on
plot(X(svInd,1),X(svInd,2),'ro','MarkerSize',10)
contour(X1,X2,scoreGrid)
colorbar;
title('{\bf Iris Outlier Detection via One-Class SVM}')
xlabel('Sepal Length (cm)')
ylabel('Sepal Width (cm)')
legend('Observation','Support Vector')
hold off

The boundary separating the outliers from the rest of the data occurs where the contour value is 0.

Verify that the fraction of observations with negative scores in the cross-validated data is close to 5%.

CVSVMModel = crossval(SVMModel);
[~,scorePred] = kfoldPredict(CVSVMModel);
outlierRate = mean(scorePred<0)

outlierRate = 0.0467

Find Multiple Class Boundaries Using Binary SVM

Create a scatter plot of the fisheriris data set. Treat coordinates of a grid within the plot as new observations from the distribution of the data set, and find class boundaries by assigning the coordinates to one of the three classes in the data set.

Load Fisher's iris data set. Use the petal lengths and widths as the predictors.

load fisheriris
X = meas(:,3:4);
Y = species;

Examine a scatter plot of the data.

figure
gscatter(X(:,1),X(:,2),Y);
h = gca;
lims = [h.XLim h.YLim]; % Extract the x and y axis limits
title('{\bf Scatter Diagram of Iris Measurements}');
xlabel('Petal Length (cm)');
ylabel('Petal Width (cm)');
legend('Location','Northwest');

The data contains three classes, one of which is linearly separable from the others.

For each class:

1. Create a logical vector (indx) indicating whether an observation is a member of the class.

2. Train an SVM classifier using the predictor data and indx.

3. Store the classifier in a cell of a cell array.

Define the class order.

SVMModels = cell(3,1);
classes = unique(Y);
rng(1); % For reproducibility

for j = 1:numel(classes)
    indx = strcmp(Y,classes(j)); % Create binary classes for each classifier
    SVMModels{j} = fitcsvm(X,indx,'ClassNames',[false true],'Standardize',true,...
        'KernelFunction','rbf','BoxConstraint',1);
end

SVMModels is a 3-by-1 cell array, with each cell containing a ClassificationSVM classifier. For each cell, the positive class is setosa, versicolor, and virginica, respectively.

Define a fine grid within the plot, and treat the coordinates as new observations from the distribution of the training data. Estimate the score of the new observations using each classifier.

d = 0.02;
[x1Grid,x2Grid] = meshgrid(min(X(:,1)):d:max(X(:,1)),...
    min(X(:,2)):d:max(X(:,2)));
xGrid = [x1Grid(:),x2Grid(:)];
N = size(xGrid,1);
Scores = zeros(N,numel(classes));

for j = 1:numel(classes)
    [~,score] = predict(SVMModels{j},xGrid);
    Scores(:,j) = score(:,2); % Second column contains positive-class scores
end

Each row of Scores contains three scores. The index of the element with the largest score is the index of the class to which the new class observation most likely belongs.

Associate each new observation with the classifier that gives it the maximum score.

[~,maxScore] = max(Scores,[],2);

Color in the regions of the plot based on the class to which the corresponding new observation belongs.

figure
h(1:3) = gscatter(xGrid(:,1),xGrid(:,2),maxScore,...
    [0.1 0.5 0.5; 0.5 0.1 0.5; 0.5 0.5 0.1]);
hold on
h(4:6) = gscatter(X(:,1),X(:,2),Y);
title('{\bf Iris Classification Regions}');
xlabel('Petal Length (cm)');
ylabel('Petal Width (cm)');
legend(h,{'setosa region','versicolor region','virginica region',...
    'observed setosa','observed versicolor','observed virginica'},...
    'Location','Northwest');
axis tight
hold off

Optimize SVM Classifier

Optimize hyperparameters automatically using fitcsvm.

Load the ionosphere data set.

load ionosphere

Find hyperparameters that minimize five-fold cross-validation loss by using automatic hyperparameter optimization. For reproducibility, set the random seed and use the 'expected-improvement-plus' acquisition function.

rng default
Mdl = fitcsvm(X,Y,'OptimizeHyperparameters','auto', ...
    'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName', ...
    'expected-improvement-plus'))

|=====================================================================================================|
| Iter | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   | BoxConstraint|  KernelScale |
|      | result |             | runtime     | (observed)  | (estim.)    |              |              |
|=====================================================================================================|
|    1 | Best   |     0.20513 |       16.36 |     0.20513 |     0.20513 |       64.836 |    0.0015729 |
|    2 | Accept |     0.35897 |     0.11688 |     0.20513 |     0.21471 |     0.036335 |       5.5755 |
|    3 | Best   |     0.13105 |      6.4256 |     0.13105 |     0.14133 |    0.0022147 |    0.0023957 |
|    4 | Accept |     0.35897 |     0.11438 |     0.13105 |     0.13109 |       5.1259 |        98.62 |
|    5 | Accept |     0.13675 |      13.117 |     0.13105 |     0.13111 |    0.0011599 |    0.0010098 |
|    6 | Accept |     0.13675 |      2.8275 |     0.13105 |     0.13119 |    0.0010151 |    0.0059137 |
|    7 | Accept |      0.1339 |      8.0601 |     0.13105 |     0.13127 |    0.0010281 |    0.0027003 |
|    8 | Accept |     0.13675 |      13.277 |     0.13105 |     0.13232 |     0.016269 |    0.0024597 |
|    9 | Accept |     0.13105 |      10.552 |     0.13105 |     0.13137 |    0.0021526 |    0.0025081 |
|   10 | Best   |     0.12821 |      14.166 |     0.12821 |     0.12841 |    0.0086928 |    0.0010304 |
|   11 | Accept |     0.12821 |      14.547 |     0.12821 |     0.12828 |     0.010039 |    0.0010077 |
|   12 | Accept |     0.13675 |      14.233 |     0.12821 |     0.13162 |    0.0071238 |    0.0010245 |
|   13 | Accept |      0.1339 |      13.196 |     0.12821 |     0.13179 |    0.0050166 |    0.0016385 |
|   14 | Best   |     0.11966 |      14.624 |     0.11966 |     0.12919 |     0.013746 |    0.0010181 |
|   15 | Accept |     0.13105 |      15.437 |     0.11966 |      0.1294 |     0.025222 |     0.001012 |
|   16 | Accept |     0.13105 |      15.316 |     0.11966 |     0.12958 |     0.024019 |     0.001013 |
|   17 | Accept |     0.35897 |     0.16209 |     0.11966 |     0.12962 |    0.0010245 |       994.95 |
|   18 | Accept |     0.13675 |      14.017 |     0.11966 |     0.12925 |       990.28 |      0.36736 |
|   19 | Accept |     0.18234 |      16.694 |     0.11966 |     0.12917 |       949.33 |     0.082849 |
|   20 | Accept |      0.1339 |      6.0125 |     0.11966 |     0.12914 |       979.17 |       1.3107 |
|=====================================================================================================|
| Iter | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   | BoxConstraint|  KernelScale |
|      | result |             | runtime     | (observed)  | (estim.)    |              |              |
|=====================================================================================================|
|   21 | Accept |     0.14245 |       15.34 |     0.11966 |     0.13062 |     0.024598 |    0.0010041 |
|   22 | Accept |     0.13675 |      11.276 |     0.11966 |     0.13058 |          907 |      0.70254 |
|   23 | Accept |     0.35897 |    0.097641 |     0.11966 |     0.13062 |       999.82 |       995.02 |
|   24 | Accept |     0.15385 |    0.090972 |     0.11966 |     0.13042 |     0.001002 |      0.30762 |
|   25 | Accept |     0.12251 |     0.11237 |     0.11966 |     0.12275 |    0.0010105 |     0.056734 |
|   26 | Accept |      0.1339 |     0.19372 |     0.11966 |     0.12251 |     0.021279 |     0.054708 |
|   27 | Accept |     0.12821 |     0.10274 |     0.11966 |     0.12517 |    0.0010127 |     0.064566 |
|   28 | Accept |     0.12821 |     0.12003 |     0.11966 |     0.12557 |    0.0010014 |     0.036667 |
|   29 | Accept |      0.1339 |     0.10127 |     0.11966 |     0.12743 |    0.0010509 |     0.078319 |
|   30 | Accept |     0.12251 |      0.1367 |     0.11966 |     0.12706 |    0.0030057 |     0.028009 |

__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 278.6746 seconds
Total objective function evaluation time: 236.8276

Best observed feasible point:
    BoxConstraint    KernelScale
    _____________    ___________

      0.013746        0.0010181 

Observed objective function value = 0.11966
Estimated objective function value = 0.13091
Function evaluation time = 14.6241

Best estimated feasible point (according to models):
    BoxConstraint    KernelScale
    _____________    ___________

      0.0010105       0.056734  

Estimated objective function value = 0.12706
Estimated function evaluation time = 0.10367

Mdl = 
  ClassificationSVM
                         ResponseName: 'Y'
                CategoricalPredictors: []
                           ClassNames: {'b'  'g'}
                       ScoreTransform: 'none'
                      NumObservations: 351
    HyperparameterOptimizationResults: [1x1 BayesianOptimization]
                                Alpha: [117x1 double]
                                 Bias: -2.6521
                     KernelParameters: [1x1 struct]
                       BoxConstraints: [351x1 double]
                      ConvergenceInfo: [1x1 struct]
                      IsSupportVector: [351x1 logical]
                               Solver: 'SMO'


  Properties, Methods

Input Arguments

Tbl — Sample data

table

Sample data used to train the model, specified as a table. Each row of Tbl corresponds to one observation, and each column corresponds to one predictor variable. Optionally, Tbl can contain one additional column for the response variable. Multicolumn variables and cell arrays other than cell arrays of character vectors are not allowed.

• If Tbl contains the response variable, and you want to use all remaining variables in Tbl as predictors, then specify the response variable by using ResponseVarName.

• If Tbl contains the response variable, and you want to use only a subset of the remaining variables in Tbl as predictors, then specify a formula by using formula.

• If Tbl does not contain the response variable, then specify a response variable by using Y. The length of the response variable and the number of rows in Tbl must be equal.

Data Types: table

ResponseVarName — Response variable name

name of variable in Tbl

Response variable name, specified as the name of a variable in Tbl.

You must specify ResponseVarName as a character vector or string scalar. For example, if the response variable Y is stored as Tbl.Y, then specify it as 'Y'. Otherwise, the software treats all columns of Tbl, including Y, as predictors when training the model.

The response variable must be a categorical, character, or string array; a logical or numeric vector; or a cell array of character vectors. If Y is a character array, then each element of the response variable must correspond to one row of the array.

A good practice is to specify the order of the classes by using the ClassNames name-value argument.

Data Types: char | string

formula — Explanatory model of response variable and subset of predictor variables

character vector | string scalar

Explanatory model of the response variable and a subset of the predictor variables, specified as a character vector or string scalar in the form 'Y~x1+x2+x3'. In this form, Y represents the response variable, and x1, x2, and x3 represent the predictor variables.

To specify a subset of variables in Tbl as predictors for training the model, use a formula. If you specify a formula, then the software does not use any variables in Tbl that do not appear in formula.

The variable names in the formula must be both variable names in Tbl (Tbl.Properties.VariableNames) and valid MATLAB^® identifiers. You can verify the variable names in Tbl by using the isvarname function. If the variable names are not valid, then you can convert them by using the matlab.lang.makeValidName function.

Data Types: char | string

Y — Class labels

categorical array | character array | string array | logical vector | numeric vector | cell array of character vectors

Class labels to which the SVM model is trained, specified as a categorical, character, or string array, logical or numeric vector, or cell array of character vectors.

• Y must contain at most two distinct classes. For multiclass learning, see fitcecoc.

• If Y is a character array, then each element of the class labels must correspond to one row of the array.

• The length of Y and the number of rows in Tbl or X must be equal.

• It is a good practice to specify the class order by using the ClassNames name-value pair argument.

Data Types: categorical | char | string | logical | single | double | cell

X — Predictor data

matrix of numeric values

Predictor data to which the SVM classifier is trained, specified as a matrix of numeric values.

Each row of X corresponds to one observation (also known as an instance or example), and each column corresponds to one predictor (also known as a feature).

The length of Y and the number of rows in X must be equal.

To specify the names of the predictors in the order of their appearance in X, use the 'PredictorNames' name-value pair argument.

Data Types: double | single

Name-Value Pair Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside quotes. You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

Example: fitcsvm(X,Y,'KFold',10,'Cost',[0 2;1 0],'ScoreTransform','sign') performs 10-fold cross-validation, applies double the penalty to false positives compared to false negatives, and transforms scores using the sign function.

SVM Options

'BoxConstraint' — Box constraint

1 (default) | positive scalar

Box constraint, specified as the comma-separated pair consisting of 'BoxConstraint' and a positive scalar.

For one-class learning, the software always sets the box constraint to 1.

For more details on the relationships and algorithmic behavior of BoxConstraint, Cost, Prior, Standardize, and Weights, see Algorithms.

Example: 'BoxConstraint',100

Data Types: double | single

'KernelFunction' — Kernel function

'linear' | 'gaussian' | 'rbf' | 'polynomial' | function name

Kernel function used to compute the elements of the Gram matrix, specified as the comma-separated pair consisting of 'KernelFunction' and a kernel function name. Suppose G(x_j,x_k) is element (j,k) of the Gram matrix, where x_j and x_k are p-dimensional vectors representing observations j and k in X. This table describes supported kernel function names and their functional forms.

Kernel Function Name	Description	Formula
'gaussian' or 'rbf'	Gaussian or Radial Basis Function (RBF) kernel, default for one-class learning	$$G\left(x_j,x_k\right)=\exp \left(-{\Vert x_j-x_k\Vert }^2\right)$$
'linear'	Linear kernel, default for two-class learning	$$G(x_j,x_k)=x_j\prime x_k$$
'polynomial'	Polynomial kernel. Use 'PolynomialOrder',q to specify a polynomial kernel of order q.	$$G(x_j,x_k)={(1+x_j\prime x_k)}^q$$

You can set your own kernel function, for example, kernel, by setting 'KernelFunction','kernel'. The value kernel must have this form.

function G = kernel(U,V)

where:

• U is an m-by-p matrix. Columns correspond to predictor variables, and rows correspond to observations.

• V is an n-by-p matrix. Columns correspond to predictor variables, and rows correspond to observations.

• G is an m-by-n Gram matrix of the rows of U and V.

kernel.m must be on the MATLAB path.

It is a good practice to avoid using generic names for kernel functions. For example, call a sigmoid kernel function 'mysigmoid' rather than 'sigmoid'.

Example: 'KernelFunction','gaussian'

Data Types: char | string

'KernelScale' — Kernel scale parameter

1 (default) | 'auto' | positive scalar

Kernel scale parameter, specified as the comma-separated pair consisting of 'KernelScale' and 'auto' or a positive scalar. The software divides all elements of the predictor matrix X by the value of KernelScale. Then, the software applies the appropriate kernel norm to compute the Gram matrix.

• If you specify 'auto', then the software selects an appropriate scale factor using a heuristic procedure. This heuristic procedure uses subsampling, so estimates can vary from one call to another. Therefore, to reproduce results, set a random number seed using rng before training.

• If you specify KernelScale and your own kernel function, for example, 'KernelFunction','kernel', then the software throws an error. You must apply scaling within kernel.

Example: 'KernelScale','auto'

Data Types: double | single | char | string

'PolynomialOrder' — Polynomial kernel function order

3 (default) | positive integer

Polynomial kernel function order, specified as the comma-separated pair consisting of 'PolynomialOrder' and a positive integer.

If you set 'PolynomialOrder' and KernelFunction is not 'polynomial', then the software throws an error.

Example: 'PolynomialOrder',2

Data Types: double | single

'KernelOffset' — Kernel offset parameter

nonnegative scalar

Kernel offset parameter, specified as the comma-separated pair consisting of 'KernelOffset' and a nonnegative scalar.

The software adds KernelOffset to each element of the Gram matrix.

The defaults are:

• 0 if the solver is SMO (that is, you set 'Solver','SMO')

• 0.1 if the solver is ISDA (that is, you set 'Solver','ISDA')

Example: 'KernelOffset',0

Data Types: double | single

'Standardize' — Flag to standardize predictor data

false (default) | true

Flag to standardize the predictor data, specified as the comma-separated pair consisting of 'Standardize' and true (1) or false (0).

If you set 'Standardize',true:

• The software centers and scales each predictor variable (X or Tbl) by the corresponding weighted column mean and standard deviation. For details on weighted standardizing, see Algorithms. MATLAB does not standardize the data contained in the dummy variable columns generated for categorical predictors.

• The software trains the classifier using the standardized predictors, but stores the unstandardized predictors as a matrix or table in the classifier property X.

Example: 'Standardize',true

Data Types: logical

'Solver' — Optimization routine

'ISDA' | 'L1QP' | 'SMO'

Optimization routine, specified as the comma-separated pair consisting of 'Solver' and a value in this table.

Value	Description
'ISDA'	Iterative Single Data Algorithm (see [30])
'L1QP'	Uses quadprog (Optimization Toolbox) to implement L1 soft-margin minimization by quadratic programming. This option requires an Optimization Toolbox™ license. For more details, see Quadratic Programming Definition (Optimization Toolbox).
'SMO'	Sequential Minimal Optimization (see [17])

The default value is 'ISDA' if you set 'OutlierFraction' to a positive value for two-class learning, and 'SMO' otherwise.

Example: 'Solver','ISDA'

'Alpha' — Initial estimates of alpha coefficients

numeric vector of nonnegative values

Initial estimates of alpha coefficients, specified as the comma-separated pair consisting of 'Alpha' and a numeric vector of nonnegative values. The length of Alpha must be equal to the number of rows in X.

• Each element of 'Alpha' corresponds to an observation in X.

• 'Alpha' cannot contain any NaNs.

• If you specify 'Alpha' and any one of the cross-validation name-value pair arguments ('CrossVal', 'CVPartition', 'Holdout', 'KFold', or 'Leaveout'), then the software returns an error.

If Y contains any missing values, then remove all rows of Y, X, and 'Alpha' that correspond to the missing values. That is, enter:

idx = ~isundefined(categorical(Y));
Y = Y(idx,:);
X = X(idx,:);
alpha = alpha(idx);

Then pass Y, X, and alpha as the response, predictors, and initial alpha estimates, respectively.

The default values are:

• 0.5*ones(size(X,1),1) for one-class learning

• zeros(size(X,1),1) for two-class learning

Example: 'Alpha',0.1*ones(size(X,1),1)

Data Types: double | single

'CacheSize' — Cache size

1000 (default) | 'maximal' | positive scalar

Cache size, specified as the comma-separated pair consisting of 'CacheSize' and 'maximal' or a positive scalar.

If CacheSize is 'maximal', then the software reserves enough memory to hold the entire n-by-n Gram matrix.

If CacheSize is a positive scalar, then the software reserves CacheSize megabytes of memory for training the model.

Example: 'CacheSize','maximal'

Data Types: double | single | char | string

'ClipAlphas' — Flag to clip alpha coefficients

true (default) | false

Flag to clip alpha coefficients, specified as the comma-separated pair consisting of 'ClipAlphas' and either true or false.

Suppose that the alpha coefficient for observation j is α_j and the box constraint of observation j is C_j, j = 1,...,n, where n is the training sample size.

Value	Description
true	At each iteration, if αj is near 0 or near Cj, then MATLAB sets αj to 0 or to Cj, respectively.
false	MATLAB does not change the alpha coefficients during optimization.

MATLAB stores the final values of α in the Alpha property of the trained SVM model object.

ClipAlphas can affect SMO and ISDA convergence.

Example: 'ClipAlphas',false

Data Types: logical

'Nu' — ν parameter for one-class learning

0.5 (default) | positive scalar

ν parameter for One-Class Learning, specified as the comma-separated pair consisting of 'Nu' and a positive scalar. Nu must be greater than 0 and at most 1.

Set Nu to control the tradeoff between ensuring that most training examples are in the positive class and minimizing the weights in the score function.

Example: 'Nu',0.25

Data Types: double | single

'NumPrint' — Number of iterations between optimization diagnostic message output

1000 (default) | nonnegative integer

Number of iterations between optimization diagnostic message output, specified as the comma-separated pair consisting of 'NumPrint' and a nonnegative integer.

If you specify 'Verbose',1 and 'NumPrint',numprint, then the software displays all optimization diagnostic messages from SMO and ISDA every numprint iterations in the Command Window.

Example: 'NumPrint',500

Data Types: double | single

'OutlierFraction' — Expected proportion of outliers in training data

0 (default) | numeric scalar in the interval [0,1)

Expected proportion of outliers in the training data, specified as the comma-separated pair consisting of 'OutlierFraction' and a numeric scalar in the interval [0,1).

Suppose that you set 'OutlierFraction',outlierfraction, where outlierfraction is a value greater than 0.

• For two-class learning, the software implements robust learning. In other words, the software attempts to remove 100*outlierfraction% of the observations when the optimization algorithm converges. The removed observations correspond to gradients that are large in magnitude.

• For one-class learning, the software finds an appropriate bias term such that outlierfraction of the observations in the training set have negative scores.

Example: 'OutlierFraction',0.01

Data Types: double | single

'RemoveDuplicates' — Flag to replace duplicate observations with single observations

false (default) | true

Flag to replace duplicate observations with single observations in the training data, specified as the comma-separated pair consisting of 'RemoveDuplicates' and true or false.

If RemoveDuplicates is true, then fitcsvm replaces duplicate observations in the training data with a single observation of the same value. The weight of the single observation is equal to the sum of the weights of the corresponding removed duplicates (see Weights).

Tip

If your data set contains many duplicate observations, then specifying 'RemoveDuplicates',true can decrease convergence time considerably.

Data Types: logical

'Verbose' — Verbosity level

0 (default) | 1 | 2

Verbosity level, specified as the comma-separated pair consisting of 'Verbose' and 0, 1, or 2. The value of Verbose controls the amount of optimization information that the software displays in the Command Window and saves the information as a structure to Mdl.ConvergenceInfo.History.

This table summarizes the available verbosity level options.

Value	Description
0	The software does not display or save convergence information.
1	The software displays diagnostic messages and saves convergence criteria every numprint iterations, where numprint is the value of the name-value pair argument 'NumPrint'.
2	The software displays diagnostic messages and saves convergence criteria at every iteration.

Example: 'Verbose',1

Data Types: double | single

Other Classification Options

'CategoricalPredictors' — Categorical predictors list

vector of positive integers | logical vector | character matrix | string array | cell array of character vectors | 'all'

Categorical predictors list, specified as one of the values in this table.

Value	Description
Vector of positive integers	Each entry in the vector is an index value corresponding to the column of the predictor data that contains a categorical variable. The index values are between 1 and p, where p is the number of predictors used to train the model. If fitcsvm uses a subset of input variables as predictors, then the function indexes the predictors using only the subset. The 'CategoricalPredictors' values do not count the response variable, the observation weight variable, and any other variables that the function does not use.
Logical vector	A true entry means that the corresponding column of predictor data is a categorical variable. The length of the vector is p.
Character matrix	Each row of the matrix is the name of a predictor variable. The names must match the entries in PredictorNames. Pad the names with extra blanks so each row of the character matrix has the same length.
String array or cell array of character vectors	Each element in the array is the name of a predictor variable. The names must match the entries in PredictorNames.
'all'	All predictors are categorical.

By default, if the predictor data is in a table (Tbl), fitcsvm assumes that a variable is categorical if it is a logical vector, categorical vector, character array, string array, or cell array of character vectors. If the predictor data is a matrix (X), fitcsvm assumes that all predictors are continuous. To identify any other predictors as categorical predictors, specify them by using the 'CategoricalPredictors' name-value argument.

For the identified categorical predictors, fitcsvm creates dummy variables using two different schemes, depending on whether a categorical variable is unordered or ordered. For an unordered categorical variable, fitcsvm creates one dummy variable for each level of the categorical variable. For an ordered categorical variable, fitcsvm creates one less dummy variable than the number of categories. For details, see Automatic Creation of Dummy Variables.

Example: 'CategoricalPredictors','all'

Data Types: single | double | logical | char | string | cell

'ClassNames' — Names of classes to use for training

categorical array | character array | string array | logical vector | numeric vector | cell array of character vectors

Names of classes to use for training, specified as a categorical, character, or string array; a logical or numeric vector; or a cell array of character vectors. ClassNames must have the same data type as the response variable in Tbl or Y.

If ClassNames is a character array, then each element must correspond to one row of the array.

Use ClassNames to:

• Specify the order of the classes during training.

• Specify the order of any input or output argument dimension that corresponds to the class order. For example, use ClassNames to specify the order of the dimensions of Cost or the column order of classification scores returned by predict.

• Select a subset of classes for training. For example, suppose that the set of all distinct class names in Y is {'a','b','c'}. To train the model using observations from classes 'a' and 'c' only, specify 'ClassNames',{'a','c'}.

The default value for ClassNames is the set of all distinct class names in the response variable in Tbl or Y.

Example: 'ClassNames',{'b','g'}

Data Types: categorical | char | string | logical | single | double | cell

'Cost' — Misclassification cost

square matrix | structure array

Misclassification cost specified as the comma-separated pair consisting of 'Cost' and a square matrix or structure array.

• If you specify the square matrix Cost and the true class of an observation is i, then Cost(i,j) is the cost of classifying a point into class j. That is, rows correspond to the true classes and columns correspond to predicted classes. To specify the class order for the corresponding rows and columns of Cost, also specify the ClassNames name-value pair argument.

• If you specify the structure S, then it must have two fields:

  • S.ClassNames, which contains the class names as a variable of the same data type as Y

  • S.ClassificationCosts, which contains the cost matrix with rows and columns ordered as in S.ClassNames

For two-class learning, if you specify a cost matrix, then the software updates the prior probabilities by incorporating the penalties described in the cost matrix. Consequently, the cost matrix resets to the default. For more details on the relationships and algorithmic behavior of BoxConstraint, Cost, Prior, Standardize, and Weights, see Algorithms.

The default values are:

• Cost = 0 for one-class learning

• Cost(i,j) = 1 if i ~= j and Cost(i,j) = 0 if i = j for two-class learning

Example: 'Cost',[0,1;2,0]

Data Types: double | single | struct

'PredictorNames' — Predictor variable names

string array of unique names | cell array of unique character vectors

Predictor variable names, specified as a string array of unique names or cell array of unique character vectors. The functionality of PredictorNames depends on the way you supply the training data.

• If you supply X and Y, then you can use PredictorNames to assign names to the predictor variables in X.

  • The order of the names in PredictorNames must correspond to the column order of X. That is, PredictorNames{1} is the name of X(:,1), PredictorNames{2} is the name of X(:,2), and so on. Also, size(X,2) and numel(PredictorNames) must be equal.

  • By default, PredictorNames is {'x1','x2',...}.

• If you supply Tbl, then you can use PredictorNames to choose which predictor variables to use in training. That is, fitcsvm uses only the predictor variables in PredictorNames and the response variable during training.

  • PredictorNames must be a subset of Tbl.Properties.VariableNames and cannot include the name of the response variable.

  • By default, PredictorNames contains the names of all predictor variables.

  • A good practice is to specify the predictors for training using either 'PredictorNames' or formula, but not both.

Example: 'PredictorNames',{'SepalLength','SepalWidth','PetalLength','PetalWidth'}

Data Types: string | cell

'Prior' — Prior probabilities

'empirical' (default) | 'uniform' | numeric vector | structure array

Prior probabilities for each class, specified as the comma-separated pair consisting of 'Prior' and a value in this table.

Value	Description
'empirical'	The class prior probabilities are the class relative frequencies in Y.
'uniform'	All class prior probabilities are equal to 1/K, where K is the number of classes.
numeric vector	Each element in the vector is a class prior probability. Order the elements according to Mdl.ClassNames or specify the order using the ClassNames name-value pair argument. The software normalizes the elements to sum to 1.
structure	A structure S with two fields: S.ClassNames contains the class names as a variable of the same type as Y. S.ClassProbs contains a vector of corresponding prior probabilities. The software normalizes the elements of the vector to sum to 1.

For two-class learning, if you specify a cost matrix, then the software updates the prior probabilities by incorporating the penalties described in the cost matrix. For more details on the relationships and algorithmic behavior of BoxConstraint, Cost, Prior, Standardize, and Weights, see Algorithms.

Example: struct('ClassNames',{{'setosa','versicolor','virginica'}},'ClassProbs',1:3)

Data Types: char | string | double | single | struct

'ResponseName' — Response variable name

'Y' (default) | character vector | string scalar

Response variable name, specified as a character vector or string scalar.

• If you supply Y, then you can use 'ResponseName' to specify a name for the response variable.

• If you supply ResponseVarName or formula, then you cannot use 'ResponseName'.

Example: 'ResponseName','response'

Data Types: char | string

'ScoreTransform' — Score transformation

'none' (default) | 'doublelogit' | 'invlogit' | 'ismax' | 'logit' | function handle | ...

Score transformation, specified as a character vector, string scalar, or function handle.

This table summarizes the available character vectors and string scalars.

Value	Description
'doublelogit'	1/(1 + e–2x)
'invlogit'	log(x / (1 – x))
'ismax'	Sets the score for the class with the largest score to 1, and sets the scores for all other classes to 0
'logit'	1/(1 + e–x)
'none' or 'identity'	x (no transformation)
'sign'	–1 for x < 0 0 for x = 0 1 for x > 0
'symmetric'	2x – 1
'symmetricismax'	Sets the score for the class with the largest score to 1, and sets the scores for all other classes to –1
'symmetriclogit'	2/(1 + e–x) – 1

For a MATLAB function or a function you define, use its function handle for the score transform. The function handle must accept a matrix (the original scores) and return a matrix of the same size (the transformed scores).

Example: 'ScoreTransform','logit'

Data Types: char | string | function_handle

'Weights' — Observation weights

numeric vector | name of variable in Tbl

Observation weights, specified as the comma-separated pair consisting of 'Weights' and a numeric vector of positive values or the name of a variable in Tbl. The software weighs the observations in each row of X or Tbl with the corresponding value in Weights. The size of Weights must equal the number of rows in X or Tbl.

If you specify the input data as a table Tbl, then Weights can be the name of a variable in Tbl that contains a numeric vector. In this case, you must specify Weights as a character vector or string scalar. For example, if the weights vector W is stored as Tbl.W, then specify it as 'W'. Otherwise, the software treats all columns of Tbl, including W, as predictors or the response variable when training the model.

By default, Weights is ones(n,1), where n is the number of observations in X or Tbl.

The software normalizes Weights to sum up to the value of the prior probability in the respective class. For more details on the relationships and algorithmic behavior of BoxConstraint, Cost, Prior, Standardize, and Weights, see Algorithms.

Data Types: double | single | char | string

Note

You cannot use any cross-validation name-value pair argument along with the 'OptimizeHyperparameters' name-value pair argument. You can modify the cross-validation for 'OptimizeHyperparameters' only by using the 'HyperparameterOptimizationOptions' name-value pair argument.

Cross-Validation Options

'CrossVal' — Flag to train cross-validated classifier

'off' (default) | 'on'

Flag to train a cross-validated classifier, specified as the comma-separated pair consisting of 'Crossval' and 'on' or 'off'.

If you specify 'on', then the software trains a cross-validated classifier with 10 folds.

You can override this cross-validation setting using the CVPartition, Holdout, KFold, or Leaveout name-value pair argument. You can use only one cross-validation name-value pair argument at a time to create a cross-validated model.

Alternatively, cross-validate later by passing Mdl to crossval.

Example: 'Crossval','on'

'CVPartition' — Cross-validation partition

[] (default) | cvpartition partition object

Cross-validation partition, specified as a cvpartition partition object created by cvpartition. The partition object specifies the type of cross-validation and the indexing for the training and validation sets.

To create a cross-validated model, you can specify only one of these four name-value arguments: CVPartition, Holdout, KFold, or Leaveout.

Example: Suppose you create a random partition for 5-fold cross-validation on 500 observations by using cvp = cvpartition(500,'KFold',5). Then, you can specify the cross-validated model by using 'CVPartition',cvp.

'Holdout' — Fraction of data for holdout validation

scalar value in the range (0,1)

Fraction of the data used for holdout validation, specified as a scalar value in the range (0,1). If you specify 'Holdout',p, then the software completes these steps:

1. Randomly select and reserve p*100% of the data as validation data, and train the model using the rest of the data.

2. Store the compact, trained model in the Trained property of the cross-validated model.

To create a cross-validated model, you can specify only one of these four name-value arguments: CVPartition, Holdout, KFold, or Leaveout.

Example: 'Holdout',0.1

Data Types: double | single

'KFold' — Number of folds

10 (default) | positive integer value greater than 1

Number of folds to use in a cross-validated model, specified as a positive integer value greater than 1. If you specify 'KFold',k, then the software completes these steps:

1. Randomly partition the data into k sets.

2. For each set, reserve the set as validation data, and train the model using the other k – 1 sets.

3. Store the k compact, trained models in a k-by-1 cell vector in the Trained property of the cross-validated model.

To create a cross-validated model, you can specify only one of these four name-value arguments: CVPartition, Holdout, KFold, or Leaveout.

Example: 'KFold',5

Data Types: single | double

'Leaveout' — Leave-one-out cross-validation flag

'off' (default) | 'on'

Leave-one-out cross-validation flag, specified as 'on' or 'off'. If you specify 'Leaveout','on', then for each of the n observations (where n is the number of observations, excluding missing observations, specified in the NumObservations property of the model), the software completes these steps:

1. Reserve the one observation as validation data, and train the model using the other n – 1 observations.

2. Store the n compact, trained models in an n-by-1 cell vector in the Trained property of the cross-validated model.

To create a cross-validated model, you can specify only one of these four name-value arguments: CVPartition, Holdout, KFold, or Leaveout.

Example: 'Leaveout','on'

Convergence Control Options

'DeltaGradientTolerance' — Tolerance for gradient difference

nonnegative scalar

Tolerance for the gradient difference between upper and lower violators obtained by Sequential Minimal Optimization (SMO) or Iterative Single Data Algorithm (ISDA), specified as the comma-separated pair consisting of 'DeltaGradientTolerance' and a nonnegative scalar.

If DeltaGradientTolerance is 0, then the software does not use the tolerance for the gradient difference to check for optimization convergence.

The default values are:

• 1e-3 if the solver is SMO (for example, you set 'Solver','SMO')

• 0 if the solver is ISDA (for example, you set 'Solver','ISDA')

Example: 'DeltaGradientTolerance',1e-2

Data Types: double | single

'GapTolerance' — Feasibility gap tolerance

0 (default) | nonnegative scalar

Feasibility gap tolerance obtained by SMO or ISDA, specified as the comma-separated pair consisting of 'GapTolerance' and a nonnegative scalar.

If GapTolerance is 0, then the software does not use the feasibility gap tolerance to check for optimization convergence.

Example: 'GapTolerance',1e-2

Data Types: double | single

'IterationLimit' — Maximal number of numerical optimization iterations

1e6 (default) | positive integer

Maximal number of numerical optimization iterations, specified as the comma-separated pair consisting of 'IterationLimit' and a positive integer.

The software returns a trained model regardless of whether the optimization routine successfully converges. Mdl.ConvergenceInfo contains convergence information.

Example: 'IterationLimit',1e8

Data Types: double | single

'KKTTolerance' — Karush-Kuhn-Tucker complementarity conditions violation tolerance

nonnegative scalar

Karush-Kuhn-Tucker (KKT) complementarity conditions violation tolerance, specified as the comma-separated pair consisting of 'KKTTolerance' and a nonnegative scalar.

If KKTTolerance is 0, then the software does not use the KKT complementarity conditions violation tolerance to check for optimization convergence.

The default values are:

• 0 if the solver is SMO (for example, you set 'Solver','SMO')

• 1e-3 if the solver is ISDA (for example, you set 'Solver','ISDA')

Example: 'KKTTolerance',1e-2

Data Types: double | single

'ShrinkagePeriod' — Number of iterations between reductions of active set

0 (default) | nonnegative integer

Number of iterations between reductions of the active set, specified as the comma-separated pair consisting of 'ShrinkagePeriod' and a nonnegative integer.

If you set 'ShrinkagePeriod',0, then the software does not shrink the active set.

Example: 'ShrinkagePeriod',1000

Data Types: double | single

Hyperparameter Optimization Options

'OptimizeHyperparameters' — Parameters to optimize

'none' (default) | 'auto' | 'all' | string array or cell array of eligible parameter names | vector of optimizableVariable objects

Parameters to optimize, specified as the comma-separated pair consisting of 'OptimizeHyperparameters' and one of these values:

• 'none' — Do not optimize.

• 'auto' — Use {'BoxConstraint','KernelScale'}.

• 'all' — Optimize all eligible parameters.

• String array or cell array of eligible parameter names.

• Vector of optimizableVariable objects, typically the output of hyperparameters.

The optimization attempts to minimize the cross-validation loss (error) for fitcsvm by varying the parameters. For information about cross-validation loss see Classification Loss. To control the cross-validation type and other aspects of the optimization, use the HyperparameterOptimizationOptions name-value pair argument.

Note

'OptimizeHyperparameters' values override any values you set using other name-value pair arguments. For example, setting 'OptimizeHyperparameters' to 'auto' causes the 'auto' values to apply.

The eligible parameters for fitcsvm are:

• BoxConstraint — fitcsvm searches among positive values, by default log-scaled in the range [1e-3,1e3].

• KernelScale — fitcsvm searches among positive values, by default log-scaled in the range [1e-3,1e3].

• KernelFunction — fitcsvm searches among 'gaussian', 'linear', and 'polynomial'.

• PolynomialOrder — fitcsvm searches among integers in the range [2,4].

• Standardize — fitcsvm searches among 'true' and 'false'.

Set nondefault parameters by passing a vector of optimizableVariable objects that have nondefault values. For example:

load fisheriris
params = hyperparameters('fitcsvm',meas,species);
params(1).Range = [1e-4,1e6];

Pass params as the value of OptimizeHyperparameters.

By default, iterative display appears at the command line, and plots appear according to the number of hyperparameters in the optimization. For the optimization and plots, the objective function is log(1 + cross-validation loss) for regression and the misclassification rate for classification. To control the iterative display, set the Verbose field of the 'HyperparameterOptimizationOptions' name-value pair argument. To control the plots, set the ShowPlots field of the 'HyperparameterOptimizationOptions' name-value pair argument.

For an example, see Optimize SVM Classifier.

Example: 'auto'

'HyperparameterOptimizationOptions' — Options for optimization

structure

Options for optimization, specified as the comma-separated pair consisting of 'HyperparameterOptimizationOptions' and a structure. This argument modifies the effect of the OptimizeHyperparameters name-value pair argument. All fields in the structure are optional.

Field Name	Values	Default
Optimizer	'bayesopt' — Use Bayesian optimization. Internally, this setting calls bayesopt. 'gridsearch' — Use grid search with NumGridDivisions values per dimension. 'randomsearch' — Search at random among MaxObjectiveEvaluations points. 'gridsearch' searches in a random order, using uniform sampling without replacement from the grid. After optimization, you can get a table in grid order by using the command sortrows(Mdl.HyperparameterOptimizationResults).	'bayesopt'
AcquisitionFunctionName	'expected-improvement-per-second-plus' 'expected-improvement' 'expected-improvement-plus' 'expected-improvement-per-second' 'lower-confidence-bound' 'probability-of-improvement' Acquisition functions whose names include per-second do not yield reproducible results because the optimization depends on the runtime of the objective function. Acquisition functions whose names include plus modify their behavior when they are overexploiting an area. For more details, see Acquisition Function Types.	'expected-improvement-per-second-plus'
MaxObjectiveEvaluations	Maximum number of objective function evaluations.	30 for 'bayesopt' or 'randomsearch', and the entire grid for 'gridsearch'
MaxTime	Time limit, specified as a positive real. The time limit is in seconds, as measured by tic and toc. Run time can exceed MaxTime because MaxTime does not interrupt function evaluations.	Inf
NumGridDivisions	For 'gridsearch', the number of values in each dimension. The value can be a vector of positive integers giving the number of values for each dimension, or a scalar that applies to all dimensions. This field is ignored for categorical variables.	10
ShowPlots	Logical value indicating whether to show plots. If true, this field plots the best objective function value against the iteration number. If there are one or two optimization parameters, and if Optimizer is 'bayesopt', then ShowPlots also plots a model of the objective function against the parameters.	true
SaveIntermediateResults	Logical value indicating whether to save results when Optimizer is 'bayesopt'. If true, this field overwrites a workspace variable named 'BayesoptResults' at each iteration. The variable is a BayesianOptimization object.	false
Verbose	Display to the command line. 0 — No iterative display 1 — Iterative display 2 — Iterative display with extra information For details, see the bayesopt Verbose name-value pair argument.	1
UseParallel	Logical value indicating whether to run Bayesian optimization in parallel, which requires Parallel Computing Toolbox™. Due to the nonreproducibility of parallel timing, parallel Bayesian optimization does not necessarily yield reproducible results. For details, see Parallel Bayesian Optimization.	false
Repartition	Logical value indicating whether to repartition the cross-validation at every iteration. If false, the optimizer uses a single partition for the optimization. true usually gives the most robust results because this setting takes partitioning noise into account. However, for good results, true requires at least twice as many function evaluations.	false
Use no more than one of the following three field names.
CVPartition	A cvpartition object, as created by cvpartition.	'Kfold',5 if you do not specify any cross-validation field
Holdout	A scalar in the range (0,1) representing the holdout fraction.
Kfold	An integer greater than 1.

Example: 'HyperparameterOptimizationOptions',struct('MaxObjectiveEvaluations',60)

Data Types: struct

Output Arguments

Mdl — Trained SVM classification model

ClassificationSVM model object | ClassificationPartitionedModel cross-validated model object

Trained SVM classification model, returned as a ClassificationSVM model object or ClassificationPartitionedModel cross-validated model object.

If you set any of the name-value pair arguments KFold, Holdout, Leaveout, CrossVal, or CVPartition, then Mdl is a ClassificationPartitionedModel cross-validated classifier. Otherwise, Mdl is a ClassificationSVM classifier.

To reference properties of Mdl, use dot notation. For example, enter Mdl.Alpha in the Command Window to display the trained Lagrange multipliers.

Limitations

• fitcsvm trains SVM classifiers for one-class or two-class learning applications. To train SVM classifiers using data with more than two classes, use fitcecoc.

• fitcsvm supports low-dimensional and moderate-dimensional data sets. For high-dimensional data sets, use fitclinear instead.

More About

Box Constraint

A box constraint is a parameter that controls the maximum penalty imposed on margin-violating observations, which helps to prevent overfitting (regularization).

If you increase the box constraint, then the SVM classifier assigns fewer support vectors. However, increasing the box constraint can lead to longer training times.

Gram Matrix

The Gram matrix of a set of n vectors {x₁,..,x_n; x_j ∊ R^p} is an n-by-n matrix with element (j,k) defined as G(x_j,x_k) = <ϕ(x_j),ϕ(x_k)>, an inner product of the transformed predictors using the kernel function ϕ.

For nonlinear SVM, the algorithm forms a Gram matrix using the rows of the predictor data X. The dual formalization replaces the inner product of the observations in X with corresponding elements of the resulting Gram matrix (called the “kernel trick”). Consequently, nonlinear SVM operates in the transformed predictor space to find a separating hyperplane.

Karush-Kuhn-Tucker (KKT) Complementarity Conditions

KKT complementarity conditions are optimization constraints required for optimal nonlinear programming solutions.

In SVM, the KKT complementarity conditions are

$$\{\begin{array}{l}{\alpha }_j\left[y_{j}f\left(x_j\right)-1+{\xi }_j\right]=0\\ {\xi }_j\left(C-{\alpha }_j\right)=0\end{array}$$

for all j = 1,...,n, where $$f\left(x_j\right)=\varphi \left(x_j\right)\prime \beta +b,$$ ϕ is a kernel function (see Gram matrix), and ξ_j is a slack variable. If the classes are perfectly separable, then ξ_j = 0 for all j = 1,...,n.

One-Class Learning

One-class learning, or unsupervised SVM, aims to separate data from the origin in the high-dimensional predictor space (not the original predictor space), and is an algorithm used for outlier detection.

The algorithm resembles that of SVM for binary classification. The objective is to minimize the dual expression

$$0.5{\displaystyle \sum _{jk}{\alpha }_j}{\alpha }_{k}G(x_j,x_k)$$

with respect to $${\alpha }_1,\mathrm{...},{\alpha }_n$$, subject to

$${\displaystyle \sum {\alpha }_j}=n\nu$$

and $$0\le {\alpha }_j\le 1$$ for all j = 1,...,n. The value of G(x_j,x_k) is in element (j,k) of the Gram matrix.

A small value of ν leads to fewer support vectors and, therefore, a smooth, crude decision boundary. A large value of ν leads to more support vectors and, therefore, a curvy, flexible decision boundary. The optimal value of ν should be large enough to capture the data complexity and small enough to avoid overtraining. Also, 0 < ν ≤ 1.

For more details, see [5].

Support Vector

Support vectors are observations corresponding to strictly positive estimates of α₁,...,α_n.

SVM classifiers that yield fewer support vectors for a given training set are preferred.

Support Vector Machines for Binary Classification

The SVM binary classification algorithm searches for an optimal hyperplane that separates the data into two classes. For separable classes, the optimal hyperplane maximizes a margin (space that does not contain any observations) surrounding itself, which creates boundaries for the positive and negative classes. For inseparable classes, the objective is the same, but the algorithm imposes a penalty on the length of the margin for every observation that is on the wrong side of its class boundary.

The linear SVM score function is

$$f(x)=x\prime \beta +b,$$

where:

• x is an observation (corresponding to a row of X).

• The vector β contains the coefficients that define an orthogonal vector to the hyperplane (corresponding to Mdl.Beta). For separable data, the optimal margin length is $$2/\Vert \beta \Vert .$$

• b is the bias term (corresponding to Mdl.Bias).

The root of f(x) for particular coefficients defines a hyperplane. For a particular hyperplane, f(z) is the distance from point z to the hyperplane.

The algorithm searches for the maximum margin length, while keeping observations in the positive (y = 1) and negative (y = –1) classes separate.

• For separable classes, the objective is to minimize $$\Vert \beta \Vert$$ with respect to the β and b subject to y_jf(x_j) ≥ 1, for all j = 1,..,n. This is the primal formalization for separable classes.

• For inseparable classes, the algorithm uses slack variables (ξ_j) to penalize the objective function for observations that cross the margin boundary for their class. ξ_j = 0 for observations that do not cross the margin boundary for their class, otherwise ξ_j ≥ 0.

  The objective is to minimize $$0.5{\Vert \beta \Vert }^2+C{\displaystyle \sum {\xi }_j}$$ with respect to the β, b, and ξ_j subject to $$y_{j}f(x_j)\ge 1-{\xi }_j$$ and $${\xi }_j\ge 0$$ for all j = 1,..,n, and for a positive scalar box constraint C. This is the primal formalization for inseparable classes.

The algorithm uses the Lagrange multipliers method to optimize the objective, which introduces n coefficients α₁,...,α_n (corresponding to Mdl.Alpha). The dual formalizations for linear SVM are as follows:

• For separable classes, minimize

  $$0.5{\displaystyle \sum _{j=1}^n{\displaystyle \sum }_{k=1}^n}{\alpha }_j{\alpha }_k{y}_j{y}_k{x}_j\prime x_k-{\displaystyle \sum }_{j=1}^n{\alpha }_j$$

  with respect to α₁,...,α_n, subject to $${\displaystyle \sum {\alpha }_j}{y}_j=0$$, α_j ≥ 0 for all j = 1,...,n, and Karush-Kuhn-Tucker (KKT) complementarity conditions.

• For inseparable classes, the objective is the same as for separable classes, except for the additional condition $$0\le {\alpha }_j\le C$$ for all j = 1,..,n.

The resulting score function is

$$\widehat{f}(x)={\displaystyle \sum _{j=1}^n{\widehat{\alpha }}_j}{y}_{j}x\prime x_j+\widehat{b}.$$

$$\widehat{b}$$ is the estimate of the bias and $${\widehat{\alpha }}_j$$ is the jth estimate of the vector $$\widehat{\alpha }$$, j = 1,...,n. Written this way, the score function is free of the estimate of β as a result of the primal formalization.

The SVM algorithm classifies a new observation z using $$\text{sign}\left(\widehat{f}\left(z\right)\right).$$

In some cases, a nonlinear boundary separates the classes. Nonlinear SVM works in a transformed predictor space to find an optimal, separating hyperplane.

The dual formalization for nonlinear SVM is

$$0.5{\displaystyle \sum _{j=1}^n{\displaystyle \sum _{k=1}^n{\alpha }_j}}{\alpha }_k{y}_j{y}_{k}G(x_j,x_k)-{\displaystyle \sum _{j=1}^n{\alpha }_j}$$

with respect to α₁,...,α_n, subject to $${\displaystyle \sum {\alpha }_j}{y}_j=0$$, $$0\le {\alpha }_j\le C$$ for all j = 1,..,n, and the KKT complementarity conditions. G(x_k,x_j) are elements of the Gram matrix. The resulting score function is

$$\widehat{f}(x)={\displaystyle \sum _{j=1}^n{\widehat{\alpha }}_j}{y}_{j}G(x,x_j)+\widehat{b}.$$

For more details, see Understanding Support Vector Machines, [1], and [3].

Tips

• Unless your data set is large, always try to standardize the predictors (see Standardize). Standardization makes predictors insensitive to the scales on which they are measured.

• It is a good practice to cross-validate using the KFold name-value pair argument. The cross-validation results determine how well the SVM classifier generalizes.

• For one-class learning:

  • The default setting for the name-value pair argument Alpha can lead to long training times. To speed up training, set Alpha to a vector mostly composed of 0s.

  • Set the name-value pair argument Nu to a value closer to 0 to yield fewer support vectors and, therefore, a smoother but crude decision boundary.

• Sparsity in support vectors is a desirable property of an SVM classifier. To decrease the number of support vectors, set BoxConstraint to a large value. This action increases the training time.

• For optimal training time, set CacheSize as high as the memory limit your computer allows.

• If you expect many fewer support vectors than observations in the training set, then you can significantly speed up convergence by shrinking the active set using the name-value pair argument 'ShrinkagePeriod'. It is a good practice to specify 'ShrinkagePeriod',1000.

• Duplicate observations that are far from the decision boundary do not affect convergence. However, just a few duplicate observations that occur near the decision boundary can slow down convergence considerably. To speed up convergence, specify 'RemoveDuplicates',true if:

  • Your data set contains many duplicate observations.

  • You suspect that a few duplicate observations fall near the decision boundary.

  To maintain the original data set during training, fitcsvm must temporarily store separate data sets: the original and one without the duplicate observations. Therefore, if you specify true for data sets containing few duplicates, then fitcsvm consumes close to double the memory of the original data.

• After training a model, you can generate C/C++ code that predicts labels for new data. Generating C/C++ code requires MATLAB Coder™. For details, see Introduction to Code Generation.

Algorithms

• For the mathematical formulation of the SVM binary classification algorithm, see Support Vector Machines for Binary Classification and Understanding Support Vector Machines.

• NaN, <undefined>, empty character vector (''), empty string (""), and <missing> values indicate missing values. fitcsvm removes entire rows of data corresponding to a missing response. When computing total weights (see the next bullets), fitcsvm ignores any weight corresponding to an observation with at least one missing predictor. This action can lead to unbalanced prior probabilities in balanced-class problems. Consequently, observation box constraints might not equal BoxConstraint.

• fitcsvm removes observations that have zero weight or prior probability.

• For two-class learning, if you specify the cost matrix $$\mathcal{C}$$ (see Cost), then the software updates the class prior probabilities p (see Prior) to p_c by incorporating the penalties described in $$\mathcal{C}$$.

  Specifically, fitcsvm completes these steps:

  1. Compute $$p_c^{\ast }=p\prime \mathcal{C}.$$

  2. Normalize p_c^* so that the updated prior probabilities sum to 1.

     $$p_c=\frac{1}{{\displaystyle \sum _{j=1}^K{p}_{c,j}^{\ast }}}{p}_c^{\ast }.$$

     K is the number of classes.

  3. Reset the cost matrix to the default

     $$\mathcal{C}=\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right].$$

  4. Remove observations from the training data corresponding to classes with zero prior probability.

• For two-class learning, fitcsvm normalizes all observation weights (see Weights) to sum to 1. The function then renormalizes the normalized weights to sum up to the updated prior probability of the class to which the observation belongs. That is, the total weight for observation j in class k is

  $$w_j^{\ast }=\frac{w_j}{{\displaystyle \sum _{\forall j\in \text{Class }k}{w}_j}}{p}_{c,k}.$$

  w_j is the normalized weight for observation j; p_c,k is the updated prior probability of class k (see previous bullet).

• For two-class learning, fitcsvm assigns a box constraint to each observation in the training data. The formula for the box constraint of observation j is

  $$C_j=n{C}_0{w}_j^{\ast }.$$

  n is the training sample size, C₀ is the initial box constraint (see the 'BoxConstraint' name-value pair argument), and $$w_j^{\ast }$$ is the total weight of observation j (see previous bullet).

• If you set 'Standardize',true and the 'Cost', 'Prior', or 'Weights' name-value pair argument, then fitcsvm standardizes the predictors using their corresponding weighted means and weighted standard deviations. That is, fitcsvm standardizes predictor j (x_j) using

  $$x_j^{\ast }=\frac{x_j-{\mu }_j^{\ast }}{{\sigma }_j^{\ast }}.$$

  $${\mu }_j^{\ast }=\frac{1}{{\displaystyle \sum _k{w}_k^{\ast }}}{\displaystyle \sum _k{w}_k^{\ast }{x}_{jk}}.$$

  x_jk is observation k (row) of predictor j (column).

  $${\left({\sigma }_j^{\ast }\right)}^2=\frac{v_1}{v_1^2-v_2}{\displaystyle \sum _k{w}_k^{\ast }{\left(x_{jk}-{\mu }_j^{\ast }\right)}^2}.$$

  $$v_1={\displaystyle \sum _j{w}_j^{\ast }}.$$

  $$v_2={\displaystyle \sum _j{\left(w_j^{\ast }\right)}^2}.$$

• Assume that p is the proportion of outliers that you expect in the training data, and that you set 'OutlierFraction',p.

  • For one-class learning, the software trains the bias term such that 100p% of the observations in the training data have negative scores.

  • The software implements robust learning for two-class learning. In other words, the software attempts to remove 100p% of the observations when the optimization algorithm converges. The removed observations correspond to gradients that are large in magnitude.

• If your predictor data contains categorical variables, then the software generally uses full dummy encoding for these variables. The software creates one dummy variable for each level of each categorical variable.

  • The PredictorNames property stores one element for each of the original predictor variable names. For example, assume that there are three predictors, one of which is a categorical variable with three levels. Then PredictorNames is a 1-by-3 cell array of character vectors containing the original names of the predictor variables.

  • The ExpandedPredictorNames property stores one element for each of the predictor variables, including the dummy variables. For example, assume that there are three predictors, one of which is a categorical variable with three levels. Then ExpandedPredictorNames is a 1-by-5 cell array of character vectors containing the names of the predictor variables and the new dummy variables.

  • Similarly, the Beta property stores one beta coefficient for each predictor, including the dummy variables.

  • The SupportVectors property stores the predictor values for the support vectors, including the dummy variables. For example, assume that there are m support vectors and three predictors, one of which is a categorical variable with three levels. Then SupportVectors is an n-by-5 matrix.

  • The X property stores the training data as originally input and does not include the dummy variables. When the input is a table, X contains only the columns used as predictors.

• For predictors specified in a table, if any of the variables contain ordered (ordinal) categories, the software uses ordinal encoding for these variables.

  • For a variable with k ordered levels, the software creates k – 1 dummy variables. The jth dummy variable is –1 for levels up to j, and +1 for levels j + 1 through k.

  • The names of the dummy variables stored in the ExpandedPredictorNames property indicate the first level with the value +1. The software stores k – 1 additional predictor names for the dummy variables, including the names of levels 2, 3, ..., k.

• All solvers implement L1 soft-margin minimization.

• For one-class learning, the software estimates the Lagrange multipliers, α₁,...,α_n, such that

  $${\displaystyle \sum _{j=1}^n{\alpha }_j}=n\nu .$$

References

[1] Christianini, N., and J. C. Shawe-Taylor. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge, UK: Cambridge University Press, 2000.

[2] Fan, R.-E., P.-H. Chen, and C.-J. Lin. “Working set selection using second order information for training support vector machines.” Journal of Machine Learning Research, Vol. 6, 2005, pp. 1889–1918.

[3] Hastie, T., R. Tibshirani, and J. Friedman. The Elements of Statistical Learning, Second Edition. NY: Springer, 2008.

[4] Kecman V., T. -M. Huang, and M. Vogt. “Iterative Single Data Algorithm for Training Kernel Machines from Huge Data Sets: Theory and Performance.” Support Vector Machines: Theory and Applications. Edited by Lipo Wang, 255–274. Berlin: Springer-Verlag, 2005.

[5] Scholkopf, B., J. C. Platt, J. C. Shawe-Taylor, A. J. Smola, and R. C. Williamson. “Estimating the Support of a High-Dimensional Distribution.” Neural Comput., Vol. 13, Number 7, 2001, pp. 1443–1471.

[6] Scholkopf, B., and A. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond, Adaptive Computation and Machine Learning. Cambridge, MA: The MIT Press, 2002.

Extended Capabilities

Automatic Parallel Support

Accelerate code by automatically running computation in parallel using Parallel Computing Toolbox™.

To perform parallel hyperparameter optimization, use the 'HyperparameterOptimizationOptions', struct('UseParallel',true) name-value argument in the call to this function.

For more information on parallel hyperparameter optimization, see Parallel Bayesian Optimization.

For general information about parallel computing, see Run MATLAB Functions with Automatic Parallel Support (Parallel Computing Toolbox).

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.

Usage notes and limitations:

• The 'KernelFunction' option must not specify a custom kernel.

• The 'Solver' option can be set to only 'SMO'.

• The 'OutlierFraction' option is not supported.

• The 'Alpha' option must specify a feasible starting point.

• The 'OptimizeHyperparameters' option is not supported.

• One-class classification is not supported. The labels must contain two different classes.

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).