Is there a way to create a custom cvpartition?

14 views (last 30 days)
Hi,
I'd like you use crossvalidation in a model with a custom set of training/test sample vectors.
To use some of the built-in functionalities of Matlab I'd like to pass the input via the CVPartition Name/Value input parameter. Is there a way to define a custom set of train/test indices using cvpartition?
Thanks,
Christian

Accepted Answer

Shraddha Jain
Shraddha Jain on 25 Jun 2021
Hi Christian,
As of now, the functionality to use custom indices in the cvpartition object is not available in the Statistics and Machine Learning toolbox. It might be considered in a future release.
  3 Comments
Giovanni Attolico
Giovanni Attolico on 13 Oct 2021
It seems to me that the indices are stored inside the cvpartition variable but they are read-only using the functions "training" and "test". While waiting for the "future release", it is not possible to define two functions "settraining" and "settest" that write indices in the variable? That would be really useful in many situation to allow the use of the resulting variable in all the situations where the current random partition can be used ...
Thanks
Christian
Christian on 13 Oct 2021
I had the same thoughts on this. However, I stopped trying to alter the code in cvpartition after a few attempts and ended up in writing a complete custom approach for crossvalidation procedures. In addition to custom indices you're also more flexible with the used metrics and and and...

Sign in to comment.

More Answers (1)

Giovanni
Giovanni on 9 Jan 2024
Custom partition has been recently introduced in MATLAB r2023B, but I suspect it does not work as expected since cross validation results from 2 neural netwroks differ even if trained and validated with the same folds (cv partition using the custom partition options with indices generated by crossvalind) and the networks are trained with the same parameters.
clear variables
close all
clc
tab = readtable("three_selected_feature.xlsx"); %features dataset
featuresTab = tab(:,2:end-1);
feature = table2array(tab(:,2:end-1));
y = table2array(tab(:,end));
for i = 1:6 %repeat for different training test splitting.
cTest = cvpartition(y,'HoldOut',0.2,'Stratify',true);
train = training(cTest);
y_train = y(train);
testing = test(cTest);
cvIndices = crossvalind('Kfold',size(y_train,1),5);
cv = cvpartition("CustomPartition",cvIndices);
% cv = cvpartition(y(train),"Resubstitution");
features_scaled = normalize(feature,'zscore');
modelNET1 = fitcnet(features_scaled(train,:),y_train,'PredictorNames',featuresTab.Properties.VariableNames,'CVPartition',cv);
modelNET2 = fitcnet(features_scaled(train,:),y_train,'PredictorNames',featuresTab.Properties.VariableNames,'CVPartition',cv);
%models trained with the same parameters and features and crossvalidated
%using custom partitions. cv is based on cvIndices generated by the
%crossvalind function. For the same "i" (for loop) results of the two
%models should be the same, but they differ.
% cvmodelLDA1 = crossval(modelLDA1,'CVPartition',cv);
lossNET1(i) = kfoldLoss(modelNET1);
accNET1(i) = 1 - lossNET1(i);
% cvmodelLDA2= crossval(modelLDA2,'CVPartition',cv);
lossNET2(i) = kfoldLoss(modelNET2);
accNET2(i) = 1 - lossNET2(i);
end

Products


Release

R2017b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!