d(:,1) is [1;5] . d1(:,1) is [1;1] . You ask ismember() for matching rows. The row [1] of d(:,1) matches the row [1] of d1(:,1) so the first value returned by ismember() is true. The row [5] of d(:,1) does not match any rows of dl(:,1) so the second value returned by ismember is false. The ismember result is thus [true;false] . You use that [true;false] as the first index of d, and you select all columns of d through the second index, so the result is going to be to select just the first row, d(:,1) which is [1 2 3]. You try to combine that single row via horzcat, with the multiple rows of d1, so you get an error.
The above is why you get an error. In order for us to show you how to get the output you want, you will need to explain the rules you want.
I note that if you were to use
[tf, ua, ub] = ismember(d1(:,1), d(:,1), 'rows');
out = [d1(tf), d(ub,:)];
then that would happen to give the output you say you need. This is probably just a coincidence, however.