From what I see, the issue is that you are using a table despite wanting to use the matrix syntax. Grpstats will compute the indicated stats for a single variable at a time, even if you specify multiple variables. You don't have access to multiple variables at the same time. To do that, the values must be part of the same table variable.
Since you didn't share your data, the best I can do is show you how you could do this using some made-up data. Basically, you need to combine your low and high CI values into an array and add that as a single variable to your table. This way, the matrix is passed to your input function, and you can extract the low and high values inside the function.
observer = [1 2 3 4 5]';
ci_low = [0 1 2 3 4]';
ci_high = [10 11 12 13 14]';
ds_o = table(observer,ci_low,ci_high);
ds_o.ci = [ci_low ci_high];
ds_o = grpstats( ds_o,'observer',@ci_estimate,'DataVars','ci')
function [ci_new] = ci_estimate(ci_array)
ci_int = sqrt(sum(diff(ci_array,,2)).^2);
ci_mean = mean(mean(ci_array, 2));
ci_new_low = ci_mean - ci_int;
ci_new_high = ci_mean + ci_int;
ci_new = [ci_new_low, ci_new_high];