Never mind. I see now that the memory consumption is on ly 3.3KB, not MB. And I see from the test below that the data size does not affect the size of the fit object, so I don't think the fit object could be carrying the data around.
N=1e4;
[x,y]=deal(rand(N,1));
fobj1=fit(x,y,'poly1');
N=1e7;
[x,y]=deal(rand(N,1));
fobj2=fit(x,y,'poly1');
whos fobj1 fobj2
Name Size Bytes Class Attributes
fobj1 1x1 740 cfit
fobj2 1x1 740 cfit