Never mind. I see now that the memory consumption is on ly 3.3KB, not MB. And I see from the test below that the data size does not affect the size of the fit object, so I don't think the fit object could be carrying the data around.
N=1e4;
[x,y]=deal(rand(N,1));
fobj1=fit(x,y,'poly1');
N=1e7;
[x,y]=deal(rand(N,1));
fobj2=fit(x,y,'poly1');
whos fobj1 fobj2
  Name       Size            Bytes  Class    Attributes
  fobj1      1x1               740  cfit               
  fobj2      1x1               740  cfit