sfit and large datasets - speed issue

I have a sfit object that resulted from fitting surface data from about 16 points. Now I need to populate an image distortion map for an image that is about 6,000 x 12,000 pixels = 72,000,000 points. I just tried with a truncated dataset of 900*900 = 810,000 pixels and killed it when it took over an hour. 72 million isn't going to happen at that speed.

Does anybody have any ideas of how to create a distortion map of 72 million data points with my sfit object?