Ego-motion compensated Optical flow field

I have a sequence of images recorded from a moving camera. I wish to calculate the ego motion compensated optical flow field between successive images. This is what I have done so far. (I'm working with mex-opencv functions).

I calculate the image transformation ([R|t] a 3x4 matrix using the visual odometry algorithms from the libviso2 toolbox) between successive images. I also calculate the dense optical flow using the TV-L1 algorithm.

I'm not sure how to use the transformation matrix and the calculated optical flow field to obtain a ego motion compensated flow field. The initial idea was to project the translation part of the transformation matrix to image coordinates and then subtract that value from the flow field but that doesn't work even though it kind of makes sense in my head. Could anyone point me in the right direction or give me some sort of clue.

Thanks in anticipation.