Peculiar. If you look at g(t) you will see (well in my version of matlab):
g(t) = (mod(t,5))^2
Which shows that the mod is instantly evaluated. Further it didn't work to define five as a symbolic 5 either, the mod was determined. Peculiarly it did not work to include it explicitly in the vpaintegral either:
vpaintegral(mod(t,5)^2, t, 0, 10)
This was an interesting "feature".