I am not exactly certain why it is so, but I can share why I think that 'nsamp' must be an integer.
'nsamp' denotes the number of samples per symbol in y. As MSK is a digital modulation scheme, we deal with data in bits. Since we have to deal with bits in whole (because there is no concept of a half-bit), each symbol is expected to have a certain number of bits.
If the value of 'nsamp' is not an integer, it will contradict this. This is my understanding, it may be inaccurate.