You can choose to define that operator in ANY way you like.
However, if you had bothered to read the help for nchoosek, you would have found this:
nchoosek Binomial coefficient or all combinations.
nchoosek(N,K) where N and K are non-negative integers returns N!/K!(N-K)!.
So to call your answer the correct answer is not at all correct. In fact, since nchoosek is defined as a classic binomial coefficient, your answer is not even "correct" in any sense, since factorials are not defined for non-integer input. But suppose we chose to use the gamma function to extend those factorials to allow non-integer input?
Classically, the binomial coefficient is defined as
nchoosek(N,K) = factorial(N)/(factorial(K)*factorial(N-K))
Lets rewrite this using the relation between factorial and the gamma function.
factorial(m) = gamma(m+1)
So now technically, your case would have
Completely irrelevant as I said, since binomial coefficients are generally undefined for non-integer input. I can see what you did for the non-integer case. The reason why binomial coefficients are often written in the form they are is because some of the terms in the factorials cancel out in that fraction. That would not work in any case for non-integers.
I'll add that the documentation does indicate that N may be any general real, non-negative values. But that only applies when N is a vector. In the case where N is a vector, nchoosek actually generates all possible combinations of those N elements, taken K at a time.
nchoosek([1.1 2.2 3.3],2)
Admittedly, the documentation should be clearer here, to distinguish the case where N is a scalar.
I hope the documentation folks are listening in to this one.