Your first problem is you are not using it correctly.
The last argument should be one of the choices, to give it a default button selection.
answer = questdlg('choose a selection method?', ...
'selection methods', ...
(You spelled tournament incorrectly, not that it matters.)
But the above will give you a dialog box with three options, one of which (the first) is highlighted.
How it caught you in an infinite loop is difficult to know, since we lack your other code. Perhaps something in your code calls itself recursively, although if it did, then you would get a stack limit exceeded error, not a true infinite loop, because it will crash long before things go near infinite.
Another error in the code is you cannot name a function roulette-wheel. You CAN name it roulette_wheel, which is what I put in the call to questdlg. A minus sign cannot be part of a function name, but an underscore can be.