Shift bin index in fft formula

For a uni project, I implemented an algorithm from a signal processing paper. Within this project, I use a small function which is similar to matlab fft with a shift in the what would be the frequency index. Here is a minimal example of what I do (I would like to do it faster): (And is repeated in a loop other different integer values for shift)

N=512;
x=randn(N,1);
X=zeros(N,1);
shift=5;

for indBin=1:N
    for j=1:N
        X(indBin)=X(indBin)+x(j)*exp(-1*1i*2*pi/N*(j-1)*(shift-indBin));
    end
end

It does what it is supposed to do but I'm pretty sure there is a better, faster way. I thought I could do it using the czt function but I cannot put my finger on how. It may save me quite a lot of time if I could replace those two loops. Thanks for your help.