Matlab crashes when the external function in DLL is called (x64 code)

Hello,

I have a code in assembly language that includes few functions. The code was translated into .DLL (x64). Now I want to call the function in x64 Matlab, but when I do so, Matlab crashes.

I use fastcall (as it's the only calling convention for x64), header file is adjusted to fastcall also.

No call function or load library errors, it just crashes.

Any help appreciated. Thanks!

Answer 1

[Samuel SLenker]

So I found a mistake in my assembly code. Now when I use the new version, Matlab crashes again but with crash dialog where it says on which address of the DLL is the problem. It always crashes at the first instruction that uses two 128 bit registers (xmm). I tried to remove the line, and it again crashed at the next instruction. And then again.

Have no idea where's the problem because in .exe file + dbg the code works fine.

[Philip Borghesani]

Have you tried debugging your dll when called by matlab? What is the actual violation reported? Are registers and values as you expect? For power debugging on Windows I recommend using windbg which is free with the debugging tools for Windows.

Please try to respond with comments, not answers so conversations can have some consistency.

[Samuel SLenker]

I use x64dbg which is free and works fine.

I tried to debug whole Matlab, I mean, I run DBG, then run Matlab through DBG and I had a problem within my code, so the whole process stopped at one instruction. I found the issue, corrected the assembly code and now it works fine.

So, the function in .dll is now correct, everything runs smoothly, but I have a problem that Matlab crashes at some point when there is a return into "libmwcli.dll". That means the whole computation is done and after that Matlab crashes. I tried to find some answers according to mentioned .dll problems but I'm unable to solve the issue.

[Philip Borghesani]

According to this: Microsoft Register Usage info XMM registers above 6 must be preserved by the callee. I don't see that happening in your code.

[Samuel SLenker]

Yeah, I knew about that need to save registers, but somehow I forgot about it. Well, after saving all the callee-save registers, and of course popping them at the end, Matlab works. Thank you Philip!

Answer 2

[Jan]

Well, it sounds like the code of the DLL is buggy. Do you have any evidence that this assembler code works?

Answer 3

[Samuel SLenker]

Hello Jan,

the code should be OK. Before I even moved to Matlab, I went through several tests and optimizations.

I created .exe that calls the function from the .dll and it works without any problems. Tested through debugger. No errors, program runs as expected, like I said, I was optimizing and fixing errors (it took days due to many differences between x86 and x64) in DBG before I moved to Matlab.

I can attach files here once I reach my PC.

Answer 4

[Philip Borghesani]

Sounds like a job for the debugger. Without specifics we can't be of much help. The 64bit abi is sometimes called fastcall and sometimes not. LOADLIBRARY does not need or desire the function to be decorated with the microsoft extension "fastcall" decoration. I assume you are doing this on Windows?

Matlab makes some assumptions about processor state that can be violated by assembly code, my first thought is why are you using assembly? If you change any processor settings make sure you put them back before returning to MATLAB...

The more of the flowing information you supply the more likely we are to be helpful:

• Matlab version and OS.
• Function source or shortened example source if possible.
• Header file contents.
• Matlab example code used to call function.
• Information on how library was built and example c test code you used to test the library function.
• If you cant supply enough code to allow reproduction at least give us the stack trace so we can reason about why things are crashing.

Answer 5

[Samuel SLenker]

Hi Philip,

find the details below:

• Matlab R2015b x64, Win7 x64
• SOURCE CODE OF THE DLL IS IN ATTACHMENT (notes are non-english, just ignore them)
• Script in Matlab (notes are non-english, just ignore them)

clear all; clc

f1=1000;
f2=1100;
fs=8000;

N=600;
n=0:N-1;
k=0:N-1;

x=single(0.7*cos(2*pi*f1/fs*(0:N-1))+0.3*cos(2*pi*f2/fs*(0:N-1)));

M=single(cos(pi/N*(n'+1/2)*(k+1/2)));  

disp('Computation time in Matlab:')
tic
x*M;
toc
y=x*M;

%number of rounds the function is called
Np=500;

plot(y * sqrt(2/N),'r')  
hold on

% load lib
hfile1 = 'kody64.h';
[notfound,warnings]=loadlibrary('kody.dll', hfile1,'mfilename','kody_mx');
pM = libpointer('singlePtr', M);
px = libpointer('singlePtr', x);

% call vxm_sse
disp('SSE computation:')
td=0;
for  i=1:Np
y0=zeros(1,N); py0 = libpointer('singlePtr', y0);
tic;
calllib('kody', 'vxm_sse', pM, px, py0, length(x)/4);
td=td+toc/Np;
end
disp(num2str(td,6))
y_sse1=get(py0, 'Value');
plot(y_sse1 * sqrt(2/N),'g')

hold off

unloadlibrary kody
legend('DCT IV jako maticové násobení v Matlabu','DCT IV pomocí funkce SSE1','DCT IV pomocí funkce SSE2','DCT IV pomocí funkce SSE3','DCT IV pomocí funkce AVX1','DCT IV pomocí funkce AVX2','DCT IV pomocí funkce AVX3',1)

• Header file is "kody64.h" and looks following:

void vxm_sse(float *a,float *b,float  *c,int  d);

Answer 6

[Jan]

void vxm_sse(float *a,float *b,float  *c,int  d);
calllib('kody', 'vxm_sse', pM, px, py0, length(x)/4);

While pM, px and py0 are float *, length(x)/4 replies a double, not an int. Let's assume that the int has 64 bits, then please try:

calllib('kody', 'vxm_sse', pM, px, py0, int64(length(x)/4));

or

calllib('kody', 'vxm_sse', pM, px, py0, int32(length(x)/4));

I assume that this should be changed also:

% y0=zeros(1,N); py0 = libpointer('singlePtr', y0);
y0 = zeros(1, N, 'single'); py0 = libpointer('singlePtr', y0);

Otherwise you provide a single pointer to double data. This should not crash, but reply messed up data.

Answer 7

[Samuel SLenker]

@Jan

The same script was used for x32 DLL in x32 Matlab and it worked without any problems. Of course, the .h file was defined with _stdcall.

None of the changes you mention above was an error. It worked fine. However, I tried to implement your suggestments and Matlab does the same as before - just crashes, no errors, no warnings.