Hard to say without knowing what operations your program is doing. But since you have a lot of free memory left, you might be running in to memory fragmentation problems rather than outright memory exhaustion. When you allocate a Matlab array, it gets carved out of the existing memory space. When you clear that array, the memory gets released back to the pool (at the C malloc level). But that chunk doesn't necessarily get coalesced with the other freed chunks to produce bigger chunks. If you allocate a lot of arrays over time, that can fragment your memory into small chunks so that you can no longer allocate large arrays, even though you have lots of memory available to you in aggregate.
The matlab function memory() will give you some stats that help you diagnose this. Have your program run that and display the results. Look for the "Maximum possible array"/`MaxPossibleArrayBytes` value. If that's a lot smaller than the total memory you have available to you, then you've got fragmentation.
Unless the Out of Memory error you're getting specifically says that it is a Java heap OOM (as opposed to a regular OOM), don't bother messing with your Java settings; that won't do anything.
Unfortunately, fixing memory fragmentation requires modifying your code to just allocate fewer arrays in the first place. Cellstrs, strings, and cells or objects containing many small arrays are a common culprit for fragmentation. Switching to categorical arrays and tables can help a lot there.
As for the 32-bit app working fine on a 64-bit OS: IIRC, on 32-bit Windows, the OS reserves the top half of virtual memory space for itself, so user programs only get 2 GB of RAM to work with. But on 64-bit Windows, this is removed, so 32-bit programs get a full 4 GB of virtual memory space to work with. That might be enough to get you over the threshold where this works. There is a kernel setting on 32-bit Windows to modify things so user programs get 3 GB instead of 2 GB, which may also help you. (Google "Windows 3gb switch" for details.)
