Using Tensorlab with large tensors

I keep receiving the following error when trying to use Tensorlab for a CPD of structured data fusion.

Error using mtkrprod

Intermediate result is too large. Try to format data with fmt first so that the

large-scale version of the algorithm can be applied.

However, I am using the fmt function on the tensor. Any help you could provide would be much appreciated. Here is the code:

addpath(genpath('/data/epifluidlab/david'));

resolutions = [25000, 10000];

chromosomes = {'1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22'};

for resolution = resolutions

for i = 1:numel(chromosomes)

chromosome = chromosomes{i};

% Load the correlation matrix

file_path = sprintf('%sWorkspaces/workspace_ch%s_res%d_oe_NONE_reconstructed.mat', path, chromosome, resolution);

load(file_path);

matrix = reconstructed_matrix;

A1 = downsample_matrix(matrix, 10);

A2 = downsample_matrix(matrix, 20);

A3 = downsample_matrix(matrix, 30);

A4 = downsample_matrix(matrix, 40);

A5 = downsample_matrix(matrix, 50);

A6 = downsample_matrix(matrix, 60);

A7 = downsample_matrix(matrix, 70);

A8 = downsample_matrix(matrix, 80);

A9 = downsample_matrix(matrix, 90);

A10 = matrix;

r = 2;

sol_cell_downsampled = cell(1,r);

sol_factors_downsampled = cell(1,r);

sol_cell_downsampled_negative = cell(1,r);

sol_factors_downsampled_negative = cell(1,r);

ranks = 1:r;

n = size(A1,1);

size_tensor = [n n 10];

R = 2;

U = cpd_rnd(size_tensor,R);

T = fmt(cpdgen(U));

T(:,:,1) = A1;

T(:,:,2) = A2;

T(:,:,3) = A3;

T(:,:,4) = A4;

T(:,:,5) = A5;

T(:,:,6) = A6;

T(:,:,7) = A7;

T(:,:,8) = A8;

T(:,:,9) = A9;

T(:,:,10) = A10;

T = fmt(T);

for i = 1:r

model = struct;

model.variables.u = randn(n,i);

model.variables.s = randn(10,i);

model.factors.U = {'u',@struct_nonneg};

model.factors.S = {'s',@struct_nonneg};

model.factorizations.myfac.data = T;

model.factorizations.myfac.cpd = {'U', 'U', 'S'};

options.Display = 100;

%options.TolFun=1e-50;

%options.TolX=1e-50;

options.MaxIter=200;

sol = sdf_nls(model,options);

sol_factors_downsampled{1,i} = {sol.factors.U, sol.factors.S};

end

% Save the processed data for each chromosome

output_file = sprintf('%sWorkspaces/ch%s_res%d_structedData_10Downsampled_oe_first3_200iterations.mat', path, chromosome, resolution);

save(output_file);

end

end

Answer 1

[Shivansh]

Hi David,

The error message you are encountering suggests that the intermediate result in the ‘Tensorlab’ library is too large to handle. The error message also suggests using the `fmt` function to format the data so that the large-scale version of the algorithm can be applied. However, it seems that you are already using the `fmt` function on the tensor.

In this case, there are a few potential reasons for the error:

1. Data size: The size of your data might still be too large even after applying the `fmt` function or you might be applying the ‘fmt’ in wrong places. Try assigning the values to T in a loop and use 'fmt' after every iteration.

The error could also be due to the limitations of the algorithm or the available memory. You may need to further reduce the size of your data or consider using a different algorithm or library that can handle larger datasets.

2. Implementation issue: There could be an issue with the implementation of the ‘Tensorlab’ library or the specific function you are using. You may want to check for any updates or bug reports related to the library or function you are using. You can check the Tensorlab documentation here.

You can also try to use batches of data instead of working with the entire data at the same time. One possible way to achieve this is by splitting the computation into smaller chunks instead of processing all the chromosomes and resolutions together.

3. Resource limitations: It is possible that your system does not have enough memory or computational resources to handle the intermediate result. You can try running the code on a machine with higher memory capacity or consider optimizing your code to reduce memory usage.

You can try increasing the downsampling factor further to decrease the size of matrices which will result in even smaller tensors.