Receiving I/O error 32 when opening another program (EnergyPlan) inside parfor

Hello everyone,

I am trying to use parralel computing to run the same program (EnergyPlan) simultaneously. EnergyPlan is a energy system tool that computes whether a given energy system is able to supply the given demand. I would like to run different energy system configurations (e.g. different amounts of wind and solar power) and use an optimization algorithm to find the best setup for the given demand.

The program is set up that it opens EnergyPlan using the "system" command to execute EnergyPlan. Afterwards the results of each run are retrieved compared to see, which system scores highest. When I run the program in a simple for-loop, it works without problems. However, when I change the for-loop to a parfor-loop, the opened EnergyPlan displays the error "I/O error 32". According to sources I found this error indicates a sharing violation. The first EnergyPlan program is executed (unfortunately the program pops up so I can always see when a run is performed) just fine and it closes again. However, there is a second EnergyPlan opened while the first one is still open and this one shows the beforementioned error. According to EnergyPlan.eu the tool can be run in parallel and other researchers have done so succesfully using python or java. Therefore, I don't know why this error occurs with Matlab.

Does anyone know how to circumvent this issue? Any help would be much appreciated.

Regards,

Markus

Minimum working example (provided EnergyPlan and toolbox for Matlab are installed):

   %%Paths and Folder definitions
%Path where it is located EnergyPLAN executable.
dir.energyPlanPath = 'energyPLAN.exe';
%Path of reference file.
dir.inputFilePath = 'energyPlan Data\Data\Denmark2030Alternative.txt';
%Folder where will be output resulf of EnergyPLAN.
dir.outputFolder = 'Outputs\';
REvalue = 1e4*rand(10,2);
result = zeros(size(REvalue,1),1);
parfor i = 1:length(REvalue)

        % Call EnergyPlan
        annualData = energyPlan(dir.energyPlanPath,...
            dir.inputFilePath,dir.outputFolder,...
            'input_RES1_capacity=', REvalue(i,1),...    % 1st variable to be changed
            'input_RES2_capacity=', REvalue(i,2));      % 2nd variable to be changed 

        result(i) = annualData(55);      % random value taken out
end

[OCDER]

https://www.energyplan.eu/useful_resources/matlab-toolbox-for-energyplan/

Is it a toolbox, or a standalone .exe that you are using via system command? If it's a matlab program, it should work WITHOUT invoking something like system('EnergyPlan.exe')

Can you copy & paste the help info for the function you want to use? Also provide the code for using parfor, minus any extra code that isn't needed.

[MD]

Hey OCDER,

Thanks for your response! That is the toolbox I am using. Inside the main function "energyPlan" is the command "system(executionString)". EnergyPlan itself is a standalone program. The toolbox was created to be able to access the program via Matlab.

This is the info of the function "energyPlan":

% ENERGYPLAN  Call EnergyPLAN tool from Matlab and save results. 

%   [A, M, H] = ENERGYPLAN('energyPlanPath','inputFilePath', 'outputFolder', param,value, ...) 

%    ENERGYPLAN is useful for executing one or several configurations of 

%    models in EnergyPLAN, save the results and generate vectors with 

%    annual, monthly and hourly data.

%    USER CONFIGURABLE OPTIONS

%    Possible param/value options are:

%         'input_RES1_capacity=' - RES1 installed capacity

%         'input_RES2_capacity=' - RES2 installed capacity

%         'input_RES1_factor='   - factor correction of RES1

%           .

%           .

%           .

%         'Input_FOM_ind_CHP_heat=' 

% Other m-files required: none

% Subfunctions: changeInputEnergyPlan, readOutputEnergyPlan

% MAT-files required: none

%  See also CHANGEINPUTENERGYPLAN, READOUTPUTENERGYPLAN.

I put a minimum working example (provided both EnergyPlan and the toolbox are installed) in the question.

[OCDER]

I guess I'm still not seeing the system(executionString). Is this something you wrong into your code, or it's a part of EnergyPlan?

What do you get if you do:

which energyPlan

When you use system, it's the same as running from the cmd of windows, and thus it's not really running from inside Matlab. Can you copy paste the line where system is used, and tell what .m file it's in?

[MD]

Hey OCDER,

I get "C:\Users\MD\Documents\MATLAB\Add-Ons\Toolboxes\EnergyPLAN Toolbox\code\energyPlan.m"

Inside that file there are the following lines:

 %%Function that changes input reference file.
%Call to function that changes inputs.
changeInputEnergyPlan(inputFilePath,newInputFilePath,varargin);
%%EnergyPLAN is executed.
executionString=['"',energyPlanPath,'" -i "', newInputFilePath, '" -ascii "', outputFilePath,'"'];
system(executionString); %execution of EnergyPLAN.
%%Annual, monthly and hourly results are obtained from the output file.
[annualData, monthlyData, hourlyData]=readOutputEnergyPlan(outputFilePath);

Under "%%EnergyPlan is executed" you find

 system(executionString); %execution of EnergyPLAN.

[OCDER]

I see what you mean now in the initial post. Hmmm... how about adding a "start" in that system command of that .m file?

https://stackoverflow.com/questions/1449188/running-windows-batch-file-commands-asynchronously

So something like this:

system(['start ' executionString])

[MD]

Thanks again for your response! Unfortunately, when I add start to the beginning of the command as recommended in your link, I receive a different error.

According to https://www.energyplan.eu/training/faqs/#eleven I need to write "C:\kt\EnergyPLAN\run\energyPLAN.exe -i c:\MyInputFolder\Inputfile.txt -ascii c:\MyResultFolder\MyResult.txt" to run the program from cmd. This works just fine. However, when I add "start" it shows the error "'-i' could not be found. Ensure that the name is typed correctly and repeat the process". It seems that sees "-i" as directory.

Unfortunately, I am not very familiar with using cmd and I do not know how to change the command line accordingly. Would you know? Maybe it has to do with the fact that I add quotes to my directories:

start "C:\Users\...\EnergyPlan\energyPLAN.exe" -i "C:\Users\...\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp.txt" -ascii "C:\Users\...\EnergyPlan\Outputs\out_Denmark2030Alternative.txt"

[OCDER]

so this doesn't work?

start C:\kt\EnergyPLAN\run\energyPLAN.exe -i c:\MyInputFolder\Inputfile.txt -ascii c:\MyResultFolder\MyResult.txt

I see a space in your command.

 -i c:\MyInputFolder\ Inputfile.txt
                     ^

Or modifying that matlab code as suggested?

 system(['start ' executionString])  %Note the space after the word start.

[MD]

The first example is just a copy of the text in the link. The command at the end is the one that I use (without the full directory as it is unnecessary and a bit long).

I tried it without "start" and as normal for-loop and it works without a problem. When I add the "'start '" to the command, I receive the command about not finding "-i". Instead of interpreting it as command, cmd seems to think that "-i" is a file and not the instruction on what to do with EnergyPlan. I inserted a screenshot of the error (unfortunately in German but the translation is in the previous comment).

[OCDER]

I see. If you have spaces in the full file path to the .exe and have to use quotes, then you need to also specify the title for the start window it seems.

start "title1" "C:\Users\...\EnergyPlan\energyPLAN.exe" -i "C:\Users\...\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp.txt" -ascii "C:\Users\...\EnergyPlan\Outputs\out_Denmark2030Alternative.txt"  
      ^^^^^^^^  title of this window

[MD]

Thanks! It seems like we are getting closer to the solution. However, I am not quite sure what you mean with title of this windows. Is there a specific title name that I have to add? When I just use "title1", I get the following error:

Warning: File 'C:\Users\...\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp.txt' not found.
> In energyPlan (line 66)
  In parallel_function>make_general_channel/channel_general (line 917)
  In remoteParallelFunction (line 41)

I tried giving each run and individual temporary file (the Matlab toolbox creates a "..._tmp.txt" file for the evaluation of the system) but it still didn't work.

I also tried to eliminate all spaces to do without the qotes. Thus my command line was

start C:\Users\...\EnergyPlan\energyPLAN.exe -i C:\Users\...\EnergyPlan\Data\Denmark2030Alternative_tmp.txt -ascii C:\Users\...\EnergyPlan\Outputs\out_Denmark2030Alternative.txt

However, this gave me the error that it could not find the file since it thinks that the file is not only one directory but all of them combined:

I also receive the error I/O 103.

Once I remove "start" from the command line again, the program works just fine with a normal for-loop.

C:\Users\...\EnergyPlan\energyPLAN.exe -i C:\Users\...\EnergyPlan\Data\Denmark2030Alternative_tmp.txt -ascii C:\Users\...\EnergyPlan\Outputs\out_Denmark2030Alternative.txt

[OCDER]

Wait, are you using "..." in your file path, or you're just doing this to condense the file path for the posting here?

Try this in matlab, filling in the "..." with the full path to the files. The title is any title name for the window it's going to open - not important:

 system('start "random_title" "c:\...\energyPlan.exe" -i "C:\Users\...\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp.txt" -ascii "C:\Users\...\EnergyPlan\Outputs\out_Denmark2030Alternative.txt"')

Also note the apostrophe encasing EVERYTHING.

system('....')
       ^    ^

In my system, something like this works in Matlab:

parfor j = 1:4
   system('start "blah" "C:\somePath\someFile.exe"')
end

[OCDER]

Just wondering, have you asked them directly how to do use their energy plan in Matlab parfor loops? They might have a document for that somewhere that's not published.

https://www.energyplan.eu/contact/

[Walter Roberson]

The line

changeInputEnergyPlan(inputFilePath,newInputFilePath,varargin);

looks to me as if it is likely doing an assignin('caller') or assignin('base') or changing global variables.

If so then you will need to toss your code into a function that you call from the parfor loop: poofing variables into existence directly inside a parfor block gets you a sharing violation.

[MD]

I am using "..." in my file path since they contain spaces. However, in my previous post I tried to get rid of the spaces in the file paths to be able to also get rid of the quotes.

These are the combinations that work. I took out the dots now to make sure that they don't cause confusion.

1. Quotes are used. Simple for-loop.

executionString=['"',energyPlanPath,'" -i "', newInputFilePath, '" -ascii "', outputFilePath,'"'];
system(executionString); %execution of EnergyPLAN.

The executionString yields:

"C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPLAN.exe" -i "C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp.txt" -ascii "C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\Outputs\out_Denmark2030Alternative.txt"

2. Quotes are not used (spaces are not used in file path). Simple for-loop

executionString=[energyPlanPath, ' -i ', newInputFilePath, ' -ascii ', outputFilePath];
system(executionString); %execution of EnergyPLAN.

The executionString yields:

C:\Users\HumptyDumpty\Desktop\Programmeetc\Studium\4oSemestre\MasterThesis\EnergyPlan\energyPLAN.exe -i C:\Users\HumptyDumpty\Desktop\Programmeetc\Studium\4oSemestre\MasterThesis\EnergyPlan\Data\Denmark2030Alternative_tmp.txt -ascii C:\Users\HumptyDumpty\Desktop\Programmeetc\Studium\4oSemestre\MasterThesis\EnergyPlan\Outputs\out_Denmark2030Alternative.txt

These are the combinations that don't work.

1. Quotes are used. Either for-loop or parloop. I used num2str(i) here to ensure that every title is different but also when I simply use "title1", it doesn't work.

executionString=['start "title',num2str(i),'" "',energyPlanPath,'" -i "', newInputFilePath, '" -ascii "', outputFilePath,'"'];
system(executionString); %execution of EnergyPLAN.

The executionString yields:

start "title1" "C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPLAN.exe" -i "C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp.txt" -ascii "C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\Outputs\out_Denmark2030Alternative.txt"

The error I get says that it cannot the file. You can see the error in the screenshot above that one of the error I/O 103.

2. Quotes are not used. Either simple for-loop or parfor. This time I don't use "title1" as this is apparently only necessary when using quotes. I changed the folder names again so they don't contain blanks.

executionString=['start ', energyPlanPath, ' -i ', newInputFilePath, ' -ascii ', outputFilePath];
system(executionString); %execution of EnergyPLAN.

The executionString yields:

start C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPLAN.exe -i C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp.txt -ascii C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\Outputs\out_Denmark2030Alternative.txt

The error is the same as before that it cannot find the file as the command seems to mix up which is simply information about what to treat the file as (like -i and -ascii) and the actual directory paths.

Both ways - quotes or without - work just fine when I don't add "start" before it and use a simple for-loop. As soon as "start" is added it cannot find the file anymore.

[MD]

I have contacted them and this was their response:

I am not sure you can run it parallel if they use they rely on the same input/outputs. If not, it should not be a problem to run to versions next to each other.

However, this is the case. There are other studies that compute simultaneously but not using Matlab.

I also contacted the creator of the toolbox. However, he has not responded yet.

[MD]

@Walter: Thanks for your input. This is the function:

function changeInputEnergyPlan(inputFilePath,outputFilePath, varargin)
        if iscell(varargin{1,1})==1; varargin=varargin{1,1}; end
        warning('off','all')
        % Read txt into cell A
        fid = fopen(inputFilePath,'rt', 'n', 'UTF16LE');
        i = 1;
        tline = fgetl(fid);
        A{i,1} = tline;
        while ischar(tline)
            i = i+1;
            tline = fgetl(fid);
            A{i,1} = tline;
        end
        fclose(fid);

         A{1,1}='EnergyPLAN version';
         A{2,1}=[];
         A=A(~cellfun('isempty',A));

         % % Change cell A
         for j=1:length(varargin)
              if mod(j,2)== 0 % solo cojo los valores par de las entradas
                  tf=strcmp(A,varargin(j-1));
                  tf = circshift(tf,1);
                  if iscellstr(varargin(j))==1
                      A{tf}=varargin{j};  
                  else
                      A{tf}=sprintf('%1.2f',varargin{j});        
                  end
              end
         end

          %A{18} = sprintf('%d',100);

          % Write cell A into txt
          fid = fopen(outputFilePath, 'wt');
          for i = 1:numel(A)
              if A{i+1,1} == -1
                  fprintf(fid,'%s', A{i,1});
                  break
              else
                  fprintf(fid,'%s\n', A{i,1});
              end
          end
          fclose(fid);
          warning('on','all')
  end

It is used to change the values in the .txt file that EnergyPlan reads in. So yes there could be an issue as all agents try to access this file. I changed the filename for each run using "i" to change the name of each input file and output file:

newInputFilePath =[inputFilePath(1:end-4),num2str(i),'_tmp.txt'];
outputFileName = ['out_',num2str(i),inputFilePath(k(end)+1:end)];

When I run it with a simple for-loop there is no problem. I run it and it simply creates more temporary files. However, when I run it with parfor it shows the same error that it cannot find the file as shown in the screenshot. So there might also be an assignment issue but I would only know, if the program would correctly identify the file names.

[OCDER]

energyPlanPath = 'C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPLAN.exe';
origInput = 'C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp.txt'; 
inputPre = 'C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp';
outputPre = 'C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\Outputs\out_Denmark2030Alternative';
%
assert(~isempty(dir(energyPlanPath)), 'energyPlanPath is not valid. Check file name.');
assert(~isempty(dir(origInput)), 'origInput file is not valid. Check file name.');
%
%To prevent parfor from accessing and writing to same file, make copies and specify separate outputs
Inputs = cell(4, 1);
Outputs = cell(4, 1);
for j = 1:4
    Inputs{j}  = [inputPre  num2str(j) '.txt'];
    [Success, Msg] = copyfile(origInput, Inputs{j});
    assert(Success, Msg);
    Outputs{j} = [outputPre num2str(j) '.txt'];
end
%Attempt to run in parallel
parfor j = 1:4
    executionString = sprintf('start "title" "%s" -i "%s" -ascii "%s"', energyPlanPath, Inputs{j}, Outputs{j});
    system(executionString); %execution of EnergyPLAN.
end

If this fails, it seems like the author of the code did not program this to run in parallel if it is going to share a common file. The other people might be using a workaround to contain each instance of the energy plan in a separate container and then run multiple versions, similar to running it on multiple computers.

[MD]

@OCDER: It worked!!! The strange thing is that the loop doesn't wait for the system command to finish but continues with the readOutputEnergyPlan. Then it opens the EnergyPlan instances afterwards while Matlab has already given me an error (which makes sense since the output cannot be read when it is not there). It actually opens even more instances of EnergyPlan than there are logical cores at the same time. Furthermore, it even opens more than 4 instances in your example. I guess, I will still have to investigate a bit more.

However, it does not have problems finding the file anymore. The computation is run and it obtains an output. Thank you very much for solving the issue! I would have never found it without you.

[OCDER]

Nice! If that works, can you go ahead an accept the answer below? That way we know the question is answered with a working solution, and I get some reputation points :) Play around with removing start so that it waits for it to finish per parallel worker. I think the file naming and having multiple input file copies was the real issue, not start. If you do need start, then you might need another step to wait for the output file to be generated before continuing.

Answer 1

[OCDER]

energyPlanPath = 'C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPLAN.exe';
origInput = 'C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp.txt'; 
inputPre = 'C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\energyPlan Data\Data\Denmark2030Alternative_tmp';
outputPre = 'C:\Users\HumptyDumpty\Desktop\Programme etc\Studium\4o Semestre\Master Thesis\EnergyPlan\Outputs\out_Denmark2030Alternative';
%
assert(~isempty(dir(energyPlanPath)), 'energyPlanPath is not valid. Check file name.');
assert(~isempty(dir(origInput)), 'origInput file is not valid. Check file name.');
%
%To prevent parfor from accessing and writing to same file, make copies and specify separate outputs
Inputs = cell(4, 1);
Outputs = cell(4, 1);
for j = 1:4
    Inputs{j}  = [inputPre  num2str(j) '.txt'];
    [Success, Msg] = copyfile(origInput, Inputs{j});
    assert(Success, Msg);
    Outputs{j} = [outputPre num2str(j) '.txt'];
end
%Attempt to run in parallel
parfor j = 1:4
    executionString = sprintf('start "title" "%s" -i "%s" -ascii "%s"', energyPlanPath, Inputs{j}, Outputs{j});
    system(executionString); %execution of EnergyPLAN.
    %to wait for output file to be generated
    while isempty(dir(Outputs{j}))
       pause(0.1); %check again in 0.1 sec
    end
    ... rest of your codes
end

[MD]

I managed to run it smoothly without any problems now. The reason for the other problems was actually the use of "start". This forces the program to open regardless of whether Matlab is ready or not. With this executionString it works perfectly since a new instance of EnergyPlan is only opened when the old one was computed:

executionString = sprintf('"%s" -i "%s" -ascii "%s"', energyPlanPath, Inputs{j}, Outputs{j});

Now I only need to find a way to force the program to run in the background and not pop up every single time it runs.

Thanks again!