How to extract some information from a txt file

Hi all,

I am facing with an issue on how to extract some information from a txt file in a matrix format. Find attached the txt file.

So lets say I want to extract the values of cNH3 over time in a matrix format (let's call matrix 1) and then the values of cNH2CL over time (matrix 2) for nodes ID Node 10, Node 40,Node 265 (not consecutive nodes), how could I do?

Actually I need these information for more than three nodes, lets say for i ID nodes.

Thanks in advance,

Stefania

Answer 1

[Star Strider]

Try this:

fidi = fopen('net3 report.txt');
for k = 1:26
    getline = fgetl(fidi);
end

k1 = 1;
while ~feof(fidi)
    NN = regexp(fgetl(fidi),'\d*', 'match');
    if (isempty(NN)) || (numel(NN) > 1)
        break
    else
        NodeNr(k1,:) = str2double(NN{:});
        secread{k1,:} = textscan(fidi,'%s%f%f', 'HeaderLines',4, 'CollectOutput',1);
        secread{k1,:}{1} = datetime(secread{k1}{1}, 'InputFormat','mmm:ss','Format','mmm:ss');
        k1 = k1 + 1;
    end
end
fclose(fidi);

.

[Stefania Avvedimento]

Thanks Star.

Could you please explain me what k=1:26 refers to?

Instead of having as many cells as there are nodes, is there any way to collect results in:

• 1 matrix time vs values of NH2CL for all nodes
• 1 matrix time vs values of NH3 for all nodes

The results format I would like to get is showed in the xlsx file I sent.

[Star Strider]

‘Could you please explain me what k=1:26 refers to?’

That reads the first 26 lines of the file and discards them.

‘Instead of having as many cells as there are nodes, is there any way to collect results ...’

To collect the values as matrices for each compound:

for k = 1:numel(secread)
    cNH3(:,k) = secread{k}{2}(:,1);
    cNH2Cl(:,k) = secread{k}{2}(:,2);
end
Tv = 0:numel(secread{1}{1})-2;                                  % Time Vector (minutes)

and to plot them:

figure
plot(Tv, cNH3)
grid
title('[NH_3]')
xlabel('Time (min)')
ylabel('Concentration')
xlim([min(Tv) max(Tv)])
lgd = legend(compose('%3d',NodeNr), 'Location','EastOutside');
lgd.NumColumns = 8;

figure
plot(Tv, cNH2Cl)
grid
title('[NH_2Cl]')
xlabel('Time (min)')
ylabel('Concentration')
xlim([min(Tv) max(Tv)])
lgd = legend(compose('%3d',NodeNr), 'Location','EastOutside');
lgd.NumColumns = 8;

Matrices for the Excel File:

cHN3_Mtx = [[NaN Tv]; [NodeNr cNH3.']];
cHN2Cl_Mtx = [[NaN Tv]; [NodeNr cNH2Cl.']];

I will let you write the Excel file with this information, since it is not obvious to me how to format it the way you want to.

[Stefania Avvedimento]

Great :)

thank you so much!

[Star Strider]

As always, my pleasure!

The matrices are quite large, so it might be best to put them on different sheets in the Excel file rather than the way you originally specified. I have not worked with Excel for a while, however it should still be possible to link them and do other calculations with them even if they are on different sheets.

[Star Strider]

‘if I have an HeaderlLine that can change from Node JUNCTION to Node RESERVOIR and Node TANK (see figure attached) how could I skip them?’

I do not understand what you want to do or how they are involved with the earlier code, since it does not appear that they were part of it.

Something like this would probably work:

while ~feof(fidi)
    getline = fgetl(fidi);
    if any(contains(getline, {'JUNCTION','RESERVOIR','TANK'}))
        continue
    end
    NN = regexp(getline,'\d*', 'match');
    if (isempty(NN)) || (numel(NN) > 1)
        break
    else
        NodeNr(k1,:) = str2double(NN{:});
        secread{k1,:} = textscan(fidi,'%s%f%f', 'HeaderLines',4, 'CollectOutput',1);
        secread{k1,:}{1} = datetime(secread{k1}{1}, 'InputFormat','mmm:ss','Format','mmm:ss');
        k1 = k1 + 1;
    end
end

I cannot test that, so I leave that to you. Let me know how it works.

‘Is there a reference where I can learn the code you wrote step by step? I don't want to just use it but understand it. For example, I can't understand the output of NN 1x5 cell (17 11 17 03 2021).’

The code first uses a for loop to read the first 26 lines of the file and discard them. (You can save them to a cell array if you want to. That would require only a minor change.)

The while loop iterates through the rest of the file until it either hits an end-of-file (eof) indicator or until the ‘NN’ variable is either empty (indicating that the code has read beyond the end of the file if there is no valid eof indicator) or if there are more than one element in ‘NN’ (so it should have trapped and excluded the (1x5) cell). The (1x5) cell will show up in ‘NN’, however it will not be saved in ‘NodeNr’, since the if,else block will cause that condition to exit the while loop before that. The rest uses the textscan function to read that section of the file, saves it in the ‘secread’ cell array, and converts the first cell to a datetime array. (Another possibility would save it as a duration array, however I was busy and a bit rushed yesterday and did not have time initially to explore that. I will re-consider it today and if that improves the code, I will post back here with the updated code.) It then closes the file and continues with the rest of the processing.

As for references, I linked to the most important functions here, so you can explore them.

(It could conceivably be possible to use functions other than textscan to do this, however textscan lends itself to such applications, and I have been using it for problems such as this before the newer file reading functions were introduced, and I am used to using it in this context.)

’I promise not to bother you anymore.’

No worries! If you have questions, I will do my best to reply to them.

[Star Strider]

That would likely be easier.

As always, my pleasure!

[Stefania Avvedimento]

Hi Star,

I hope you can help me again.

I have a similar .txt file that I want to extract in a matrix format. Just the heading at each results is different. For space limitations, I send you the txt file reduced in size.

So as in the previous task, I want to extract the values of EPANET over 240-h in a matrix format (let's call matrix 1) and then the values of WUDESIM over time (matrix 2) for i Junc_ID nodes and repeat the process til the end. Eventually (not necessary) extract the Pipe results also.

The problem is that I can't deal with the rows n. 7-8 since I'd like to extract just Junc_ID with the number below. So I am facing with the match expression function you previously called

NN = regexp(getline,'\d*', 'match');

Thanks a lot,

Stefania

[Star Strider]

I’m not exactly certain what you want.

T1 = readtable('https://www.mathworks.com/matlabcentral/answers/uploaded_files/770721/extract.txt', 'VariableNamingRule','preserve')

T1 = 494×7 table
    Time      EPANET      WUDESIM    Reyn_WUDESIM    Pecl_WUDESIM    ResT_WUDESIM       Var7   
    ____    __________    _______    ____________    ____________    ____________    __________

      0              0       0          266.73           4.228          72834        {0×0 char}
      1     7.4449e-18       0          264.46          4.2642          73457        {0×0 char}
      2     3.4526e-13       0          263.06           4.287          73850        {0×0 char}
      3     6.6482e-11       0          262.69           4.293          73954        {0×0 char}
      4      2.086e-09       0          261.98          4.3047          74155        {0×0 char}
      5     2.0398e-08       0          262.01          4.3041          74145        {0×0 char}
      6     9.8362e-08       0          264.35           4.266          73489        {0×0 char}
      7     3.2544e-07       0           270.9          4.1628          71711        {0×0 char}
      8     9.3243e-07       0          273.69          4.1204          70980        {0×0 char}
      9      2.364e-06       0          273.35          4.1256          71070        {0×0 char}
     10     4.9397e-06       0          272.98          4.1311          71165        {0×0 char}
     11     8.6201e-06       0          272.63          4.1364          71257        {0×0 char}
     12     1.3135e-05       0          272.58          4.1372          71269        {0×0 char}
     13     1.8303e-05       0           272.8          4.1338          71212        {0×0 char}
     14     2.4263e-05       0          273.02          4.1305          71154        {0×0 char}
     15     3.1836e-05       0          272.17          4.1435          71378        {0×0 char}

NANs = nnz(isnan(T1.Time))                                              % Determine 'NaN' Values

NANs = 10

T1 = rmmissing(T1(:,1:6))                                               % Remove Missing Values For Selected Variables

T1 = 484×6 table
    Time      EPANET      WUDESIM    Reyn_WUDESIM    Pecl_WUDESIM    ResT_WUDESIM
    ____    __________    _______    ____________    ____________    ____________

      0              0       0          266.73           4.228          72834    
      1     7.4449e-18       0          264.46          4.2642          73457    
      2     3.4526e-13       0          263.06           4.287          73850    
      3     6.6482e-11       0          262.69           4.293          73954    
      4      2.086e-09       0          261.98          4.3047          74155    
      5     2.0398e-08       0          262.01          4.3041          74145    
      6     9.8362e-08       0          264.35           4.266          73489    
      7     3.2544e-07       0           270.9          4.1628          71711    
      8     9.3243e-07       0          273.69          4.1204          70980    
      9      2.364e-06       0          273.35          4.1256          71070    
     10     4.9397e-06       0          272.98          4.1311          71165    
     11     8.6201e-06       0          272.63          4.1364          71257    
     12     1.3135e-05       0          272.58          4.1372          71269    
     13     1.8303e-05       0           272.8          4.1338          71212    
     14     2.4263e-05       0          273.02          4.1305          71154    
     15     3.1836e-05       0          272.17          4.1435          71378    

T240 = find(T1.Time == max(T1.Time));                                   % Find Indices Where: Time = 240
T2 = T1(1:T240(end),:)                                                  % Truncate 'T1'

T2 = 482×6 table
    Time      EPANET      WUDESIM    Reyn_WUDESIM    Pecl_WUDESIM    ResT_WUDESIM
    ____    __________    _______    ____________    ____________    ____________

      0              0       0          266.73           4.228          72834    
      1     7.4449e-18       0          264.46          4.2642          73457    
      2     3.4526e-13       0          263.06           4.287          73850    
      3     6.6482e-11       0          262.69           4.293          73954    
      4      2.086e-09       0          261.98          4.3047          74155    
      5     2.0398e-08       0          262.01          4.3041          74145    
      6     9.8362e-08       0          264.35           4.266          73489    
      7     3.2544e-07       0           270.9          4.1628          71711    
      8     9.3243e-07       0          273.69          4.1204          70980    
      9      2.364e-06       0          273.35          4.1256          71070    
     10     4.9397e-06       0          272.98          4.1311          71165    
     11     8.6201e-06       0          272.63          4.1364          71257    
     12     1.3135e-05       0          272.58          4.1372          71269    
     13     1.8303e-05       0           272.8          4.1338          71212    
     14     2.4263e-05       0          273.02          4.1305          71154    
     15     3.1836e-05       0          272.17          4.1435          71378    

Time_r = bufr(T2.Time, 241)                                             % 'Time' Matrix

Time_r = 241×2

0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9

EPANET_r = bufr(T2.EPANET,241)                                          % 'EPANET' Matrix

EPANET_r = 241×2

1.0e+00 * 0 0 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

WUDESIM_r = bufr(T2.WUDESIM,241)                                        % 'WUDESIM' Matrix

WUDESIM_r = 241×2

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

figure
plot(Time_r(:,1), [EPANET_r, WUDESIM_r])
grid

function M = bufr(s,r)
% Star Strider's Emulation Of The Basic 'buffer' Function
%   s = Signal Vector
%   r = Desired number of rows
M = zeros(r,ceil(numel(s)/r));
M(1:numel(s)) = s;
end

This was more of a challenge than I’d anticipated, because I wasn’t expecting so many missing values. Removing them was necessary to get this working. (Since the last column of ‘T1’ is entirely missing, I removed it from ‘T2’.)

This uses my emulation of the buffer function to reshape the matrices.

.

[Stefania Avvedimento]

Hi Star,

sorry if I could not explain the aim of my work. To give you the idea I attach below a figure in which I show you how I'd like to get the results. So, starting from the extract.txt file (which is reduced in size and contains only 2 branch_IDs, let's imagine I have n branch_ID) I would like to get two matrices:

-Mtx1 matrix containing the values of Chlorine (mg/L) over time of EPANET for all Junc_ID

-Mtx2 matrix containing the values of Chlorine (mg/L) over time of WUDESIM for all Junc_ID

I am not interest in getting the Pipe results.

Is there any way to develop a code to automate the process of extraction?

Anyway, when I run T1 = readtable('extract.txt', 'VariableNamingRule','preserve') Matlab gives me this error: Invalid parameter name: VariableNamingRule.

Thanks,

Stefania

[Star Strider]

I am lost.

I am not certain what you want, and I do not understand ‘example results.png’.

The data only imported for ‘Chlorine’ anyway, and for whatever reason readtable did not import the ‘Pipe results’ columns. (Those would require a different appproach to read.)

So in the end, ‘Time_r’, ‘EPANET_r’, and ‘WUDESIM_r’, are all for ‘Chlorine’. The ‘Junc_ID’ then represents each column of the resulting matrices. That code would be essentially the same regardless, because the results would be essentially the same. To get the matrices for the column data to be row vectors instead, simply transpose them.

With respect to 'VariableNamingRule', either substitute that with 'ReadVariableNames',true, or just omit it. Not all versions of readtable support those options.

.