how to make monthly graph from daily file data
Show older comments
thank to per isakson
from this code (daily/1 file) can u make monthly graph (30file data)?? =====
function RainData = ReadManySoniData( folder_name, file_spec )
sad = dir( fullfile( folder_name, file_spec ) );
RainData = struct([]);
for sa = transpose( sad )
RainData = cat( 2, RainData, ReadOneSoniData( folder_name, sa.name ) );
end
[ dummy, ixs ] = sort( [ RainData(:).DayNumber ] );
RainData = RainData( ixs );
end
function rain_data = ReadOneSoniData( folder_name, file_name )
fid = fopen( fullfile( folder_name, file_name ), 'r' );
if not( fid >= 3 )
error( 'ReadOneSoniData:NoFileFound' ...
, 'Cannot find file "%s"' ...
, fullfile( folder_name, file_name ) )
end
cac = textscan( fid, '%s', 'Whitespace','', 'Delimiter','\n' );
fclose( fid );
cac = cac{:};
tmp = cellfun( 'isempty' ...
, regexp( cac, '\d{2}:\d{2}:\d{2}\s+C\s*$' ) );
isc = not( tmp );
tmp = cellfun( 'isempty' ...
, regexp( cac, '\d{2}:\d{2}:\d{2}\s+\*\*\s+----' ) );
iss = not( tmp );
cac( isc | iss ) = [];
str = transpose( char( cac ) );
nl = sprintf('\n');
str = cat( 1, str, repmat( nl(:), [length(nl),size(str,2)] ) );
cac = cell(1,9);
[cac{:}] = strread( str, '%8c%2c%4f%7f%4c%4u%4u%4u+%2u' ...
, 'delimiter', ' ', 'whitespace', '' );
try
date_vec = nan(1,3);
date_vec( [2,3,1] ) = sscanf( file_name, '%2u-%2u-%4u%*s' );
catch me
if strcmp( me.identifier, 'MATLAB:index_assign_element_count_mismatch' )
warning( 'ReadOneSoniData:CannotParseFileName' ...
, 'Cannot extract a date from file name: "%s"' ...
, file_name )
rain_data = struct([]);
return
else
rethrow( me )
end
end
str = transpose( char( cac{1} ) );
vec = nan( size(str,2), 3 );
[ vec(:,1), vec(:,2), vec(:,3) ] ...
= strread( str, '%2u:%2u:%2u', 'delimiter','','whitespace','' );
rain_data.Created = datestr( now, 'yyyy-mm-dd HH:MM:SS' );
rain_data.DataFile = fullfile( folder_name, file_name );
rain_data.Datevec = [ repmat( date_vec, [size(vec,1),1] ), vec ];
rain_data.DayNumber = datenum( date_vec );
rain_data.Rain = cac{3};
rain_data.DailyRain = sum( rain_data.Rain );
% and more as you see fit.
end
2 Comments
per isakson
on 6 Jul 2012
"from this code (daily/1 file) can u make monthly graph (30file data)??"
Put more effort in describing what you need!
Accepted Answer
per isakson
on 6 Jul 2012
Edited: per isakson
on 6 Jul 2012
Here is a function that returns total monthly rain. Try
>> mr = MonthlyRain( RainData );
>> plot( mr(1).Rain, 'd' );
>> bar( mr.Rain );
The values of the monthly rain could they be correct?
function monthly_rain = MonthlyRain( RainData )
day_number = [ RainData(:).DayNumber ];
month_number= month( day_number );
year_number = year( day_number );
year_list = unique( year_number );
monthly_rain = struct( 'Year', num2cell( year_list ), 'Rain', nan(12,1) );
ix_yy = 0;
for yy = year_list
is_yy = ( yy == year_number );
ix_yy = ix_yy + 1;
for mm = 1 : 12
is_mm = ( mm == month_number );
is_ym = ( is_yy & is_mm );
if any( is_ym )
monthly_rain(ix_yy).Rain(mm) = sum([RainData( is_ym ).DailyRain]);
end
end
end
end
63 Comments
Soni huu
on 6 Jul 2012
per isakson
on 6 Jul 2012
Edited: per isakson
on 6 Jul 2012
Not needed! I'm worried about missing data and how that should be reported. In next version?
Soni huu
on 11 Jul 2012
per isakson
on 11 Jul 2012
Have you solved the problem with reading the 2010 data?
what does
[ mr.Rain ]
return?
per isakson
on 12 Jul 2012
Edited: per isakson
on 12 Jul 2012
It's close to impossible for me to know what causes your problems. I made a bar graph at one point, which showed to bars.
I don't know what changes you made to the code. I don't try to keep my own version.
You avoid to answer to my questions. Thus, I repeat:
Have you solved the problem with reading the 2010 data?
what does
[ mr.Rain ]
return?
Soni huu
on 12 Jul 2012
per isakson
on 12 Jul 2012
Did you delete the short lines?
When you type
[ mr.Rain ]
in the command window what does matlab print in the command window?
Soni huu
on 12 Jul 2012
Soni huu
on 12 Jul 2012
per isakson
on 12 Jul 2012
Edited: per isakson
on 12 Jul 2012
That is a really good question!
- inspect and understand the code
- test the code, e.g make a file with synthetic data for which you know the answers
- especially, you need to make tests with files with missing data
Goggle for "Testing software" :-) ... and there is always another bug!
per isakson
on 12 Jul 2012
Try
>> mr(ii).Year
The output below is that from the test file with a few days in november?
...
NaN
4.0540
NaN
per isakson
on 12 Jul 2012
"yea i delete the short line coz the instrument ..."
The problem with that is that next time you will also need to do it manually.
My comment per isakson on 10 Jul 2012 at 16:32
First step: In ReadManySoniData replace
....
aimed at an automatic solution.
Soni huu
on 12 Jul 2012
per isakson
on 12 Jul 2012
Edited: per isakson
on 12 Jul 2012
You missed "try"
...
for sa = transpose( sad )
try
RainData = cat(2,RainData,ReadOneSoniData(folder_name,sa.name);
catch
fprintf( 'Folder: %s\nFile: %s\n', foldername, sa.name )
disp( lasterr )
end
end
...
Soni huu
on 12 Jul 2012
per isakson
on 12 Jul 2012
Edited: per isakson
on 12 Jul 2012
You know - not me - whats on line: 7 column: 71. Fix the line! I guess it is is a typing error.
Make sure the Code Analyzer box is green before you run the code.
Soni huu
on 12 Jul 2012
per isakson
on 12 Jul 2012
Edited: per isakson
on 12 Jul 2012
Replace
foldername
by
folder_name
.
The Code Analyzer box was it green before you run the code?
Soni huu
on 12 Jul 2012
Soni huu
on 12 Jul 2012
per isakson
on 12 Jul 2012
Edited: per isakson
on 12 Jul 2012
Now you have a "list" of dat-files with lines that the code cannot read. These two steps remain:
Third step: Inspect the files (causing trouble) with an editor. Make a list with the lines, which you think are the cause of the trouble.
Fourth step: What do these lines have in common? How can these lines be identified? By what rule?
.
The message
Error in ==> ReadManySoniData at 6
RainData = cat(2,RainData,ReadOneSoniData( folder_name, sa.name ));
does not mean anything to me!
else
fprinft(fidd,tline) ;
I cannot guess what this is! There is a typing mistake in the name of the function.
per isakson
on 12 Jul 2012
"06:39:00" appears three times. Does that mean anything?
What rule would you propose for removing rows?
per isakson
on 12 Jul 2012
Edited: per isakson
on 12 Jul 2012
What rule would you propose for removing rows?
.
This line should not have cause an error. It should have been removed by existing code. Thus, there is a problem.
06:59:00 ** ---- ---.--- 01** 0000 0000 00
in File: 01-01-2010.dat
Soni huu
on 12 Jul 2012
Soni huu
on 12 Jul 2012
per isakson
on 12 Jul 2012
Edited: per isakson
on 12 Jul 2012
Why is these lines in error
08:10:00 .000 000.430 01** 4862 0057 0058 +21
08:11:00 .000 000.430 01** 4862 0057 0058 +22
I cannot see anything wrong in these lines!
.
"but the code still continiou to next code(step).. the data still valid?" I do not understand!
.
We have still not done
Fourth step: What do these lines have in common? How can these lines be identified? By what rule?
per isakson
on 12 Jul 2012
Edited: per isakson
on 12 Jul 2012
Try to be more exact in your wording. Do not let me guess so much!
.
Add the following three lines to ReadSoniData.m
magic_length = 47; % ignore lines with length <= magic_length
is_row_too_short = cellfun( @(str) length(str) <= magic_length, cac);
cac( is_row_too_short ) = [];
.
between the two existing lines
cac( isc | iss ) = [];
and
str = transpose( char( cac ) );
.
This is drastic and may cause problems in the future. Do not forget that these lines are in the code.
Soni huu
on 12 Jul 2012
Soni huu
on 13 Jul 2012
per isakson
on 13 Jul 2012
I cannot guess what is going on. I cannot see that anything in ReadOneSoniData.m could give that error message.
- Put a break point at the first line of ReadOneSoniData
- Start ReadManySoniData the same way that gave the error
- Step one line at a time
- Try to understand what happens; make notes
Which is the line of ReadOneSoniData that causes the error?
per isakson
on 13 Jul 2012
In ReadManySoniData replace
disp( lasterr )
by
le = lasterror;
disp( le.stack(1) )
disp( le.message )
That should provide a better message.
.
Run the function
>> rd = ReadManySoniData( 'C:\matlab7\work\org\2010', '12-31-2010.dat' )
per isakson
on 13 Jul 2012
This is another error!
- Put a break point on the first line of ReadManySoniData
- Step one line at a time
- For every step note the value of the variable, folder_name
per isakson
on 13 Jul 2012
Why do you think that "this code ignore the file name" when the message says:
??? Input argument "folder_name" is undefined.
? .
What do you mean by "file name <= 47)"?
Soni huu
on 13 Jul 2012
per isakson
on 13 Jul 2012
Edited: per isakson
on 13 Jul 2012
- Put a break point at the line
le = lasterror;
- Step one line
- Check carefully all the fields of the structure, le
I cannot test because I run R2012a. In the command window run
>> le=lasterror
le =
message: 'Undefined function or variable 'lastwarning'.'
identifier: 'MATLAB:UndefinedFunction'
stack: [0x1 struct]
Do you see other fields?
Soni huu
on 13 Jul 2012
Soni huu
on 13 Jul 2012
per isakson
on 13 Jul 2012
This looks like a Matlab problem. Try
>> soni
Undefined function or variable 'soni'.
>> le=lasterror
le =
message: 'Undefined function or variable 'soni'.'
identifier: 'MATLAB:UndefinedFunction'
stack: [0x1 struct]
>>
per isakson
on 13 Jul 2012
In ReadManySoniData replace
disp( le.stack(1) )
disp( le.message )
by
disp( le.message )
disp( le.identifier )
disp( le.stack(1) )
per isakson
on 13 Jul 2012
Where does this come from?
??? le =
Have you typed
le = <return>
Soni huu
on 13 Jul 2012
per isakson
on 13 Jul 2012
Edited: per isakson
on 13 Jul 2012
You have an old Matlab version :(
Comment out
% disp( le.stack(1) )
.
The question is: what line causes the error message
Function name must be a string.
Put a copy of the line
RainData = cat( 2, RainData, ReadOneSoniData(folder_name,sa.name) );
before the line
try
Run
>> rd = ReadManySoniData( 'C:\matlab7\work\org\2010', '01-01-2010.dat' );
Soni huu
on 13 Jul 2012
per isakson
on 13 Jul 2012
Run ReadOneSoniData
>> ReadOneSoniData( 'C:\matlab7\work\org\2010', '01-01-2010.dat' )
Soni huu
on 13 Jul 2012
Soni huu
on 13 Jul 2012
per isakson
on 13 Jul 2012
Edited: per isakson
on 13 Jul 2012
In ReadOneSoniData replace
magic_length = 47; % ignore lines with length <= magic_length
is_row_too_short = cellfun( @(str) length(str) <= magic_length, cac);
cac( is_row_too_short ) = [];
by
magic_length = 47; % ignore lines with length <= magic_length
is_too_short = cellfun( 'length', cac ) <= magic_length;
cac( is_too_short ) = [];
and run
>> ReadOneSoniData( 'C:\matlab7\work\org\2010', '01-01-2010.dat' )
if successful run
>> RainData = ReadManySoniData( 'C:\matlab7\work\org\2010', '*.dat' );
per isakson
on 13 Jul 2012
Thus the code works. Do you think the value
DailyRain: 0.2864
is correct? Open the file, 01-01-2010.dat, in an editor and check.
Soni huu
on 13 Jul 2012
per isakson
on 13 Jul 2012
Do you have problem reading the file, '12-19-2010.dat'?
I have successfully red the 2011 data, which I downloaded some days ago. However, I will not download more data. I've run out of time.
Add these lines
fid = fopen( 'c:\temp\ReadRainDataFailures.log', 'a' );
if fid >= 3
fprintf(fid, '%s | %s | %s\n',datestr(now),folder_name, sa.name );
fclose( fid );
end
after
% disp( le.stack(1) )
That will give you a list of the files, which cannot be red. You might want to change the name and folder of the file. Do you have a "c:\temp"?
If you run into specific problems try make a question at Answers.
Soni huu
on 13 Jul 2012
More Answers (1)
per isakson
on 6 Jul 2012
Here is a function that collects total daily rain for one month at a time. Try
>> [ day_number, daily_rain ] = DailyRain( RainData, 2011, 11 );
>> plot( day_number, daily_rain )
>> figure, plot( day_number, daily_rain, '.' )
>> figure, plot( day_number, '.' )
As is and all that! You must check the the values. Missing data might cause surprises.
function [ day_number, daily_rain ] = DailyRain( RainData, year_number, month_number )
day_number = [ RainData(:).DayNumber ];
is_yy = ( year( day_number ) == year_number );
is_mm = ( month( day_number ) == month_number );
is_ym = ( is_yy & is_mm );
if any( is_ym )
daily_rain = [ RainData( is_ym ).DailyRain ];
day_number = [ RainData( is_ym ).DayNumber ];
else
daily_rain = [ ];
day_number = [ ];
end
end
32 Comments
Soni huu
on 6 Jul 2012
per isakson
on 6 Jul 2012
Regarding "y= 0 to 3000" either
- the data sum up to something close to 3000 or
- an error in the code
Reagrding (7.348 to 73484) x10^5
- some hours ago you used datetick, but not here
Soni huu
on 6 Jul 2012
per isakson
on 6 Jul 2012
Try
hist( [ RainData.DailyRain ], 100 )
[ mx, ixm ] = max( [ RainData.DailyRain ] );
plot( datenum( RainData( ixm ).Datevec ), RainData( ixm ).Rain )
datetick
title( RainData(ixm).DataFile )
Soni huu
on 6 Jul 2012
per isakson
on 6 Jul 2012
Edited: per isakson
on 6 Jul 2012
Try
is_huge = ( [ RainData.DailyRain ] >= 2500 );
RainData(is_huge).DataFile
total rain of eight different files exceeds 2500. Is there a problem with units? You must check the code!
Soni huu
on 6 Jul 2012
per isakson
on 6 Jul 2012
Remember the function, EscapeBackSlash?
Soni huu
on 6 Jul 2012
per isakson
on 6 Jul 2012
Edited: per isakson
on 6 Jul 2012
Try
>> doc datetick
.
I'm lost! You know what the numbers mean. I don't!
per isakson
on 6 Jul 2012
"yes, i still save"
save what?
per isakson
on 6 Jul 2012
Edited: per isakson
on 6 Jul 2012
My points were
- title( EscapeBackSlash( RainData(ixm).DataFile ) ) to avoid the warning and
- you shouldn't rely that much on me; you must try harder to understand what the functions do
per isakson
on 7 Jul 2012
Edited: per isakson
on 7 Jul 2012
R2012a and I guess R2012b are improved in many respects compared to the 2004 version. The 64 bit version of Matlab can handle more memory, e.g larger arrays.
I'm happy you say that!
"why y= 0 to 3000 ": Did you figure out what unit is used for the rain in column three?
per isakson
on 7 Jul 2012
It's correct. However, an alternative is
rain_data.Rainrate = cac{3}; % [mm/h]
rain_data.DailyRain = 24*mean( rain_data.Rain ); % [mm]
Pros:
- this is easier to understand. The average rain rate [mm/h] times 24 hour per day, which gives [mm].
- this will handle missing data somewhat better. Missing data will be replace by the average of available data.
- this will work with a data file with other sampling rates, e.g. 2-minute data
- the confusing quantity, RainData.Rain [mm/minute], is not visible outside the function.
Soni huu
on 7 Jul 2012
per isakson
on 7 Jul 2012
Assume the rain_rate varies as a sinus plus some noise. Sample it 1,2,4,8 minutes and estimate daily total. Try
tt = transpose( 1 : 1 : 60 * 24 );
rain_rate_1_min = max( 0, 7*(1+sin(8*pi*tt/(60*24))) + 4*randn(60*24,1) );
rain_rate_2_min = rain_rate_1_min( 1 : 2 : end );
rain_rate_4_min = rain_rate_1_min( 1 : 4 : end );
rain_rate_8_min = rain_rate_1_min( 1 : 8 : end );
plot( rain_rate_1_min )
total = [ 24 * mean( rain_rate_1_min )
24 * mean( rain_rate_2_min )
24 * mean( rain_rate_4_min )
24 * mean( rain_rate_8_min ) ]
The four values are close. Now, you make an analytic prof (paper and pencil).
per isakson
on 9 Jul 2012
Edited: per isakson
on 9 Jul 2012
I guess a dat-file in 'C:\matlab7\work\org\ 2010 ' contains a line that causes this error.
There is a space after "2010" - mistake?
Hint: "(row 543, field 9)"
Soni huu
on 9 Jul 2012
per isakson
on 9 Jul 2012
dbstop if error
Soni huu
on 9 Jul 2012
Soni huu
on 9 Jul 2012
per isakson
on 10 Jul 2012
Edited: per isakson
on 10 Jul 2012
First step: In ReadManySoniData replace
RainData = cat( 2, RainData, ReadOneSoniData( folder_name, sa.name )
by
try
RainData = cat( 2, RainData, ReadOneSoniData(folder_name,sa.name)
catch
fprintf( 'Folder: %s\nFile: %s\n', foldername, sa.name )
disp( lasterr )
end
.
Second step: Run ReadManySoniData for all data files you have. That will give you a list of files, which cause troubles, in the command window.
Third step: Inspect the files (causing trouble) with an editor. Make a list with the lines, which you think are the cause of the trouble.
Fourth step: What do these lines have in common? How can these lines be identified? By what rule?
Soni huu
on 12 Jul 2012
per isakson
on 12 Jul 2012
Edited: per isakson
on 13 Jul 2012
You know - not me - whats on line: 5 column: 76. Fix the line! I guess it is a typing error.
Make sure the Code Analyzer box is green before you run the code.
Soni huu
on 24 Aug 2012
per isakson
on 24 Aug 2012
Soni, good to hear that you succeeded to read the data
Categories
Find more on Tall Arrays in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
