Anybody w/ new release since R2012b test for resolution of fixed-width field?

http://www.mathworks.com/matlabcentral/answers/120549-extract-value-from-txt-weird-lay-out

I created a Service Request from the subject thread--I just noticed it was closed with a status of "Resolved" but AFAIK there has been no change actually implemented to fix the issue and it appears based on some recent threads here there are some areas that have been broken that did formerly work...

So, can somebody w/ a recent version try any of the last examples there and see if this has actually been fixed or if TMW just closed it because tired of hearing about it?

Sample file attached, code to test would be

cc=cell2mat(textscan(fid,['%5f' repmat('%6f',1,3)],'delimiter',''))
cc =
13.1000   54.6000   99.0000    0.3300
13.1500   54.6010    0.0020    0.3400
13.2000   54.6000    0.0030         0
13.2500   54.6010   28.0000  -40.1600

The correct array is

    13.1000  54.6000   99.0000    0.33
    13.1500  54.6000  100.0000  200.34
    13.2000  54.6000    0.0000  300.00
    13.2500  54.6000  128.0000  -40.16

[Star Strider]

Not fixed. I get the same result for cc in R2014a.

[dpb]

That's what I figured. I'll petition to reopen and at least get status moved to "Unresolved". I was hoping beyond belief that perhaps the other bugs were introduced in an attempt to fix this one and similar that had previously filed but seem to have completely fallen of the radar as can't find them in the list of service requests, even, any longer. :(

Thanks, S-S...(why, oh why, couldn't TMW have vectorized FORMAT instead???)

[Star Strider]

My pleasure! The fixed-field format problem has been a problem forever. Even BASIC has it, IIRC.

[dpb]

BTW, you did the same thing you accuse me of--putting an answer in comment section so can't be accepted! :)

[Star Strider]

All I did was to verify your result!

[dpb]

But it answered the question... :)

Never mind that Per a little while later but the kibosh on the whole subject!!! :)

Answer 1

[dpb]

Well, talk about timing and coincidence!!! Just a few hours ago another query of the nature was posed at

http://www.mathworks.com/matlabcentral/answers/137606-read-ascii-non-delimited-file

and Per Isakson solved it (or at least provided a workaround) and all other cases like it for all time, apparently!!! The key is one has to give the format specfication as '%w.pf' including the precision as well as the the width as in '%wf' to give the scanner the extra help it needs to not trespass into the initial field of the next value.

Now why this is so remains a mystery, but at least it does get us somewhere on being able to parse a file. As I noted in my comments to Per, I don't know why I never seemed to have thought of doing so explicitly in all these years of complaining (but then again, nobody else pointed it out as a lack, either).

Anyway, again, kudos to Per!!!

[Star Strider]

Does it work on your TEST.TXT file? (Haven’t tried it yet myself.)

I noticed is that the data in the other file was ‘sort of’ delimited by all the values being negative. Its success could be limited to such situations.

[dpb]

Yes; I'd already tried it when I concluded it appears general, not just for a case of a single format.

cc=cell2mat(textscan(fid,['%5.2f' repmat('%6.2f',1,3)],'delimiter',''))

leads to joy...

Again, that I apparently hadn't ever tried this in frustration if nothing else over all these years astounds me, but afaict I must have not ever done so. I need to go back to my old R12 release and see how it behaves there. If this works there I presume it probably has worked since day one and I've surely wasted a tremendous amount of effort over the last 25 yrs or so...

ADDENDUM

And, indeed,

 [a b c d]=textread('test.dat',['%5.2f' repmat('%6.2f',1,3)],'delimiter',''));

does succeed in R12 as well ( textscan doesn't yet exist therein)...

[Star Strider]

This should be part of the Examples section of the textscan documentation, specifically denoting it as a way to read FORTRAN-type fixed-field files.

[dpb]

"Great minds" and all that, S-S... :)

I just submitted a comment/request as an addendum to the previous service request that suggested almost identically the same thing as the recommended "fix". I think there should be a more global link somewhere besides just textscan that addresses fixed-width field parsing, but if it were an example therein that would at least be something.

I would still like to learn whether this is consistent with C Standard behavior or if it is inconsistent with C or of it's not specifically addressed by the Standard. So far I've had no success at all in trying to determine that.

Answer 2

[per isakson]

Another test without the precision specifier (induced by a comment by dpb in the other post) (R2013a)

    N = 5;
    format_spec = '%5f%5f%6f%6f';
    fid = fopen( 'poi_fixed_format.txt', 'r' );
    cac = textscan( fid, format_spec, 'CollectOutput',true );
    fclose( fid );
    celldisp( cac )
    [a,b,c,d]=textread('poi_fixed_format.txt',format_spec, 'delimiter','');
    TR = [a,b,c,d]
    fid = fopen( 'poi_fixed_format.txt', 'r' );
    M = fscanf( fid, format_spec );
    fclose( fid );
    FS = transpose( reshape( M, [4,4] ) )

outputs

    cac{1} =
       13.1000   54.6000   99.0000    3.4500
       13.1500   54.6000  111.0000  123.4500
       13.2000   54.6000   11.0010   23.4500
       13.2500   54.6000    1.0012    3.4500
    TR =
       13.1000   54.6000   99.0000    3.4500
       13.1500   54.6000  111.0000  123.4500
       13.2000   54.6000   11.0010   23.4500
       13.2500   54.6000    1.0012    3.4500
    FS =
       13.1000   54.6000   99.0000    3.4500
       13.1500   54.6000  111.0000  123.4500
       13.2000   54.6000   11.0010   23.4500
       13.2500   54.6000    1.0012    3.4500

where poi_fixed_format.txt contains

    13.10 54.60 99.00  3.45
    13.15 54.60111.00123.45
    13.20 54.60 11.00123.45
    13.25 54.60  1.00123.45

with "1" in position one; no leading spaces

.

Tentative conclusion:

To me it looks as if

• leading spaces of "fields" are skipped; not counted to make up the width
• [+- ] and new-line are interpreted as start of new field; overruns width

This is consistent with both

• scanf Width Specification on C at MSDN Library and
• 4.9.6.2 The fscanf function of http://flash-gordon.me.uk/ansi.c.txt.

[dpb]

I've been trying to convince TMW of that for 25 yr...no success as of yet. :(

ADDENDUM

As I noted in one of the aforementioned referenced earlier threads, I've essentially given up ever getting TMW to see the light on fixing the C-based versions after all these years.

At one time I had built a Fortran mex function that could handle most "ordinary" cases via taking in a FORMAT string and returning the result to Matlab by using Fortran READ on external file or internal file if passed a character buffer. I provided this to TMW gratis some years ago in hope they would pick up the ball and run but that hasn't ever happened, either. Unfortunately, I seem to have lost the source code for it when made the move back to the farm and I've not had the ambition to redevelop it since.

My current stance is it's is still a major gap in Matlab functionality that should be fixed in one of those two ways, but seem to be mostly a voice in the wilderness...others adding their voices to enhancement requests and/or bug reports can only help but at this point it seems unlikely they'll start paying any attention now, but then again, I guess miracles are still possible.

[dpb]

This is consistent with scanf Width Specification on C at MSDN Library.

Hmmmm....that's the most readable description I've seen Per and somehow hadn't come across that particular version of the MS documentation.

It sorta' reaffirms the conclusion I had come to that it did have to do with the definition of what the width field really means in the C scanf specification. It is simply a perverted and broken definition for fixed-width fields but is "just the way it is" in C. Hence my final (reluctant) secondary conclusion it was fruitless to hope TMW would ever fix it within Matlab since its routines come from (and eventually one presumes, call the library in) C.

But, that doesn't preclude them from supporting a Fortran-like FORMAT syntax but my tertiary conclusion has been they don't care and so likely never will do so.

[per isakson]

Now I read the CSSM-thread, fscanf / sscanf / textscan with fixed format and realize this is an old topic.

My conclusions

• TMW is reluctant to point out deficiencies of Matlab, which explains why reading Fortran-fixed-format text files is not discussed in the documentation.
• TMW will not add a new function to support "Fortran-like FORMAT syntax" since fixed-format files are becoming more rare over time. Why now when TMW didn't do it long ago?
• Reading Fortran-fixed-format text files causes a small(?) number of users trouble and wasted time.
• There is no function to read Fortran-fixed-format files in the File Exchange.

dpb, did you find your "set of utilities that used Fortran mex-files"? They would be make a good FEX-contribution.

[dpb]

Unfortunately, since it's been 25 yr or more since I first started the campaign with TMW, it's pretty clear they have no intention of doing so now despite that fixed-format files are never going to disappear entirely. I'm now past using Matlab (or any other computing engine, either, for that matter) in anger so it's just no longer worth the effort to keep up the good fight.

And, no, as mentioned somewhere above, I apparently irretrievably lost the mex-file stuff in the move from TN back to the farm as apparently I had only an extant copy on the workstation at the TVA offices where the EPRI I&C Center was housed and I didn't realize it at the time that it wasn't also on the personal machines that I brought. By the time I did miss it, the I&C Center had been closed also and the machines there re-purposed or scrapped so there was no chance to retrieve it, unfortunately.