The documentation for MAT 1.0 and MAT 2.0 files is lost to history as far as I can tell.
The documentation for MAT 3.0 and MAT 4.0 files is available, but this format is nearly completely obsolete, with many restrictions.
The documentation for MAT 5.0 files can be found or deduced fairly completely. It does not have anything to do with any HDF version.
MAT 5.3 is a small variation on MAT 5.0.
MAT 7.0 is basically the same structure as MAT 5.0 but there are some undocumented extensions to support efficient use of newer object types. It still does not have anything to do with any HDF version.
MAT 7.3 format is a complete redesign, and is not publicly documented. What is documented is that it is a variation on HDF5, and that HDF5 tools can be used to explore parts of it. MAT 7.3 is the first version that can save objects larger than 2.0 Gb.
There is a preference available for the default MAT version to use when you save() without specifying a version to use at the time of the save. Most people have the preference set to MAT 7.0 files, because most objects are not larger than 2 Gb and it is common for MAT 7.3 files to have much slower performance. If you did not specify which release to save with, chances are good that it is not MAT 7.3 and so not HDF5 based.