Although in theory RS232 can transfer 10 bit bytes, in practice it is seldom done. Instead the practice is to break anything more than 8 bits up in to 8 bit chunks, padding with bits of 0. As you are programing both ends (I guess), you could choose to transfer 10 bit bytes over RS-232, but other people wanting to interface to your design will not expect 10 bit bytes (and might not know how to program their own software to handle them.)
There are different practices about where the byte breaks are done, and about where the 0's go. When you are making your decision, I recommend you take in to account the ease of extending to greater precision -- if, for example, the 10-bit ADC was replaced with a 12-bit, it would be nice of nothing in the byte protocol changed.
Beyond the above, the commands you need to know are fopen(), fclose(), fread(), fwrite(), set(), and get(), and possibly fprintf() and fscanf(). Of these, chances are that the only one that will "surprise" you is fprintf(), in that one common form of it has a different behavior than when it is used for files. Though I suppose I should say that many people have been surprised by serial port timeouts when using fread() and fscanf(), which is more a matter of using proper set() parameters rather than a real difference in the commands.
