How to convert bitstream to packets and bitxor function

I have to convert a bit stream file such as name `BL.264` source file . Now I want to parse the file into packets with packet size is 50 bytes. After that, I want to implement bit XOR between first packet and second packet. To do it, I will use `fread` and `bitxor` functions. However, I have some issues:

1. If size of file is not divisible by packet sizes, we must add some padding at the end of final packet, *Right?* Let see my implementation, is it correct?
2. The bitxor supports xor bitwise that have limit length. In my case, packet size is 50 bytes, it is impossible to bitxor two packet. How to resolve that problem.

Let see my code. If have any problem, let me know

function packet=bitstream2packet(file,psize)
      %%file='BL.264';
      %%psize=50;
      fid=fopen(file,'r');
      bytelist=fread(fid,'*uint8')';    
      bitexpanded = dec2bin(bytelist(:), 8) - '0';     
      bitstream = reshape( bitexpanded.', [],1);
      %%Add padding
      remain=rem(size(bitstream,1),psize);
      bitstream(end+psize-remain)=0;
      packet = reshape( bitstream.', [],50);
      xorPacket=bitxor(packet(1,:),packet(2,:))
      fclose(fid);
end

Answer 1

[Guillaume]

You're storing your bitstream as a logical array. In that case, it's a logical operation that you want to perform, not a bitwise xor. Therefore use xor instead of bitxor.

Compare

n1 = 8; n2=32
n1logicalbit = dec2bin(n1, 8) - '0';  %gives [0 0 0 0 1 0 0 0]
n2logicalbit = dec2bin(n2, 8) - '0';  %gives [0 0 1 0 0 0 0 0]
logicalxor = xor(n1logicalbit, n2logicalbit); %gives [0 0 1 0 1 0 0 0]
binaryxor = bitxor(n1, n2); %gives 40
isequal(logicalxor, dec2bin(binaryxor, 8)-'0') %return true

Otherwise your code is fine. There's a few unnecessary transpositions, you could simply do:

bytelist = fread(fid, '*uint8'); %no transpose
bitexpanded = dec2bin(bytelist, 8) -'0'; %no colon which in your case just transposed

I would also move the fclose to just after you've done the fread. This way if your code stops further down because of an error or the user interrupting it, the file won't be left open.

[Toan]

Thank Guillaume. I modified. It work well. However, I want to recovery these packet to orginal file (BL.264). This is my modification code for bitstream2packet. And second is packet2file. Let look at the second code and let me know is it correct?

function [packet remain]=bitstream2packet(file,psize)  
    fid=fopen(file,'r');
    bytelist=fread(fid,'*uint8');   
    fclose(fid);
    bitexpanded = dec2bin(bytelist, 8) - '0';   
    bitstream = reshape( bitexpanded.', [],1);
    %%Add padding
     remain=rem(size(bitstream,1),psize);%%bytes
    bitstream(end+psize-remain)=0;
    packet = reshape( bitstream.', [],psize);          
end

The second code to write packet to file is implemented by two steps:

1. Remove the padding at the end of last packet
2. Convert packet to bin array and use bin2dec function

function packet2file(file_out, bitstream,psize,remain)
        %%Write to file
        fwid=fopen(file_out,'w');
        bitstream_re = reshape( bitstream, [],1);
        bitstream_re(end-(psize-remain)+1:end)=[];%%Remove padding
        bitstream_8re=reshape(permute(reshape(bitstream_re,size(bitstream_re,2),8,[]),[2 1 3]),8,[]);
        bitstream_8re=bitstream_8re';
        dPacket = {};
        errorPacket=ones(1,8)*-1;
        for i_packet=1:length(bitstream_8re)
            if isequal(bitstream_8re(i_packet,:),errorPacket)
                fwrite(fwid,'NULL');
                continue;
            end
            dPacket =  bin2dec( sprintf('%d',bitstream_8re(i_packet,:)'));             
            fwrite(fwid, char(dPacket));        
        end
        fclose(fwid);
end

It works. However, it take long time due to scanning all packet and write. Besides, I my recover packet includes error packet denote by [-1 -1...]. Hence, I will write NULL instead of these pakets. Is it no problem when I want to compute PSRN between orignal file and recovery file?

[Guillaume]

I'm not sure what you're doing in your packet2file, but it looks overly complicated.

Once you've removed the padding, I would reshape to a n*8 matrix, convert it to char (by adding '0', the same way you subtracted '0' to go from char to binary) and call bin2dec. At this point you can replace your error packet by whatever you want:

bitstream_8re = reshape(bitstream_re, [], 8);
bitstream_char = char(bitstream_8re + '0');
bitstream_byte = bin2dec(bitstream_char);

I must say I don't particularly understand your error packet. How can it be [-1 -1 -1 -1 -1 -1 -1 -1] since bits are either 0 or 1?

errorpacket = '11111111'; %or whatever
errorrows = any(bsxfun(@eq, bitstream_char, errorpacket), 2);%find rows with errorpacket
bitstream_byte(errorrows) = 0; %or whatever you want to replace it with.

You can then write your bitstream as bytes. Note that I assume you want to write bytes not chars (which are two bytes each), and certainly not the string 'NULL'. I don't know what PSRN stands for.

fwrite(fwid, bitstream_byte);

[Guillaume]

Note, while your whole approach (converting bits to an array of 0 and 1) works it is very wasteful with memory since you're using a byte to store a bit. I would have taken a completely different approach from the start.

Are you aware that you can read 50 bits at a time from your original file? If you read it that way, then you can use bitxor straight on the array you read:

bit50 = fread(fid, Inf, 'ubit50=>uint64'); %or just 'bit50'
if mod(size(bit50, 1), 2) == 1 %make sure there's an even number of rows
   bit50 = [bit50; 0];
end
bit50xor = bitxor(bit50(1:2:end), bit50(2:2:end));

You can rewrite the file the exact same way

fwrite(fwid, bit50xor, 'ubit50');

That way, you avoid the conversions to/from char and to/from arrays of 0 and 1 and operates directly on 50-bits numbers

[Toan]

Great idea. Right, I assume that my file is bit stream file, so it only have value whether 0 or 1. And I denote the error bit is -1. When I read the paper, the author said that he divide the video into packet, in which each packet is have 50bytes that means 400bits for each packet. Each packet can combine together by bitxor. For example packet1 xor packet2 xor packet3...so on. Hence, I store them under a matrix where each row of matrix is a packet. Hence, I am very easy to implement bitxor between rows of matrix. Unfortunately, this way is inefficient due to memory will increase 8 times. With your suggestion, it is very good point. However, I don't know it can support for my scheme. Because of with your way, I must read file again to get the packet each iteration, right?

[Guillaume]

By definition a bit only has two values. It's either 0 or 1. You could have an error byte of -1 assuming signed bytes, but not a bit.

My scheme

bit50 = fread(fid, Inf, 'ubit50=>uint64');

will read the whole file and split it in packets of 50 bits each, where each element of bit50 is one of those packets. You only need to read the whole file once. As a bonus, it also automatically pads the last packet to 50 bits for you.

[Toan]

Thank Guillaume. I am very impressive with your scheme. However, my packet size is 50bytes=400bits (8 x 50) , instead of 50 bits as your code. I try to modify it by

   bit400 = fread(fid, Inf, 'ubit400=>uint64');

It has error

Error using fread
Invalid precision.

How to use your scheme in this case. BTW, I need to convert the bit400 to binary representation, so I will use

bitexpanded = dec2bin(bit400(:), 8) - '0';

Is it correct? The reason why I must convert to binary representation is that my algorithm is developed for bit level. Thank you

[Guillaume]

Now, I'm very confused. your original code split the file in packets of 50 bits, not bytes.

If you're operating on bytes, then you don't need to read the file in bits unit, just read it as a 50*Inf array of bytes:

byte50 = fread(fid, [50 Inf], '*uint8')'; %again it will automatically pad the last packet

Even if you're operating on bits afterward, you probably don't need to convert to binary. Certainly not for xoring packets. The bit* functions should be enough.