Starburst- based eye tracking algorithm on single images

Hi everybody, I'm trying to process some eye images for eye tracking and I am trying currently with Starburst algorithm. Since the algorithm originally takes videos while I'd like to process single images, I must break down the single functions to do what I want. I am going through the single functions to do so. I am not familiar with the file handling, so this bit of code for me turns out very hard to understand:

[sfname, spname] = uigetfile('*.mp4','Scene movie file');

sf = strcat(spname, sfname);

[efname, epname] = uigetfile('*.mp4','Eye movie file', spname);

ef = strcat(epname, efname);

eval(sprintf('!mkdir %s/Scene', spname));

eval(sprintf('!mkdir %s/Eye', epname));

eval(sprintf('!ffmpeg -i %s -img jpeg %sScene/Scene_%%5d.jpg', sf, spname));

eval(sprintf('!ffmpeg -i %s -img jpeg %sEye/Eye_%%5d.jpg', ef, epname));

Supposedly breaks the video into single images (which would be my starting point). However, the !ffmpeg command doesn't seem to be working, as it raise and error:

The syntax of the command is incorrect. 

'ffmpeg' is not recognized as an internal or external command, 
operable program or batch file.

Can anybody help me out with understanding what this snippet will actually do (I understand what 'eval' and 'sprintf' do but what it's inside there remains obscure) and why it raises and error?

Thank you very much!

Answer 1

[Walter Roberson]

ffmpeg = '/usr/local/bin/ffmpeg';   %full path to executable

[sfname, spname] = uigetfile('*.mp4','Scene movie file');

sf = fullfile(spname, sfname);

[efname, epname] = uigetfile('*.mp4','Eye movie file', spname);

ef = fullfile(epname, efname);

mkdir( fullfile(spnane, 'Scene') );
mkdir( fullfile(epnae, 'Eye') );

cmd1 = sprintf('''%s'' -i ''%s'' -img jpeg ''%s/Scene/Scene_%%5d.jpg''', ffmpeg, sf, spname);
cmd2 = sprintf('''%s'' -i ''%s'' -img jpeg ''%s/Eye/Eye_%%5d.jpg''', ffmpeg, ef, epname);

system(cmd1)
system(cmd2)

[Beatrice Pazzucconi]

Hi, thanks for the quick reply, but this does not really clarify the purpose of this code for me, nor why it raises an error. As I said before, I know very little of file handling, and this is only more difficult to understand for me. What is ffmpeg doing? What does sprintf('''%s'' -i ''%s'' -img jpeg ''%s/Scene/Scene_%%5d.jpg''', ffmpeg, sf, spname) mean? BTW it gives the same error when it comes to system(cmd1):

The syntax of the command is incorrect. 
"'" is not recognized as an internal or external command, 
operable program or batch file
ans = 
   1
end

[Walter Roberson]

You appear to be using code designed for Linux or OS-X, on MS Windows. The external program you are using, ffmpeg, is available for MS Windows, but the file names need to be modified:

ffmpeg = 'C:\Program Files\ffmpeg\ffmpeg.exe';   %adjust this to the actual path to the ffmpeg executable

[sfname, spname] = uigetfile('*.mp4','Scene movie file');

sf = fullfile(spname, sfname);

[efname, epname] = uigetfile('*.mp4','Eye movie file', spname);

ef = fullfile(epname, efname);

mkdir( fullfile(spnane, 'Scene') );
mkdir( fullfile(epname, 'Eye') );

cmd1 = sprintf('"%s" -i "%s" -img jpeg ""%s\\Scene\\Scene_%%5d.jpg"', ffmpeg, sf, spname);
cmd2 = sprintf('"%s" -i "%s" -img jpeg "%s\Eye\Eye_%%5d.jpg"', ffmpeg, ef, epname);

system(cmd1)
system(cmd2)

These commands instruct MATLAB to invoke the external program ffmpeg, and instruct ffmpeg to read in the given mp4 files and to split the movie file up into individual JPEG images.

The frames from the "Scene movie file" are stored in a subdirectory named "Scene" of the same directory that the movie file is in, and each frame is to be stored in a file whose name begins with Scene_ followed by a 5 digit frame number, such as Scene_00191.jpg for frame #191.

The frames from the "Eye movie file" are stored in a subdirectory named "Eye" of the same directory that the movie file is in, and each frame is to be stored in a file whose name begins with Eye_ followed by a 5 digit frame number, such as Eye_00191.jpg for frame #191.

This code requires the external program ffmpeg, which you can download from http://ffmpeg.zeranoe.com/builds/

Chances are that the same task can be done with MATLAB code.

ffmpeg is a good choice for video files that are in obscure formats or that use features of standard formats that systems might tend to get wrong; it is also a good choice if you have a lot of files to convert and the alternatives are too slow.

Image Analyst has posted MATLAB code to split video files, such as at https://www.mathworks.com/matlabcentral/answers/115180-splitting-video-into-frames#answer_123554 . His code should work for any file that is supported by Windows Media Viewer on your system.

[Walter Roberson]

What does sprintf('''%s'' -i ''%s'' -img jpeg ''%s/Scene/Scene_%%5d.jpg''', ffmpeg, sf, spname) mean?

sprintf() is a formatting call, taking in arguments and producing a controlled string output representing the arguments.

The first argument to sprintf() is the format specifier that controls what the output string is to look like. The remaining arguments are the items to be converted.

In this case, the items to be converted are the character vector named ffmpeg, the character vector named sf, and the character vector named spname. ffpmeg was used to store the path to the ffmpeg executable, sf was used to store the full path to the input .mp4 file, and spname was used to store the directory name that the input .mp4 file is stored in.

In the format specifier, the pattern %s means to take the next unused input value, and write it into that position of the output as a character string. When the input is a character vector, that is just copying the input to the output.

The only other thing you need to know at the moment about sprintf is that when it sees two % in a row, such as the %%5d part, then that means it should output a single literal % in place of the two % .

The command

cmd1 = sprintf('''%s'' -i ''%s'' -img jpeg ''%s/Scene/Scene_%%5d.jpg''', ffmpeg, sf, spname)

is therefor equivalent to

s1 = ''''        %that is, a single apostrophe
s2 = ffmpeg;     %content of the variable ffmpeg copied unchanged
s3 = ''' -i '''  %a single apostrophe, followed by space then -i then space then a single apostrophe
s4 = sf;         %content of the variable sf copied unchanged
s5 = ''' -img jpeg ''';   %a single apostrophe, followed by space then -img then space the jpeg then space then a single apostrophe
s6 = spname;     %content of the variable spname copied unchanged
s7 = '/Scene/Scene_%5d.jpg''';  %/Scene/Scene_%5d.jpg followed by a single apostrophe
cmd1 = [s1 s2 s3 s4 s5 s6 s7];   %put them all together in one string

The output of this would look something like,

'C:\Program Files\ffpmeg\ffpmeg.exe' -i 'C:\Users\bpazzucconi\Documents and Files\Starburst\scenes/Scene_elvis3.mp4' -img jpeg 'C:\Users\bpazzucconi\Documents and Files\Starburst\scenes/Scene/Scene_%5d.jpg'

which is valid Unix code that is not valid for MS Windows, so my replacement code would instead output

"C:\Program Files\ffpmeg\ffpmeg.exe" -i "C:\Users\bpazzucconi\Documents and Files\Starburst\scenes\Scene_elvis3.mp4" -img jpeg "C:\Users\bpazzucconi\Documents and Files\Starburst\scenes\Scene\Scene_%5d.jpg"

This is a creating a Windows shell command line. Windows (and Unix) normally take spaces as marking the end of a parameter, but keep together parts that are quoted.

The command produced (the one with quotes) tells the shell to run a command named C:\Program Files\ffpmeg\ffpmeg.exe ; with first input argument -i ; and second input argument C:\Users\bpazzucconi\Documents and Files\Starburst\scenes\Scene_elvis3.mp4 ; and third input argument -img ; and fourth input argument jpeg ; and fifth input argument C:\Users\bpazzucconi\Documents and Files\Starburst\scenes\Scene\Scene_%5d.jpg

The original code that you had posted would have produced an output like

C:\Program Files\ffpmeg\ffpmeg.exe -i C:\Users\bpazzucconi\Documents and Files\Starburst\scenes/Scene_elvis3.mp4 -img jpeg C:\Users\bpazzucconi\Documents and Files\Starburst\scenes\Scene/Scene_%5d.jpg

which Windows would have interpreted as meaning that it was to run a command named C:\Program with first input argument Files\ffpmeg\ffpmeg.exe and second input argument -i ; and third input argument C:\Users\bpazzucconi\Documents ; and fourth input argument and ; and fifth input argument Files\Starburst\scenes/Scene_elvis3.mp4 ; and sixth input argument -img ; and seventh input argument jpeg ; and eighth input argument C:\Users\bpazzucconi\Documents ; and ninth input argument and ; and tenth input argument Files\Starburst\scenes\Scene/Scene_%5d.jpg

You can see from this that adding in apostrophes or double-quotes is crucial when names might include spaces.

[Beatrice Pazzucconi]

Alright, since I'm using Windows platform, I guess I should modify the above command so that it can be understood by the shell: would it be something like this? PS - the syntax for single spostrophe is ''' or '''' because I've seen them both in your code

s1 = "      % double quotes
s2 = ffmpeg;    
s3 = " -i "  % - i between double quotes
s4 = sf;         
s5 = " -img jpeg ";   % a double quote + space + -img + space + jpeg + space + double quote
s6 = spname;    
s7 = /Scene/Scene_%5d.jpg ";  %/Scene/Scene_%5d.jpg + double quotes
cmd1 = [s1 s2 s3 s4 s5 s6 s7];

Now my code looks something like this:

    ffmpeg = 'D:\Programmi\installer\ffmpeg-20171022-72c3d9a-win32-static\ffmpeg.exe';   %full path to executable

[sfname, spname] = uigetfile('*.mp4','Scene movie file');

sf = fullfile(spname, sfname);

[efname, epname] = uigetfile('*.mp4','Eye movie file', spname);

ef = fullfile(epname, efname);

mkdir( fullfile(spname, 'Scene') );
mkdir( fullfile(epname, 'Eye') );

cmd1 = sprintf('"%s" -i "%s" -img jpeg "%s/Scene/Scene_%%5d.jpg"', ffmpeg, sf, spname);
cmd2 = sprintf('"%s" -i "%s" -img jpeg "%s/Eye/Eye_%%5d.jpg"', ffmpeg, ef, epname);

system(cmd1)
system(cmd2)

and still gives an error for the ffmpeg variable:

Unrecognized option 'img'. 
Error splitting the argument list: Option not found

Although if I try to print out the variuos strings, the format should be the one you suggested for Windows shell:

cmd1 =

"D:\Programmi\installer\ffmpeg-20171022-72c3d9a-win32-static\ffmpeg.exe" -i "D:\Chicago\MATLAB\Thesis\small.mp4" -img jpeg "D:\Chicago\MATLAB\Thesis\/Scene/Scene_%5d.jpg"

[Walter Roberson]

It appears that -img jpeg has been replaced by -f image2. It also appears that you should be able to leave that out and it should figure out what to use. So the command should become

cmd1 = sprintf('"%s" -i "%s" ""%s\\Scene\\Scene_%%5d.jpg"', ffmpeg, sf, spname);

[Beatrice Pazzucconi]

Ok, I'll try that out, meanwhile I tries the code from Image Analyst and it looks it works fine for me with some arrangements. Also, I think that using more high level commands would be better as to make my algorithm more platform-independent. Thank you very much for the help!

[Mohy Faid]

Beatrice Pazzucconi , Do you solve this problem ?

[Beatrice Pazzucconi]

Actually in the end I did not use it, because I did not need to break into frames a video. I just used the part relative to file handling

dir_name = uigetdir(pwd,'Select working folder, must be the same     
for algorithm and Eye folder');
mkdir(fullfile(dir_name, 'Results'))
res_name = strcat(dir_name, '/Results/');
eye_file_name = sprintf('%s/Eye/Eye_', dir_name);
results_data_name = sprintf('%s/Results.mat', res_name);
% some image processing happens [...]
% save image (inside for loop, with image handle open and then closed)
save_image(res_name, frame_index);
frame_index = frame_index + 1;    
% save results (out of for loop)
save(results_data_name, 'first_frame', 'last_frame', 'CR_matrix', 'Ellipse_matrix', 'Diff_vector_matrix')

The parameters I save are the variables I get from image processing. I create a folder with the results.mat file in which I save those and the image I processed with some markers applied on to decide if the processing went right. The frame index goes from the first image index found to the last and increase with each cycle. I attach here the save_image function:

function save_image(res_name, frame_index)
% image must be existsing already, so that one can choose if it will be
% visible or not
 F = getframe ;
 res_filename = strcat(res_name, sprintf('Eye_res_%5.5d.jpg', frame_index));
 imwrite(F.cdata, res_filename)
return