classifySequence

Classify video sequence

Since R2021b

collapse all in page

Syntax

label = classifySequence(classifier)
[label,score] = classifySequence(classifier)
[___] = classifySequence(classifier,ExecutionEnvironment=env)

Description

label = classifySequence(classifier) classifies a video sequence using the video classifier classifier. The function returns label, a scalar categorical that specifies the classification of the video or optical flow sequence. label is one of the values of the Classes property of the video classifier object.

example

[label,score] = classifySequence(classifier) additionally returns the classification score associated with the label. The score represents the confidence of the predicted class label, and contains values between 0 and 1.

[___] = classifySequence(classifier,ExecutionEnvironment=env) specifies the hardware resources for running the classifier in addition to any combination of arguments from previous syntaxes, as one of these options:

  • "auto" — Sets the execution environment to the GPU, if available. Otherwise the function sets it to the CPU.

  • "gpu" — Sets the execution environment to the GPU. Usage of the GPU requires Parallel Computing Toolbox™ and a CUDA® enabled NVIDIA® GPU. For information about the supported compute capabilities, see GPU Computing Requirements (Parallel Computing Toolbox).

  • "cpu" — Sets the execution environment to the CPU.

Examples

collapse all

This example uses:

Open Live Script

This example shows how to classify video sequences in a video file using a SlowFast Video Classifier pretrained on the Kinetics-400 video activity recognition dataset. To learn more about how to train a video classifier network for your dataset, see Gesture Recognition using Videos and Deep Learning.

Load SlowFast Video Classifier

sf = slowFastVideoClassifier();

Set Up Video Player and Video Reader

Specify the video file name to stream video frames.

videoFilename = "pushup.mp4";

Create a VideoReader to read video.

reader = VideoReader(videoFilename);

Set up a video player.

player = vision.VideoPlayer;

Classify Video Sequences

Specify how frequently the classifier should be applied to incoming video frames.

classifyInterval = 10;

A value of 10 balances runtime performance against classification performance. Increase this value to improve runtime performance at the cost of missing actions from the video file.

Obtain the sequence length of the SlowFast Video Classifier. Classify only after capturing at least sequenceLength number of frames from the video file.

sequenceLength = sf.InputSize(4);

Read video frames using the hasFrame and readFrame functions of the VideoReader. Using the updateSequence function update the video classifier's sequence. Using the classifySequence function classify the updated sequence.

numFrames = 0;
text = "";

while hasFrame(reader)
    frame = readFrame(reader);
    numFrames = numFrames + 1;

    % Update the sequence with the next video frame.
    sf = updateSequence(sf,frame);

    % Classify the sequence only at every classifyInterval number of frames.
    if mod(numFrames, classifyInterval) == 0 && numFrames >= sequenceLength
        [label,score] = classifySequence(sf);
        text = string(label) + "; " + num2str(score, "%0.2f");
    end
    frame = insertText(frame,[30,30],text,'FontSize',24);
    step(player,frame);
end

This example uses:

Open Live Script

This example shows how to classify a streaming video from a webcam using a pretrained SlowFast Video Classifier. To learn more about how to train a video classifier network for your dataset, see Gesture Recognition using Videos and Deep Learning.

Download Pretrained Video Classifier

Download the pretrained SlowFast video classifier.

downloadFolder = fullfile(tempdir,"gesture");
zipFile = "slowFastPretrained_fourClasses.zip";
if ~isfile(fullfile(downloadFolder,zipFile))
    disp("Downloading the pretrained network...");    
    downloadURL = "https://ssd.mathworks.com/supportfiles/vision/data/" + zipFile;
    zipFile = fullfile(downloadFolder,zipFile);
    websave(zipFile,downloadURL);
    unzip(zipFile,downloadFolder);
end

Load the pretrained SlowFast video classifier.

pretrainedDataFile = fullfile(downloadFolder,"slowFastPretrained_fourClasses.mat");
pretrained = load(pretrainedDataFile);
slowFastClassifier = pretrained.data.slowFast;

Display the class label names of the pretrained video classifier. Any gesture such as "clapping" and "wavingHello" on to the webcam will be recognized by the SlowFast Video Classifier.

classes = slowFastClassifier.Classes
classes = 4×1 categorical
     clapping 
     noAction 
     somethingElse 
     wavingHello

Set Up the Webcam and the Video Player

In this example, a webcam object is used to capture streaming video. A Video Player is used to display the streaming video along with the predicted class.

Create a webcam object using the webcam function.

cam = webcam;

Create a Video Player using vision.VideoPlayer function. Make sure to place the Video Player in a position where you can clearly see the streaming video when running the classification.

player = vision.VideoPlayer;

Classify the Webcam Streaming Video

Specify how frequently the classifier should be applied to incoming video frames. 

classifyInterval = 10;

A value of 10 balances runtime performance against classification performance. Increase this value to improve runtime performance at the cost of missing gestures from the live video stream.

Obtain the sequence length of the SlowFast Video Classifier. Classify only after capturing at least sequenceLength number of frames from the webcam.

sequenceLength = slowFastClassifier.InputSize(4);

Specify the maximum number of frames to capture in a loop using the maxNumFrames variable. Make sure you wave one of your hands to recognize "wavingHello" label, and clap using both your hands for the classifier to recognize "clapping" label.

maxNumFrames = 280;

Capture the webcam snapshot in a loop. Update the streaming video sequence of the classifier using the updateSequence method, and classify the streaming sequence using the classifySequence method.

numFrames = 0;
text = "";

while numFrames <= maxNumFrames
    frame = snapshot(cam);
    
    numFrames = numFrames + 1;
    slowFastClassifier = updateSequence(slowFastClassifier,frame);
    if mod(numFrames, classifyInterval) == 0 && numFrames >= sequenceLength
        [label,scores] = classifySequence(slowFastClassifier);
        if ~isempty(label)
            text = string(label) + "; " + num2str(max(scores), "%0.2f");
        end
    end
    frame = insertText(frame,[30,30],text,'FontSize',18);
    step(player,frame);
end

Input Arguments

collapse all

Classifier, specified as a r2plus1dVideoClassifier or slowFastVideoClassifier object.

Output Arguments

collapse all

Classification of the video or optical flow sequence, returned as a categorical scalar.

Classification score associated with the label, returned as a scalar value between 0 and 1. The score represents the confidence of the predicted class label.

Version History

Introduced in R2021b

See Also

Functions

Objects

Topics