Human behavior recognition from video

I've a trained CNN providing an accuracy of 85%, classifying Humans and vehicles. Now, I need to classify Human behavior like walking,sitting which are Normal, and other like punching and kicking as Abnormal. How to train my network dynamically?

My code is attached below-:

%Load Data
rootFolder = 'C:\New folder\Project\Train\';
categories = {'persons','cars'};
imds = imageDatastore(fullfile(rootFolder, categories), 'LabelSource', 'foldernames');
tbl = countEachLabel(imds);
minSetCount = min(tbl{:,2});
imds = splitEachLabel(imds, minSetCount, 'randomize');
rootFolder = 'C:\New folder\Project\Test\';
imds_test =  imageDatastore(fullfile(rootFolder, categories), 'LabelSource', 'foldernames');
tbl = countEachLabel(imds_test);
minSetCount = min(tbl{:,2});
imds_test = splitEachLabel(imds_test, minSetCount, 'randomize');
%Define Layers
layers = [imageInputLayer([64 64 3]);
          convolution2dLayer(5,20);
          reluLayer();
          convolution2dLayer(5,20);
          reluLayer();
          convolution2dLayer(5,20);
          reluLayer();
          maxPooling2dLayer(2,'Stride',2);
          fullyConnectedLayer(2);
          softmaxLayer();
          classificationLayer()];
    options = trainingOptions('sgdm','MaxEpochs',20,...
  'InitialLearnRate',0.0001);

trainedNet = trainNetwork(imds,layers,options);

YTest = classify(trainedNet,imds_test);
TTest = imds_test.Labels;
accuracy = sum(YTest == TTest)/numel(TTest);

trainFeatures = activations(trainedNet,imds,6);
trainingLabels = imds.Labels;
svm = fitcsvm(trainFeatures ,trainingLabels);
testFeatures = activations(trainedNet,imds_test,6);
testPredictions = predict(svm,testFeatures);
accuracy1 = sum(TTest == testPredictions)/numel(TTest);

Answer 1

[Vivek Akkala]

Although object classification can be part of video classification its not straight forward to use object classification for video classification without additional post processing. Object classification like humans and vehicle need a single image to obtain the class whereas video classification needs a set of frames at regular intervals to analyze the behavior. You can look at Getting Started with Video Classification Using Deep Learning to get started with video classification and look at functions like slowFastVideoClassifier, r2plus1dVideoClassifier, and inflated3dVideoClassifier that might help you to recognize human activity.