processing on a live webcam streaming video???

I am using an image acquisition toolbox to obtain the live stream from my webcam. I have my own code for skin color segmentation which works well on pictures (JPEG etc). I am only able to get a snapshot of the live video and then apply the code. I need to apply that code on the live webcam video stream. How can i do it???

[David Tarkowski]

I'm not sure what you mean, a video stream is just a series of snapshots at a specified interval. Can you elaborate on what you are doing and how it differs from what you would like?

Answer 1

[Harsha Vardhan Rao Avunoori]

Well I think you can try using the Image Acquisition ToolBox for your case it will definitely help you...

For more information on Image Acquisition ToolBox have a look at this website

http://www.mathworks.com/products/imaq/

Answer 2

[Hashem Burki]

what i ment was that i want to perform a skin segmentation code on an input webcam video, and want to preview it as an output video. how can i do it?

[Sean de Wolski]

By developing/locating an algorithm to segment skin and applying it to the video.

[Walter Roberson]

You request a single image from the video stream, process that single image and send it to the output video stream. Then you go back and ask for another image from the video stream and so on.

MATLAB itself does not have any way to work with video except as a series of images: one can only hope that one is able to do the processing quickly.

You might find the information in the following Question to be useful:

http://www.mathworks.com/matlabcentral/answers/3858-simultaneous-recording-and-reprodction-of-a-file-in-matlab

[arun joshi]

will the output video be as smooth as a live stream. i tried capturing live feed images from the camera but the processed output seems to lag as the frames are being skipped during processing. By the time it gets to the next frame after processing one frame, a number of frames from the live feed are missed out resulting in a jerky video output. is there a way to solve this?

[Walter Roberson]

"will the output video be as smooth as a live stream"

As we were discussing in https://www.mathworks.com/matlabcentral/answers/7202-processing-on-a-live-webcam-streaming-video#comment_579179, you are using the Automated Driving toolbox. You are limited by the functions in that toolbox that you call. This is probably not an issue of the webcam interface being slow: this is quite likely an issue that you are doing extensive computation with each frame.

No computer can respond instantly. If you want a frame to be ready to output within 1/30 of a second so that you can get 30 fps output, then you need to restrict yourself to computations that that computer can finish within 1/30 of a second. It does not matter whether the camera interface takes 30 microseconds per frame or 30 milliseconds per frame (33 1/3 fps) if the computation you are doing takes (say) 1/4 second. That is true whether that (say) 1/4 second per frame is because you are using slow hardware; or because you are trying to use more RAM than you have available and the program is swapping; or because you wrote inefficient code such as failing to pre-initialize arrays that you are storing into; or because you are just plain trying to compute things more complex than current computer speeds could reasonably get done in that length of time.

The Automated Driving Toolbox is a premium toolbox; very few of the regular volunteers here have access to even the documentation for it, so it is difficult for us to make suggestions (especially since you do not include any code.)

Answer 3

[Florian Morsch]

You way to do it would be:

1: Create a webcam object EDIT: or create a cam object

2: You can get screenshots/snapshots from the object

3: Create a videoplayer object

4: Now you run a while loop and take a snapshot, process it and give it to the videoPlayer as output, which can step through the frames he gets. The FPS is limited to your code, if you need 1 second to work with a frame then you will get 1 FPS, if your code is fast enough you might end up with 10-15 FPS or more.

Now its up to you how fast your code works to get a smoother "video". RealTime is mostly not possible since you need to do some operations on the frame which means you have processing time (except your code is really really good and you can manage to do the processing in like under 20ms, which would be 50 FPS).

[arun joshi]

Is the solution valid for camera apart from a webcam?. I am using pointgrey camera with 30fps as the camera input but I am facing significant lag.

[Florian Morsch]

As long as you can use the camera in matlab you can create a camera object. I just used the webcam as example, you can use videoinput in the same way ( https://de.mathworks.com/help/imaq/videoinput.html ).

When do you face those laggs? When you monitor the input or while/after you work with your code?

[arun joshi]

i face the lags after the code execution. i noticed that the camera can send in input at every 0.03 sec as it is running at 30fps. while loop code execution takes approx 0.4 sec. that means approximately 2 frames per second. Is there any way to process the image at every 0.03 sec instead of missing those frames between 0.03 and 0.4 seconds so that i can get a smoother video output even though it might lag from the real time video which i can put into consideration as i move forward.

[Walter Roberson]

You are asking for your own skin color segmentation code to run in less than 0.03 seconds. However, we have not seen your code and have no idea what operations you are doing.

[arun joshi]

I am not running the process for skin colour segmentation. My code purpose is road obstacle detection based on live video input. I use automated driving system toolbox to achieve this but the process is facing the issue of lag

[Walter Roberson]

Ah, sorry, I did not look closely enough at who had posted which information. You did not appear to mention your code purpose before this.

I have no information about the potential efficiency of the Automated Driving System. That is an expensive toolbox that I do not have access to (or experience with.)

[arun joshi]

Do you happen to come across any such problem? I would like to have some help in this. Also I read parallel computing can increase the process speed. I was wondering if it's been used on a similar situation involving computer vision and image acquisition.

[Walter Roberson]

Generally speaking, when there is only one camera and it is being handled at traditional movie speeds (e.g., 20-30 fps or so), then unless the images are rather larger than usual, then it is usually more efficient to do the image acquisition in the same worker as the data processing.

Using parallel workers for image acquisition is more associated with handling multiple cameras at higher frame rates or larger image sizes, especially in the case where each device has a separate data channel. For example, it might make sense to run two workers, each acquiring data from a fast camera that is on its own USB3 controller. If you have multiple input devices on the same controller, then typically they would be in resource contention with each other and it would be less likely you would use multiple workers.

If you were using a pair of stereo cameras each at 33 fps and moderate resolution (say 768 x 1024), then you would probably run them on the same session instead of different workers.

The issue here is that the more often that a device has data read or the more data it needs to transfer, then the more you need a worker dedicated to it -- but that there is a cost and latency in transferring data between workers, so it isn't worth doing for low rate / low size.

The Automated Driving toolbox is a "premium" toolbox. With my license, I cannot even read the documentation.

I am more acquainted with the facilities of the Computer Vision toolbox, including its object detectors -- though I have never run timing tests on those.

[arun joshi]

Can the frames be stored in the memory and processed as the images are being captured in real time so that all the frames are processed?

[Walter Roberson]

For reasonable frame rates, you could probably store all of the frames in memory as they arrive -- at least until your RAM filled up.

However, storing the frames in memory is not the challenge for you. The challenge for you is getting the processing time for any one frame down to less than the interval between frames. If you were reading frames in at (say) 10 frames per second, but it took (say) 1/5 seconds per frame to do your computation, then if you were to process every frame you would consistently be processing at only half of real-time.

You need to put in a tic() call at the point where you have received a frame and start analyzing it, and a toc() at the point where you have finished analyzing the frame and are ready to display the output. If the time to process one frame exceeds the time interval between frames, then you will fall behind.

The situation to worry about how slow the camera interface is, is the situation where you are able to demonstrate that you can finish analyzing a frame faster than real time, but then find that the display process is holding you up. Ask yourself "If the frame display process were to be reduced to zero by a really fast fast output routine, then is it true that I would be able to process the frames in real time, or is my processing of the frames taking too long?"

[Florian Morsch]

https://de.mathworks.com/matlabcentral/answers/7202-processing-on-a-live-webcam-streaming-video#comment_580680

Refering to your comment: You could store the frames and then analyse every frame, but that would be video processing, not real time processing, if you code is not fast enough for real time in the first place. That would be like taking a video, process every frame and then rewatch it. Sure, you would get no "lag" like in the stream processing, but its not real time anymore, its a video you watch.

Some things can help you to make the code faster (preallocating can reduce the processing time by a lot) see: https://de.mathworks.com/help/matlab/matlab_prog/preallocating-arrays.html