Simulation of Camera Inputs

I am trying to simulate a robotic platform which has a depth camera used to track the trajectory of an object being thrown. I have tried simulating this already by making a few functions which simulates and plots a ball being thrown, but the axes option for simulating what is seen from a particular point of view (especially with the correct field of view that the real camera would have) is very finnicky. Ideally, I would have a camera whose position, direction, and field of view I can chose, so that I can take the frames it sees as inputs for other functions that would process the images and determine where the ball will land. Is there a more clear cut way to do this (maybe with simscape)?

Answer 1

[Shreshth]

Hello Philippe,

I can understand that you are trying to simulate a robotic platform with a depth camera to track the trajectory of an object, like a ball being thrown. This involves several steps, including simulating the object's motion, the camera's perception of this motion, and the processing of this perception to predict outcomes such as landing points.

While MATLAB offers several tools and toolboxes for such simulations, including Simscape for physical modeling and the Computer Vision Toolbox for camera simulations and image processing, there isn't a direct, out-of-the-box solution for your specific need. However, you can build a simulation by combining functionalities from these toolboxes.

Step 1: Simulate the Ball’s Trajectory

You can start by simulating the ball's trajectory using basic physics equations for projectile motion or using Simscape for a more physical approach that can include air resistance and other factors.

Step 2: Simulate the Camera

Use the Computer Vision Toolbox to simulate the camera with a specific position, orientation and Field of view (FOV).

Step 3: Generate Frames from the camera’s perspective

Using the simulated camera, you can generate frames that represent what the camera sees at different points in time. This involves projecting the 3D trajectory of the ball onto the 2D plane of the camera's image sensor, considering the camera's position, orientation, and FOV.

Step 4: Process the Frames to Predict the Landing Point

Once you have the frames, you can process them using image processing techniques to track the ball's position over time involving Object detection and Trajectory Estimation.

To understand the working of MATLAB tools involved in the process you can refer to the below mentioned MathWorks documentation for reference:

Simscape :

www.mathworks.com/help/simscape/ug/how-simscape-simulation-works.html

Computer vision Toolbox:

https://in.mathworks.com/help/vision/index.html?lang=en

Hope it helps.