Main Content

Apply Labels to Unreal Scene Elements for Semantic Segmentation and Object Detection

The Simulation 3D Camera block provides an option to output semantic segmentation data from a scene. If you add new scene elements, or assets (such as traffic signs or roads), to a custom scene, then in the Unreal® Editor, you must apply the correct ID to that element. This ID is known as a stencil ID. Without the correct stencil ID applied, the Simulation 3D Camera block does not recognize the scene element and does not display semantic segmentation data for it.

For example, this To Video Display window shows a stop sign that was added to a custom scene. The Semantic Segmentation Display window does not display the stop sign, because the stop sign is missing a stencil ID.

To Video Display window with stop sign displayed and Semantic Segmentation Display window with stop sign not displayed

You can also use these stencil ID labels to assign IDs to objects detected by probabilistic sensors, such as those modeled by the Simulation 3D Probabilistic Radar and Simulation 3D Vision Detection Generator blocks.

To apply a stencil ID label to a scene element, follow these steps:

  1. Open the Unreal Editor from a Simulink® model that is configured to simulate in the 3D environment. For more details, see Customize Scenes Using Simulink and Unreal Editor.

  2. In the editor window, select the scene element with the missing stencil ID.

  3. On the Details pane on the right, in the Rendering section, select Render CustomDepth Pass.

    If you do not see this option, click the Show Advanced expander to show all rendering options.

  4. In the CustomDepth Stencil Value box, enter the stencil ID that corresponds to the asset. If you are adding an asset to a scene from the Automated Driving Toolbox™ Interface for Unreal Engine® Projects support package, then enter the stencil ID corresponding to that asset type, as shown in the table. If you are adding assets other than the ones shown, then you can assign them to unused IDs. If you do not assign a stencil ID to an asset, then the Unreal Editor assigns that asset an ID of 0.

    Note

    The Simulation 3D Camera block does not support the output of semantic segmentation data for lane markings. Even if you assign a stencil ID to lane markings, the block ignores this setting.

    IDType
    0

    None/default

    1

    Building

    2

    Not used

    3

    Other

    4

    Pedestrians

    5

    Pole

    6

    Lane Markings

    7

    Road

    8

    Sidewalk

    9

    Vegetation

    10

    Vehicle

    11

    Not used

    12

    Generic traffic sign

    13

    Stop sign

    14

    Yield sign

    15

    Speed limit sign

    16

    Weight limit sign

    17-18

    Not used

    19

    Left and right arrow warning sign

    20

    Left chevron warning sign

    21

    Right chevron warning sign

    22

    Not used

    23

    Right one-way sign

    24

    Not used

    25

    School bus only sign

    26-38

    Not used

    39

    Crosswalk sign

    40

    Not used

    41

    Traffic signal

    42

    Curve right warning sign

    43

    Curve left warning sign

    44

    Up right arrow warning sign

    45-47

    Not used

    48

    Railroad crossing sign

    49

    Street sign

    50

    Roundabout warning sign

    51

    Fire hydrant

    52

    Exit sign

    53

    Bike lane sign

    54-56

    Not used

    57

    Sky

    58

    Curb

    59

    Flyover ramp

    60

    Road guard rail

    61Bicyclist
    62-66

    Not used

    67

    Deer

    68-70

    Not used

    71

    Barricade

    72

    Motorcycle

    73-255

    Not used

    For example, for a stop sign that is missing a stencil ID, enter 13.

    Tip

    If you are adding stencil ID for scene elements of the same type, you can copy (Ctrl+C) and paste (Ctrl+V) the element with the added stencil ID. The copied scene element includes the stencil ID.

  5. Visually verify that the correct stencil ID shows by using the custom stencil view. In the top-left corner of the editor window, click Lit button and select Buffer Visualization > Custom Stencil. The scene displays the stencil IDs specified for each scene element. For example, if you added the correct stencil ID to a stop sign (13) then the editor window, the stop sign displays a stencil ID value of 13.

    Stop sign with stencil ID 13

    • If you did not set a stencil ID value for a scene element, then the element appears in black and displays no stencil ID.

    • If you did not select CustomDepth Stencil Value, then the scene element does not appear at all in this view.

  6. Turn off the custom stencil ID view. In the top-left corner of the editor window, click Buffer Visualization and then select Lit.

To display semantic segmentation data for your custom labels, follow these steps:

  1. Set up your Simulink model to display semantic segmentation data from a Simulation 3D Camera block. For an example setup, see Depth and Semantic Segmentation Visualization Using Unreal Engine Simulation.

  2. Run the simulation and verify that the Simulation 3D Camera block outputs the correct data. For example, here is the Semantic Segmentation Display window with the correct stencil ID applied to a stop sign.

    Semantic Segmentation Display window with stop sign displayed

See Also

| | | |

Related Topics