raisim::SegmentationCamera

final class · #include <raisim/sensors/SegmentationSensor.hpp>

Inherits: raisim::Sensor

Segmentation camera: which object, body and semantic label each pixel sees.

It fills up to three images of width * height int32 values, row-major (index = row * width + col), row 0 at the top of the image:

• object image: world index of the object (World::getObject(index), Object::getIndexInWorld()),
• body image: body index within that object (ArticulatedSystem::getBodyNames() order; 0 for single bodies, compounds, terrain and deformables),
• label image: semantic label of that body (Object::setSegmentationLabel(), ArticulatedSystem::setBodySegmentationLabel()); 0 for geometry without a label.

Every image holds kBackground (-1) where nothing was hit within [clipNear, clipFar]. The object and body images hold kNotWorldObject (-2) for rayrai visuals that are not RaiSim objects (drawn only when a renderer is asked to include visualization objects). SegmentationType selects the images: SEMANTIC fills the label image, INSTANCE the object and body images, PANOPTIC all three. The images it does not select are empty.

World indices refer to the world at the measurement time: World::removeObject() moves the last object into the freed index.

Sensor frame convention: +x is the optical axis, +y points left and +z up, as for DepthCamera.

Measurement sources: with MANUAL (the default) a renderer writes the images, e.g. rayrai's RayraiWindow::renderSegmentation() and Camera::getRawImage(), or the rayrai TCP viewer for a RaisimServer. With RAISIM, update() ray-casts the collision geometry of the world (the owning body excluded), as DepthCamera::update() does.

Nested classes

• struct SegmentationCameraProperties — Configuration of a SegmentationCamera. Change it with setProperties().

Public types

SegmentationType

enum class SegmentationType : int

Which images the camera fills.

Enumerator	Value	Description
SEMANTIC	0	Label image only.
INSTANCE		Object and body images.
PANOPTIC		Object, body and label images.

Constructors & destructor

SegmentationCamera

SegmentationCamera(
    const SegmentationCameraProperties &prop,
    class ArticulatedSystem *as,
    const Vec<3> &pos,
    const Mat<3, 3> &rot)

Create a segmentation camera and allocate the images its type selects, filled with kBackground. The measurement source is MANUAL.

Parameters

Parameter	Description
prop (in)	Camera properties (validated; invalid values cause a fatal error).
as (in)	Owning articulated system.
pos (in)	Sensor position relative to the body (link) it is defined on.
rot (in)	Sensor orientation relative to the body (link) it is defined on.

Public member functions

serializeProp

server::BoundedWriter serializeProp(server::BoundedWriter data) const final

Used by the server. Do not use it if you don't know what it does. It serializes the sensor properties

Parameters

Parameter	Description
data (in)	where the property is written; throws server::SerializationError if it does not fit

Returns where the next data should be written

getProperties

const SegmentationCameraProperties &getProperties() const

Returns The camera properties.

setProperties

void setProperties(const SegmentationCameraProperties &prop)

Replace the camera properties transactionally. Invalid properties leave the camera unchanged. The images are reallocated for the new type and size and filled with kBackground, and a connected viewer receives the new properties before it renders the camera again.

Parameters

Parameter	Description
prop (in)	New properties. name and full_name must match the current ones (fatal error otherwise).

getObjectImage

const std::vector<int32_t> &getObjectImage() const

World index of the object seen in each pixel (see the class comment); empty for SEMANTIC.

std::vector<int32_t> &getObjectImage()

Mutable access for MANUAL measurements; keep its size.

getBodyImage

const std::vector<int32_t> &getBodyImage() const

Body index within the object seen in each pixel; empty for SEMANTIC.

std::vector<int32_t> &getBodyImage()

Mutable access for MANUAL measurements; keep its size.

getLabelImage

const std::vector<int32_t> &getLabelImage() const

Semantic label seen in each pixel; empty for INSTANCE.

std::vector<int32_t> &getLabelImage()

Mutable access for MANUAL measurements; keep its size.

setObjectImage

void setObjectImage(const std::vector<int32_t> &image)

Replace the object image. Fatal error unless it has the current image's size.

Parameters

Parameter	Description
image (in)	width * height values, or none when the type has no object image.

setBodyImage

void setBodyImage(const std::vector<int32_t> &image)

Replace the body image. Fatal error unless it has the current image's size.

Parameters

Parameter	Description
image (in)	width * height values, or none when the type has no body image.

setLabelImage

void setLabelImage(const std::vector<int32_t> &image)

Replace the label image. Fatal error unless it has the current image's size.

Parameters

Parameter	Description
image (in)	width * height values, or none when the type has no label image.

update

void update(class World &world) final

Ray-cast the images: refreshes the sensor pose from its attachment frame, then casts one ray per pixel against the collision bodies of world, ignoring the owning body. Called by RaiSim when the measurement source is RAISIM.

Parameters

Parameter	Description
world (in)	the world that the sensor is in

Static member functions

getType static

static Type getType()

Returns Sensor::Type::SEGMENTATION.

hasInstanceImages static

static bool hasInstanceImages(SegmentationType type)

Returns True when type fills the object and body images.

hasLabelImage static

static bool hasLabelImage(SegmentationType type)

Returns True when type fills the label image.

imageCount static

static int imageCount(SegmentationType type)

Returns The number of images type fills (1 for SEMANTIC, 2 for INSTANCE, 3 for PANOPTIC).

colorize static

static void colorize(
    const std::vector<int32_t> &ids,
    std::vector<unsigned char> &rgba,
    const std::array<unsigned char, 4> &background={0, 0, 0, 0})

Color an id image for display: each non-negative id gets a fixed, well-separated color that does not depend on the other ids; negative ids get background.

Parameters

Parameter	Description
ids (in)	Image of ids (any of the three images).
rgba (out)	Resized to ids.size() * 4 bytes, RGBA.
background (in)	Color of negative ids.

colorOf static

static std::array<unsigned char, 3> colorOf(int32_t id)

Parameters

Parameter	Description
id (in)	Non-negative id.

Returns The RGB color colorize() gives id (alpha is 255).

Public attributes

kBackground

static constexpr int32_t kBackground

Pixel value where nothing was hit, in every image.

kNotWorldObject

static constexpr int32_t kNotWorldObject

Object and body image value of rayrai visuals that are not RaiSim objects.

Inherited members

From raisim::Sensor: getArticulatedSystem, getFrameId, getFrameOrientation, getFramePosition, getFullName, getMeasurementSource, getName, getOrientation, getOriInSensorFrame, getPosInSensorFrame, getPosition, getSensorSetModel, getSerialNumber, getUpdateRate, getUpdateTimeStamp, isUpdateDue, lock, lockMutex, MeasurementSource, serializeMeasurements, setAttachmentFrame, setMeasurementSource, setSerialNumber, setUpdateRate, setUpdateTimeStamp, try_lock, Type, unlock, unlockMutex, updatePose