Sensors

RaiSim supports these sensor workflows:

  • URDF-attached sensors for robots. Sensor metadata is loaded from a sensor XML file attached to a URDF link. This is the right workflow for IMU, RGB, depth, and spinning LiDAR sensors that should move with an articulated system.

  • rayrai-rendered RGB/depth sensors for camera observations. When rayrai is available, use in-process rayrai rendering with Sensor::MeasurementSource::MANUAL for RGB/depth sensor buffers. This is the recommended path for rendered images, screenshots, and dataset generation.

  • World-level CPU depth-camera capture only as a deterministic headless fallback. World::captureDepthCamera casts CPU rays from an arbitrary camera pose and returns depth, object segmentation, optional hit points, and a timestamp without requiring a renderer.

The CPU ray-query paths are single-threaded and deterministic, but they are not the recommended RGB/depth image path when rayrai sensors are available. RaiSim itself does not support sensor measurement updates from the TCP visualizer. Renderer-produced images must be consumed in the rayrai process or copied into a sensor buffer by user code with MeasurementSource::MANUAL.

Sensor frames and camera convention

Camera and LiDAR sensors use the RaiSim sensor-frame convention:

  • +X: forward

  • +Y: left

  • +Z: up

Camera depth is the distance along the camera +X axis, not the Euclidean ray length. This matches the existing DepthCamera convention and is convenient for pinhole camera models.

Measurement source and update modes

The update method is selected with raisim::Sensor::setMeasurementSource:

  • RAISIM: measurement is computed by RaiSim. The owning articulated system refreshes it during World::integrate() whenever 1 / update_rate of simulated time has passed since the last update. This is the default for IMU and spinning LiDAR sensors. DepthCamera supports RaiSim-side CPU ray updates, but use this for headless fallback rather than for rendered camera images when rayrai is available. RGBCamera cannot use it.

  • MANUAL: user code writes the buffers. This is the default for RGB and depth cameras. Use it for in-process rayrai RGB/depth rendering or real hardware integration.

There is no VISUALIZER measurement source in the RaiSim API. RaiSim does not request RGB or depth frames from a TCP visualizer and does not update sensor buffers from the visualizer.

Recommended RGB/depth usage with rayrai sets camera sensors to MANUAL and lets the renderer fill or read back the buffers. See Using sensors with rayrai for a complete example. Use RAISIM CPU depth updates only when rayrai is not available or when a deterministic headless ray-query fallback is required.

For RGB sensors, RaiSim does not synthesize color in the physics engine. Write the buffer manually, preferably from in-process rayrai rendering when it is available, or from real hardware:

auto* rgb = sensorSet->getSensor<raisim::RGBCamera>("color");
rgb->setMeasurementSource(raisim::Sensor::MeasurementSource::MANUAL);
std::vector<char> rgba(size_t(rgb->getProperties().width) *
                       size_t(rgb->getProperties().height) * 4);
rgb->setImageBuffer(rgba);

When reading sensor buffers from another user-managed thread, use std::scoped_lock:

{
  std::scoped_lock lock(*depth);
  const auto& image = depth->getDepthArray();
  // read image here
}

Depth camera

raisim::DepthCamera stores a float array with width * height entries. Finite values are plane depth in meters. Missing pixels are NaN.

CPU-only fallback update:

auto* depth = anymal->getSensorSet("depth_camera_front_camera_parent")
                ->getSensor<raisim::DepthCamera>("depth");
depth->setMeasurementSource(raisim::Sensor::MeasurementSource::RAISIM);

world.integrate();     // refreshes the image at the sensor's update rate
depth->update(world);  // optional: ray-cast a new image now

const auto& z = depth->getDepthArray();
const int w = depth->getProperties().width;
const int h = depth->getProperties().height;
float centerDepth = z[(h / 2) * w + (w / 2)];

Convert depth to a point cloud:

std::vector<raisim::Vec<3>> pointsWorld;
depth->depthToPointCloud(depth->getDepthArray(), pointsWorld, false);

std::vector<raisim::Vec<3>> pointsSensor;
depth->depthToPointCloud(depth->getDepthArray(), pointsSensor, true);

The camera intrinsics are defined by width, height, horizontal field of view (hFOV), and optional pixel offsets. If you modify camera properties after loading, call updateRayDirections() before the next update.

RGB camera

raisim::RGBCamera stores an image buffer with width * height * 4 bytes (4 bytes per pixel, rows contiguous). RaiSim itself does not render RGB and does not interpret the channel order; rayrai’s Camera::getRawImage() writes BGRA. Use rayrai or manual hardware input; update() is a fatal error for an RGB camera.

Manual buffer update:

auto* rgb = anymal->getSensorSet("depth_camera_front_camera_parent")
              ->getSensor<raisim::RGBCamera>("color");

auto& prop = rgb->getProperties();
std::vector<char> rgba(size_t(prop.width) * size_t(prop.height) * 4);

// Fill rgba as R, G, B, A bytes.
for (int y = 0; y < prop.height; ++y) {
  for (int x = 0; x < prop.width; ++x) {
    const size_t k = 4 * (size_t(y) * size_t(prop.width) + size_t(x));
    rgba[k + 0] = char(255);
    rgba[k + 1] = char(0);
    rgba[k + 2] = char(0);
    rgba[k + 3] = char(255);
  }
}

rgb->setImageBuffer(rgba);

Spinning LiDAR

raisim::SpinningLidar uses RaiSim ray casting and returns hit points in the sensor frame. The world helper rayTestLidar performs one frustum-style broadphase for the scan, then raycasts only the candidate bodies.

Typical robot-attached LiDAR update:

auto* lidar = anymal->getSensorSet("lidar_link")
                ->getSensor<raisim::SpinningLidar>("lidar");
lidar->setMeasurementSource(raisim::Sensor::MeasurementSource::RAISIM);  // the default

world.integrate();     // refreshes the scan at the sensor's update rate
lidar->update(world);  // optional: scan now

const auto& scanS = lidar->getScan(); // points in sensor frame
lidar->updatePose();
const auto& p_WS = lidar->getPosition();
const auto& R_WS = lidar->getOrientation();

std::vector<raisim::Vec<3>> scanW;
scanW.reserve(scanS.size());
for (const auto& p_S : scanS) {
  scanW.push_back(p_WS + R_WS * p_S);
}

For custom scanning patterns, use World::rayTestLidar directly. Yaw and pitch are each specified as start angle, signed increment, count. See Ray Test for the exact signature, sampling order, compact-output behavior, and performance considerations.

IMU

raisim::InertialMeasurementUnit is intended for robot-attached sensor sets loaded from URDF sensor XML. With RAISIM, the default, the owning articulated system writes the measurements during World::integrate() at the sensor’s update rate; update() does nothing. This requires inverse dynamics, which the URDF and world-XML loaders enable when they create an IMU. Read the values after stepping the world:

auto* imu = anymal->getSensorSet("lidar_link")
              ->getSensor<raisim::InertialMeasurementUnit>("imu");

world.integrate();

const Eigen::Vector3d& acc = imu->getLinearAcceleration();  // specific force, IMU frame [m/s^2]
const Eigen::Vector3d& angVel = imu->getAngularVelocity();  // IMU frame [rad/s]
const raisim::Vec<4>& quat = imu->getOrientation();         // IMU frame in world, (w, x, y, z)

Like a real accelerometer, the IMU reports the specific force (acceleration minus gravity): an IMU at rest reads the negated gravity vector in the IMU frame. With MANUAL, set the values from hardware with setLinearAcceleration(), setAngularVelocity() and setOrientation().

CPU depth camera fallback

Prefer rayrai sensor rendering for RGB/depth observations when rayrai is available. World::captureDepthCamera is a world-level CPU depth camera for deterministic headless fallback and ray-query validation. It does not require a robot-mounted sensor, renderer, or visualizer. It casts one ray per pixel and can return:

  • depth image as std::vector<float>

  • object segmentation ids as std::vector<int>

  • optional world-frame hit points

  • capture timestamp from World::getWorldTime()

  • optional deterministic Gaussian or uniform depth noise

The segmentation value is the hit object’s getIndexInWorld(). Background is -1. Depth and points are NaN for background pixels.

The camera frame follows the same convention as RaiSim depth sensors: +X is forward, +Y is left, and +Z is up. The rotation passed to captureDepthCamera maps camera-frame vectors into the world frame.

CPU fallback scene and capture setup:

raisim::World world;
world.setWorldTime(1.25);
world.addGround(0.0);
auto* sphere = world.addSphere(0.5, 1.0);
sphere->setPosition(0.0, 0.0, 0.5);

raisim::World::DepthCameraProperties prop;
prop.width = 64;
prop.height = 48;
prop.clipNear = 0.01;
prop.clipFar = 10.0;
prop.hFOV = M_PI / 2.0;
prop.captureDepth = true;
prop.captureSegmentation = true;
prop.capturePoints = true;

// Camera at z=2 looking down. Columns are camera axes in world frame:
// camera +X -> world -Z, camera +Y -> world +Y, camera +Z -> world +X.
raisim::Mat<3, 3> R_WC;
R_WC.setZero();
R_WC(0, 2) = 1.0;
R_WC(1, 1) = 1.0;
R_WC(2, 0) = -1.0;

raisim::World::DepthCameraFrame frame;
world.captureDepthCamera({0.0, 0.0, 2.0}, R_WC, prop, frame);

Depth image

Enable captureDepth and read frame.depth. The vector has frame.width * frame.height entries in row-major order. Each value is the distance along the camera forward axis, not the Euclidean ray length. Pixels with no hit or hits outside [clipNear, clipFar] are NaN.

const int center = (prop.height / 2) * prop.width + (prop.width / 2);
if (std::isfinite(frame.depth[center])) {
  float centerDepthMeters = frame.depth[center];
}

Segmentation object id

Enable captureSegmentation and read frame.segmentation. A valid pixel stores the hit object’s getIndexInWorld(). Background pixels store -1. You can compare this value to object ids kept by the application.

prop.captureSegmentation = true;
raisim::World::DepthCameraFrame frame;
world.captureDepthCamera(cameraPos, cameraRot, prop, frame);

const int center = (frame.height / 2) * frame.width + (frame.width / 2);
int objectId = frame.segmentation[center];
if (objectId == static_cast<int>(sphere->getIndexInWorld())) {
  // The center pixel hit the sphere.
}

Optional hit point per pixel

Set capturePoints to true to fill frame.points. Each entry is the world-frame hit point for the corresponding pixel. This is optional because it uses more memory than depth-only or segmentation-only capture.

prop.capturePoints = true;
raisim::World::DepthCameraFrame frame;
world.captureDepthCamera(cameraPos, cameraRot, prop, frame);

const int center = (frame.height / 2) * frame.width + (frame.width / 2);
const raisim::Vec<3>& hitPointW = frame.points[center];
if (std::isfinite(hitPointW[0])) {
  std::cout << "hit point in world: "
            << hitPointW[0] << ", "
            << hitPointW[1] << ", "
            << hitPointW[2] << std::endl;
}

Timestamp

Each frame stores the current World time in frame.timeStamp. This is useful when camera captures are interleaved with control, logging, or dataset generation.

world.setWorldTime(1.25);

raisim::World::DepthCameraFrame frame;
world.captureDepthCamera(cameraPos, cameraRot, prop, frame);

double captureTimeSeconds = frame.timeStamp;  // 1.25 in this example.

Optional deterministic depth noise

Depth noise is disabled by default. To add reproducible noise, set depthNoiseType, depthNoiseMean, depthNoiseStd, and depthNoiseSeed. The same world state, camera pose, properties, and seed produce the same noisy depth image.

prop.depthNoiseType =
    raisim::World::DepthCameraProperties::DepthNoiseType::GAUSSIAN;
prop.depthNoiseMean = 0.0;
prop.depthNoiseStd = 0.002;
prop.depthNoiseSeed = 42;

raisim::World::DepthCameraFrame noisyA;
raisim::World::DepthCameraFrame noisyB;
world.captureDepthCamera(cameraPos, cameraRot, prop, noisyA);
world.captureDepthCamera(cameraPos, cameraRot, prop, noisyB);

const int center = (prop.height / 2) * prop.width + (prop.width / 2);
assert(noisyA.depth[center] == noisyB.depth[center]);

Use DepthNoiseType::UNIFORM for bounded uniform noise in [mean - std, mean + std].

Self-filtering uses the same objectId and localId convention as rayTest. This is useful for cameras mounted on a robot:

world.captureDepthCamera(cameraPos,
                         cameraRot,
                         prop,
                         frame,
                         robot->getIndexInWorld(),
                         cameraParentLocalBodyId);

Using sensors with rayrai

rayrai can render RaiSim camera sensors in process. This is the recommended RGB/depth sensor path when rayrai is available because it uses the same sensor pose and intrinsics while producing renderer-backed color and depth buffers. Use the CPU camera capture API only when rayrai is unavailable or when a CPU-only deterministic ray-query result is specifically required.

Current runnable coverage:

  • Rayrai RGB camera uses the Go1 d455_front/color RGB sensor and renders it through raisin::Camera.

  • Rayrai depth camera uses the Go1 d455_front/depth depth sensor, renders the linear depth plane, and reads a float depth buffer back to the CPU.

  • Rayrai LiDAR point cloud shows a robot-mounted SpinningLidar and visualizes its scan as a rayrai point cloud.

  • Rayrai ArUco marker covers marker rendering and camera-facing visual output.

Use the package examples above as public reference programs for sensor rendering workflows.

A typical RGB/depth setup has four parts: load a URDF with sensors, switch the camera sensors to manual measurements, create rayrai camera objects from those sensors, and render the sensor views every frame.

auto world = std::make_shared<raisim::World>();
world->addGround();

std::vector<std::string> modules = {"d455"};
auto* robot = world->addArticulatedSystem(go1Urdf, modules, go1ResourceDir);

auto* rgbCam = robot->getSensorSet("d455_front")
               ->getSensor<raisim::RGBCamera>("color");
auto* depthCam = robot->getSensorSet("d455_front")
                 ->getSensor<raisim::DepthCamera>("depth");

rgbCam->setMeasurementSource(raisim::Sensor::MeasurementSource::MANUAL);
depthCam->setMeasurementSource(raisim::Sensor::MeasurementSource::MANUAL);

auto viewer = std::make_shared<raisin::RayraiWindow>(world, 1280, 720);
auto rgbCamera = std::make_shared<raisin::Camera>(*rgbCam);
auto depthCamera = std::make_shared<raisin::Camera>(*depthCam);

const auto& rgbProp = rgbCam->getProperties();
const int rgbWidth = std::max(1, rgbProp.width);
const int rgbHeight = std::max(1, rgbProp.height);
std::vector<char> rgbBuffer(size_t(rgbWidth) * size_t(rgbHeight) * 4);

const auto& depthProp = depthCam->getProperties();
const int depthWidth = std::max(1, depthProp.width);
const int depthHeight = std::max(1, depthProp.height);
std::vector<float> depthBuffer(size_t(depthWidth) * size_t(depthHeight));

while (running) {
  world->integrate();

  viewer->renderWithExternalCamera(*rgbCam, *rgbCamera, {});
  viewer->renderWithExternalCamera(*depthCam, *depthCamera, {});
  viewer->renderDepthPlaneDistance(*depthCam, *depthCamera);

  rgbCamera->getRawImage(*rgbCam,
                         raisin::Camera::SensorStorageMode::CUSTOM_BUFFER,
                         rgbBuffer.data(),
                         rgbBuffer.size(),
                         /*flipVertical=*/false);
  depthCamera->getRawImage(*depthCam,
                           raisin::Camera::SensorStorageMode::CUSTOM_BUFFER,
                           depthBuffer.data(),
                           depthBuffer.size(),
                           /*flipVertical=*/false);
}

renderWithExternalCamera renders from the sensor pose; the raisin::Camera constructed from the sensor carries its intrinsics and resolution. renderDepthPlaneDistance converts the depth camera render into a linear plane-distance texture and the float readback buffer contains one depth value per pixel. The RGB readback buffer contains four bytes per pixel in BGRA order.

For LiDAR, rayrai_lidar_pointcloud computes the scan of the robot-mounted SpinningLidar on the CPU with update(world), transforms the sensor-frame points to world coordinates and displays them as a rayrai point cloud. To measure the scan on the GPU instead, call measureSpinningLidarSingleDrawGPU, which stores the sensor-frame hit points in the LiDAR with setScan():

auto* lidar = robot->getSensorSet("livox_lidar_0")
              ->getSensor<raisim::SpinningLidar>("lidar");

lidar->updatePose();
// toGlm(): small conversion helper defined in rayrai_lidar_pointcloud.cpp
const glm::dvec3 posW = toGlm(lidar->getPosition());
const glm::dmat3 rotW = toGlm(lidar->getOrientation());
viewer->measureSpinningLidarSingleDrawGPU(*lidar, posW, rotW);

Examples

  • Use the dedicated rayrai RGB, depth, LiDAR, and ArUco examples above for robot-attached rendered sensor buffers.

  • Use rayrai_complete_showcase when you need one runnable scene that combines RGB/depth cameras, LiDAR visualization, and custom visuals.

  • World::captureDepthCamera supports depth, segmentation, optional hit points, and timestamps without a visualizer, but it is the CPU fallback path.

Parent Class API

class Sensor

Base class of all sensors attached to an ArticulatedSystem frame (RGBCamera, DepthCamera, InertialMeasurementUnit, SpinningLidar).

Sensors are owned by a SensorSet, which belongs to a body of an ArticulatedSystem. Every sensor is attached to a fixed frame of that system (see getFrameId()); its world pose is read from that frame. When the measurement source is MeasurementSource::RAISIM, the owning ArticulatedSystem refreshes the measurement during World::integrate() whenever at least 1/getUpdateRate() seconds of simulated time have passed since getUpdateTimeStamp(). With MeasurementSource::MANUAL, user code (or a renderer) writes the measurement buffers.

Subclassed by raisim::DepthCamera, raisim::InertialMeasurementUnit, raisim::RGBCamera, raisim::SpinningLidar

Public Types

enum class Type : int

Concrete sensor type. Each subclass also provides a static getType() returning its value.

Values:

enumerator UNKNOWN

Not a known sensor type.

enumerator RGB

RGBCamera.

enumerator DEPTH

DepthCamera.

enumerator IMU

InertialMeasurementUnit.

enumerator SPINNING_LIDAR

SpinningLidar.

enum class MeasurementSource : int

Who writes the sensor’s measurement buffers.

Values:

enumerator RAISIM

RaiSim updates the measurement from the physics world.

enumerator MANUAL

User code or an in-process renderer writes the measurement buffer.

Public Functions

inline Sensor(std::string name, std::string fullName, Type type, class ArticulatedSystem *as, const Vec<3> &pos, const Mat<3, 3> &rot, MeasurementSource source)
Parameters:

  • name – [in] Sensor name (unique within its SensorSet).

  • fullName – [in] Name including the sensor-set prefix.

  • type – [in] Concrete sensor type.

  • as – [in] Owning articulated system (may be null until attached).

  • pos – [in] Sensor position relative to the body (link) it is defined on.

  • rot – [in] Sensor orientation relative to the body (link) it is defined on.

  • source – [in] Initial measurement source.

inline const Vec<3> &getPosition()

The pose is refreshed by updatePose() (which the user may call) and by every RaiSim-sourced measurement update. It is not refreshed automatically for MANUAL sensors.

Returns:

The position of the sensor frame in the world frame

inline const Mat<3, 3> &getOrientation()

The pose is refreshed by updatePose() (which the user may call) and by every RaiSim-sourced measurement update. It is not refreshed automatically for MANUAL sensors.

Returns:

The orientation of the sensor frame in the world frame (sensor-to-world rotation)

inline const Vec<3> &getFramePosition()
Returns:

The position of the frame w.r.t. the nearest moving parent. It is used to compute the frame position in the world frame

inline const Mat<3, 3> &getFrameOrientation()
Returns:

The orientation of the frame w.r.t. the nearest moving parent. It is used to compute the frame position in the world frame

inline const Vec<3> &getPosInSensorFrame()
Returns:

The position given at construction, relative to the body (link) the sensor was defined on (i.e., its parent joint frame). Unlike getFramePosition(), it is not re-expressed when fixed bodies are merged into their moving parent.

inline const Mat<3, 3> &getOriInSensorFrame()
Returns:

The orientation given at construction, relative to the body (link) the sensor was defined on (i.e., its parent joint frame). See getPosInSensorFrame().

inline const std::string &getName() const
Returns:

The name of the sensor, as given in the robot/sensor description file

inline const std::string &getFullName() const
Returns:

The full name of the sensor, including the sensor set’s name (e.g. “link_name:sensor_name” for sensors defined in URDF)

inline const std::string &getSensorSetModel()
Returns:

The sensor model identifier from the sensor set.

inline const std::string &getSerialNumber()
Returns:

The serial number of the sensor (copied from the SensorSet when the sensor is added)

inline void setSerialNumber(const std::string &serialNumber)

Set the serial number of the sensor.

Parameters:

serialNumber – [in] Serial number string.

inline Type getType()
Returns:

The type of the sensor

inline double getUpdateRate() const
Returns:

The update frequency in Hz

inline double getUpdateTimeStamp() const

This method returns a negative value if the sensor has never been updated

Returns:

The world time (s) when the last sensor measurement was recorded

inline void setUpdateRate(double rate)

change the update rate of the sensor. Fatal error if the rate is not finite and positive.

Parameters:

rate – [in] the update rate in Hz

inline void setUpdateTimeStamp(double time)

Set the time stamp for the last sensor measurement. RaiSim uses it to schedule the next RaiSim-sourced update.

Parameters:

time – [in] the time stamp in seconds (world time)

virtual server::BoundedWriter serializeProp(server::BoundedWriter data) const = 0

Used by the server. Do not use it if you don’t know what it does. It serializes the sensor properties

Parameters:

data – [in] where the property is written; throws server::SerializationError if it does not fit

Returns:

where the next data should be written

inline virtual server::BoundedWriter serializeMeasurements(server::BoundedWriter data) const

Used by the server. Do not use it if you don’t know what it does. It serializes sensor measurements.

Parameters:

data – [in] where the measurements are written; throws server::SerializationError if they do not fit

Returns:

where the next data should be written

void updatePose()

update the world pose of the sensor (getPosition(), getOrientation()) from its attachment frame in the articulated system. Fatal error if the sensor is not attached to a valid frame.

inline MeasurementSource getMeasurementSource()
Returns:

The measurement source.

inline void setMeasurementSource(MeasurementSource source)

change the measurement source

Parameters:

source – [in] The measurement source.

inline size_t getFrameId() const

Get the id of the frame on which the sensor is attached

Returns:

frame id (index into ArticulatedSystem::getFrames()); size_t(-1) if not attached yet

inline void setAttachmentFrame(const Vec<3> &pos, const Mat<3, 3> &rot, size_t frameId)

Attach this sensor to an existing articulated-system frame. This is primarily useful for editor or procedural scene builders that create sensors after an articulated system has already been instantiated.

Parameters:

  • pos – [in] Frame position relative to the nearest moving parent body.

  • rot – [in] Frame orientation relative to the nearest moving parent body.

  • frameId – [in] Index of the frame in ArticulatedSystem::getFrames().

inline void lockMutex()

locks the sensor mutex. This can be used if you use raisim in a multi-threaded environment.

inline void lock()

Same as lockMutex(); makes Sensor usable with std::lock_guard / std::unique_lock.

inline bool try_lock()

Try to lock the sensor mutex without blocking.

Returns:

true if the lock was acquired.

inline void unlockMutex()

unlock the sensor mutex. This can be used if you use raisim in a multi-threaded environment.

inline void unlock()

Same as unlockMutex(); for std::lock_guard / std::unique_lock compatibility.

Depth Camera API

class DepthCamera : public raisim::Sensor

Depth camera simulated by CPU ray casting against the collision geometry of the world.

Sensor frame convention: +x is the optical axis (forward), +y points left and +z points up. Images are stored row-major with index row * width + col; row 0 is the top of the image and column 0 its left side (as seen when looking along +x). Depth values are in meters, measured along the optical axis (+x), not along the ray.

The constructor selects MeasurementSource::MANUAL. Call setMeasurementSource(MeasurementSource::RAISIM) to let RaiSim compute the depth image at the sensor’s update rate.

Public Types

enum Frame

Coordinate-frame identifiers. Not used by the current DepthCamera API.

Values:

enumerator SENSOR_FRAME

Sensor frame.

enumerator ROOT_FRAME

Root-body frame of the articulated system.

enumerator WORLD_FRAME

World frame.

Public Functions

inline explicit DepthCamera(const DepthCameraProperties &prop, class ArticulatedSystem *as, const Vec<3> &pos, const Mat<3, 3> &rot)

Create a depth camera and precompute its ray directions. The measurement source is MANUAL.

Parameters:

  • prop – [in] Camera properties (validated; invalid values cause a fatal error).

  • as – [in] Owning articulated system.

  • pos – [in] Sensor position relative to the body (link) it is defined on.

  • rot – [in] Sensor orientation relative to the body (link) it is defined on.

inline void updateRayDirections()

This method must be called after sensor properties are modified. It validates the properties, recomputes the ray directions and resizes the depth and point buffers to width * height.

inline virtual server::BoundedWriter serializeProp(server::BoundedWriter data) const final

Used by the server. Do not use it if you don’t know what it does. It serializes the sensor properties

Parameters:

data – [in] where the property is written; throws server::SerializationError if it does not fit

Returns:

where the next data should be written

inline void set3DPoints(const std::vector<raisim::Vec<3>> &data)

This method is only useful on the real robot (and you use raisim on the real robot). You can set the 3d point array manually

Parameters:

data – [in] The 3d points, width * height entries in the image layout (world frame, to match get3DPoints()). A size mismatch is a fatal error.

inline const std::vector<float> &getDepthArray() const

Depth image of width * height floats in meters along the optical axis, row-major with row 0 at the top. Pixels with no hit within [clipNear, clipFar] are NaN. The values are zero until the first update; check that getUpdateTimeStamp() is non-negative (negative means never updated).

Returns:

depthArray

inline std::vector<float> &getDepthArray()

Mutable access to the depth image (same layout as the const overload); use it to write MANUAL measurements.

Returns:

depthArray

inline void setDepthArray(const std::vector<float> &depthIn)

Set the data manually. This can be useful on the real robot

Parameters:

depthIn – [in] Depth values (meters along the optical axis), width * height entries in the image layout. A size mismatch is a fatal error.

inline const std::vector<raisim::Vec<3>, AlignedAllocator<raisim::Vec<3>, 32>> &get3DPoints() const

Hit points of the last RaiSim update (or the values given to set3DPoints()), one per pixel in the image layout. Pixels without a valid hit are NaN. Only RaiSim-sourced updates fill this buffer.

Returns:

3D points in the world frame

inline DepthCameraProperties &getProperties()

Get the depth camera properties. MUST call updateRayDirections() after modifying the sensor properties.

Returns:

the depth camera properties

virtual void update(class World &world) final

Ray-cast one depth image: refreshes the sensor pose from its attachment frame, then fills getDepthArray() and get3DPoints(). The owning body (the attachment frame’s parent body) is excluded from the ray test. Called by RaiSim when the measurement source is RAISIM.

Parameters:

world – [in] the world that the sensor is in

void depthToPointCloud(const std::vector<float> &depthArray, std::vector<raisim::Vec<3>> &pointCloud, bool isInSensorFrame = false) const

Convert the depth values to 3D coordinates

Parameters:

  • depthArray – [in] input depth array to convert (width * height entries in the image layout; a size mismatch is a fatal error)

  • pointCloud – [out] output point cloud, resized to width * height. Non-finite depths give NaN points.

  • isInSensorFrame – [in] True to output points in the sensor frame, false for the world frame (uses the current pose of the attachment frame)

inline const auto &getPrecomputedRayDir() const
Returns:

Precomputed ray directions in the sensor frame, one per pixel in the image layout. Pinhole directions are not normalized (their x component is 1); fisheye directions are unit vectors.

Public Static Functions

static inline Type getType()
Returns:

Sensor::Type::DEPTH.

struct DepthCameraProperties

Configuration of a DepthCamera. Call updateRayDirections() after modifying it in place.

Public Types

enum class NoiseType : int

Noise model (stored and serialized; RaiSim does not add noise to the depth image).

Values:

enumerator GAUSSIAN

Gaussian noise.

enumerator UNIFORM

Uniform noise.

enumerator NO_NOISE

No noise.

Public Members

std::string name

Sensor name (without the sensor set prefix).

std::string full_name

Sensor name including the sensor set prefix.

int width

Image width in pixels. PHY-090: safe defaults so direct construction cannot allocate a huge buffer or divide by zero before the loader fills real values.

int height

Image height in pixels.

int xOffset

Horizontal pixel offset added to the column index when computing ray directions (shifts the capture window).

int yOffset

Vertical pixel offset added to the row index when computing ray directions (shifts the capture window).

double clipNear

Near clipping distance in meters, measured along the optical axis. Closer hits are ignored.

double clipFar

Far clipping distance in meters, measured along the optical axis. Also the maximum ray length.

double hFOV

Horizontal field of view in radians. Must be in (0, pi) for pinhole and (0, pi] for fisheye.

CameraLensModel lens

Lens specification. CPU depth rays follow the pinhole or equidistant-fisheye projection (fisheye uses the lens intrinsics if set); fisheye distortion coefficients are not applied.

enum raisim::DepthCamera::DepthCameraProperties::NoiseType noiseType

Selected noise model (default NO_NOISE).

double mean

Noise mean.

double std

Noise standard deviation.

std::string format

Pixel format string, stored and serialized as metadata. The depth buffer itself is always float.

Public Static Functions

static inline NoiseType stringToNoiseType(const std::string &type)
Parameters:

type – [in] Noise name.

Returns:

GAUSSIAN for “gaussian”/”Gaussian”, UNIFORM for “uniform”/”Uniform”, otherwise NO_NOISE.

RGB Camera API

class RGBCamera : public raisim::Sensor

Color camera sensor. RaiSim stores the camera configuration and an image buffer but does not render images itself: the buffer must be written by user code or an in-process renderer (e.g. rayrai’s Camera::getRawImage()), so the measurement source must stay MANUAL.

Public Functions

inline RGBCamera(const RGBCameraProperties &prop, class ArticulatedSystem *as, const Vec<3> &pos, const Mat<3, 3> &rot)

Create an RGB camera and allocate its width * height * 4 byte image buffer. The measurement source is MANUAL.

Parameters:

  • prop – [in] Camera properties (validated; invalid values cause a fatal error).

  • as – [in] Owning articulated system.

  • pos – [in] Sensor position relative to the body (link) it is defined on.

  • rot – [in] Sensor orientation relative to the body (link) it is defined on.

inline virtual server::BoundedWriter serializeProp(server::BoundedWriter data) const final

Used by the server. Do not use it if you don’t know what it does. It serializes the sensor properties

Parameters:

data – [in] where the property is written; throws server::SerializationError if it does not fit

Returns:

where the next data should be written

inline const RGBCameraProperties &getProperties() const

Return immutable camera properties.

Returns:

camera properties

inline void setProperties(const RGBCameraProperties &prop)

Replace camera properties transactionally and resize the dependent image buffer. Invalid properties leave the camera unchanged. The image contents are reset to zero.

Parameters:

prop – [in] New properties. name and full_name must match the current ones (fatal error otherwise).

inline const std::vector<char> &getImageBuffer() const

Image buffer of width * height * 4 bytes (4 bytes per pixel, rows contiguous). RaiSim does not synthesize RGB images or interpret the channel order; user code or an in-process renderer must write this buffer manually. rayrai’s Camera::getRawImage() writes BGRA.

Returns:

The image data in char vector

inline void setImageBuffer(const std::vector<char> &rgbaIn)

Replace the image buffer. Useful on the real robot or for renderer output.

Parameters:

rgbaIn – [in] 4-byte-per-pixel image of exactly width * height * 4 bytes (fatal error otherwise), in the same layout as getImageBuffer()

inline virtual void update(class World &world) final

Always a fatal error: RaiSim cannot produce RGB measurements (use MeasurementSource::MANUAL).

Parameters:

world – [in] unused

Public Static Functions

static inline Type getType()
Returns:

Sensor::Type::RGB.

struct RGBCameraProperties

Configuration of an RGBCamera. Change it with setProperties().

Public Types

enum class NoiseType : int

Noise model (stored and serialized; RaiSim does not apply it).

Values:

enumerator GAUSSIAN

Gaussian noise.

enumerator UNIFORM

Uniform noise.

enumerator NO_NOISE

No noise.

Public Members

std::string name

Sensor name (without the sensor set prefix).

std::string full_name

Sensor name including the sensor set prefix.

int width

Image width in pixels.

int height

Image height in pixels.

int xOffset

Horizontal pixel offset for the capture window.

int yOffset

Vertical pixel offset for the capture window.

double clipNear

Near clipping plane in meters.

double clipFar

Far clipping plane in meters.

double hFOV

Horizontal field of view in radians. Must be in (0, pi) for pinhole and (0, pi] for fisheye.

CameraLensModel lens

Renderer lens specification serialized with the sensor metadata. Defaults to pinhole.

enum raisim::RGBCamera::RGBCameraProperties::NoiseType noiseType

Selected noise model (default NO_NOISE).

double mean

Noise mean.

double std

Noise standard deviation.

std::string format

Pixel format string, stored and serialized as metadata. The buffer always has 4 bytes per pixel.

Public Static Functions

static inline NoiseType stringToNoiseType(const std::string &type)
Parameters:

type – [in] Noise name.

Returns:

GAUSSIAN for “gaussian”/”Gaussian”, UNIFORM for “uniform”/”Uniform”, otherwise NO_NOISE.

Inertial Measurement Unit API

class InertialMeasurementUnit : public raisim::Sensor

Inertial measurement unit. With MeasurementSource::RAISIM (the default), the owning ArticulatedSystem writes the measurements at the sensor’s update rate. This requires inverse dynamics on the articulated system (ArticulatedSystem::setComputeInverseDynamics(true)), which the URDF and world-XML loaders enable when they create an IMU.

Public Functions

inline explicit InertialMeasurementUnit(const ImuProperties &prop, class ArticulatedSystem *as, const Vec<3> &pos, const Mat<3, 3> &rot)

Create an IMU with zero-initialized measurements. The measurement source is RAISIM.

Parameters:

  • prop – [in] IMU properties.

  • as – [in] Owning articulated system.

  • pos – [in] Sensor position relative to the body (link) it is defined on.

  • rot – [in] Sensor orientation relative to the body (link) it is defined on.

inline virtual server::BoundedWriter serializeProp(server::BoundedWriter data) const final

Used by the server. Do not use it if you don’t know what it does. It serializes the sensor properties

Parameters:

data – [in] where the property is written; throws server::SerializationError if it does not fit

Returns:

where the next data should be written

inline virtual server::BoundedWriter serializeMeasurements(server::BoundedWriter data) const

Used by the server. Do not use it if you don’t know what it does. It serializes sensor measurements.

Parameters:

data – [in] where the measurements are written; throws server::SerializationError if they do not fit

Returns:

where the next data should be written

inline const Eigen::Vector3d &getLinearAcceleration() const

Get the linear acceleration measured by the sensor. In simulation, this is updated every update loop (set by the updateRate). On the real robot, this value should be set externally by setLinearAcceleration()

Returns:

linear acceleration in m/s^2, expressed in the IMU frame. Like a real accelerometer it reports specific force (acceleration minus gravity), so an IMU at rest reads the negated gravity vector (e.g. 9.81 m/s^2 pointing up) rotated into the IMU frame. Clamped in norm to ImuProperties::maxAcc.

inline const Eigen::Vector3d &getAngularVelocity() const

Get the angular velocity measured by the sensor. In simulation, this is updated every update loop (set by the updateRate). On the real robot, this value should be set externally by setAngularVelocity()

Returns:

angular velocity in rad/s, expressed in the IMU frame. Clamped in norm to ImuProperties::maxAngVel.

inline const Vec<4> &getOrientation() const

Get the orientation measured by the sensor as a quaternion (w, x, y, z). In simulation, this is updated every update loop (set by the updateRate). On the real robot, this value should be set externally by setOrientation()

Returns:

orientation of the IMU frame relative to the world frame, quaternion (w, x, y, z)

inline void setLinearAcceleration(const Eigen::Vector3d &acc)

This set method make sense only on the real robot (MANUAL measurement source).

Parameters:

acc – [in] acceleration measured from sensor (m/s^2, IMU frame)

inline void setAngularVelocity(const Eigen::Vector3d &vel)

This set method make sense only on the real robot (MANUAL measurement source).

Parameters:

vel – [in] angular velocity measured from sensor (rad/s, IMU frame)

inline void setOrientation(const Vec<4> &orientation)

This set method make sense only on the real robot (MANUAL measurement source).

Parameters:

orientation – [in] orientation of the imu sensor in quaternion (w,x,y,z convention)

inline ImuProperties &getProperties()
Returns:

The IMU configuration properties.

inline virtual void update(class World &world) final

Does nothing. The measurements are computed by the owning ArticulatedSystem.

Parameters:

world – [in] unused

Public Static Functions

static inline Type getType()
Returns:

Sensor::Type::IMU.

struct ImuProperties

Configuration of an InertialMeasurementUnit.

Public Types

enum class NoiseType : int

Noise model (stored; RaiSim does not add noise to the IMU measurements).

Values:

enumerator GAUSSIAN

Gaussian noise.

enumerator UNIFORM

Uniform noise.

enumerator NO_NOISE

No noise.

Public Members

std::string name

Sensor name (without the sensor set prefix).

std::string full_name

Sensor name including the sensor set prefix.

double maxAcc

Maximum measurable linear acceleration (m/s^2). Larger readings are scaled down to this norm.

double maxAngVel

Maximum measurable angular velocity (rad/s). Larger readings are scaled down to this norm.

enum raisim::InertialMeasurementUnit::ImuProperties::NoiseType noiseType

Selected noise model (default NO_NOISE).

double mean

Noise mean.

double std

Noise standard deviation.

Public Static Functions

static inline NoiseType stringToNoiseType(const std::string &type)
Parameters:

type – [in] Noise name.

Returns:

GAUSSIAN for “gaussian”/”Gaussian”, UNIFORM for “uniform”/”Uniform”, otherwise NO_NOISE.