rayrai Visualizer

Overview

rayrai is an in-process C++ renderer for RaiSim. It renders into an offscreen OpenGL texture and is designed to embed in custom UIs (ImGui, Qt, etc.) or headless pipelines. RaisimUnity and RaisimUnreal are no longer supported; rayrai is the supported visualization path. It runs inside the simulation process and exposes direct access to render targets, picking, and custom visuals.

Key characteristics:

• In-process rendering (no server required)

• Offscreen textures for UI embedding or sensor simulation

• Lightweight scene management for RaiSim objects and custom visuals

• glTF/GLB visual-scene import with PBR materials, normal maps, and lights

• HDR image-based lighting, reflection probes, and real-time shadow maps

• Support for point clouds, coordinate frames, and picking

• Direct ImGui/SDL2 integration patterns and headless/offscreen workflows

The public API lives in the raisin namespace (note the spelling). The primary entry point is raisin::RayraiWindow.

Dependencies

rayrai depends on SDL2, OpenGL, glbinding, glm, assimp, stb, and imgui. The installed package in rayrai/<OS> ships the headers and CMake config for these dependencies.

TCP Viewer

For most users, the easiest way to use rayrai with RaisimServer examples is the TCP viewer. Build it with the rest of the examples, source the environment, and run it from the build tree:

source ./raisim_env.sh
cmake -S . -B build -DRAISIM_EXAMPLE=ON
cmake --build build --target rayrai_tcp_viewer -j
./build/examples/rayrai_tcp_viewer

Run it with --help to see available options. On Windows, use the .exe binary. If you installed the build, run the copy from the configured install prefix’s bin directory. Source raisim_env.sh before running the viewer or server example so both RaiSim and rayrai shared libraries are visible.

The TCP viewer overlay has two compact panels: the left Raisim TCP control/object panel and the right selected-object details panel. Both panels have a small -/+ button in the header to minimize or restore the panel. The left panel also auto-minimizes when you click elsewhere in the viewer, leaving only the compact header visible.

Build and link

rayrai installs a CMake package under rayrai/<OS>. Add it to CMAKE_PREFIX_PATH and link the rayrai target.

find_package(rayrai CONFIG REQUIRED)

add_executable(my_app main.cpp)
target_link_libraries(my_app PRIVATE rayrai)

In practice, you will also set the Raisim prefix (for example, -DCMAKE_PREFIX_PATH) to include both raisim/<OS> and rayrai/<OS>.

Minimal usage

The typical workflow is:

1. Create a RaiSim world.

2. Construct a raisin::RayraiWindow for that world.

3. Update the renderer each frame and consume the output texture.

#include <memory>

#include <raisim/World.hpp>
#include <rayrai/RayraiWindow.hpp>

int main() {
  auto world = std::make_shared<raisim::World>();
  raisin::RayraiWindow viewer(world, 1280, 720);

  auto sphere = viewer.addVisualSphere("goal", 0.2, 0.9f, 0.2f, 0.2f, 1.0f);
  sphere->setPosition(0.0, 0.0, 1.0);

  while (true) {
    world->integrate();
    viewer.update(1280, 720, false, 0, 0, false);
    unsigned int tex = viewer.getImageTexture();
    (void)tex; // use the texture in your UI or pipeline
  }
}

If you are integrating with an existing OpenGL context, you can use RayraiWindow::createOffscreenGlContext and RayraiWindow::makeOffscreenContextCurrent to control the context setup.

Example: custom visuals + background color

This example adds a custom box, renders an RGB texture each frame, and reads the image texture handle for UI integration. New code should prefer explicit color-range APIs such as setBackgroundColorRgb255 or setBackgroundColorLinear.

#include <memory>

#include <raisim/World.hpp>
#include <rayrai/RayraiWindow.hpp>

int main() {
  auto world = std::make_shared<raisim::World>();
  world->addGround();

  raisin::RayraiWindow viewer(world, 1280, 720);
  viewer.setBackgroundColorRgb255({40, 45, 55, 255});

  auto box = viewer.addVisualBox("marker", 0.4, 0.2, 0.1, 0.9f, 0.6f, 0.1f, 1.0f);
  box->setPosition(1.0, 0.0, 0.3);

  while (true) {
    world->integrate();
    viewer.update(1280, 720, false, 0, 0, false);
    unsigned int colorTex = viewer.getImageTexture();
    (void)colorTex; // feed into your UI or pipeline
  }
}

Color and gamma semantics

rayrai uses explicit color ranges for new APIs:

• Visuals::setColor and most object/material color factors use linear 0..1 RGBA.

• Camera::setBackgroundColorRgb255 and RayraiWindow::setBackgroundColorRgb255 use legacy 0..255 RGBA. setBackgroundColor is kept as a compatibility wrapper for the same 0..255 range.

• setBackgroundColorLinear accepts linear 0..1 RGBA and converts it internally.

• Texture uploads distinguish color maps and data maps. Use loadColorTextureWithTiling for sRGB albedo/emissive maps, and loadDataTextureWithTiling for normal, metallic-roughness, AO, depth, mask, or other linear data textures.

The renderer applies display gamma at the shader/output stage. Do not pre-gamma-correct linear object colors before passing them to rayrai.

Shadows and lights

rayrai has a fast single-main-light path for robotics workloads and a higher-quality multi-light path for authored visual scenes. The main light is a directional light and is always the cheapest shadow-casting source. Imported glTF/Blender scenes can also use additional directional, point, spot, and area-style lights.

By default, the main shadow center tracks a point in front of the camera and the shadow box is fixed in size. You can customize both via RayraiWindow:

// shadow center is N meters ahead of the camera (default: 10.0)
viewer.setShadowCenterOffset(12.0f);
// shadow box (default: halfSize=12.5, near=0.1, far=55.0)
viewer.setShadowOrtho(20.0f, 0.1f, 80.0f);

If you need a fully custom shadow view/projection, use the lower-level raisin::Light API directly.

Additional lights are controlled explicitly and capped so the fast path stays fast. Rayrai currently supports up to RayraiWindow::kMaxAdditionalLights additional lights and up to RayraiWindow::kMaxAdditionalShadowLights additional shadow maps. Directional, spot, and area-style lights use 2D shadow maps. Point lights use cubemap shadow maps. Shadow framebuffer setup validates the current OpenGL context and recreates stale framebuffer/texture names when a viewer context is rebuilt; this matters for TCP-viewer lifetime, offscreen tests, and applications that create/destroy render contexts. For imported scenes, RenderQualitySettings::autoSelectImportedShadowLight can promote the strongest imported light to the main shadow caster, while the remaining shadow budget is assigned to additional lights.

raisin::RayraiWindow::AdditionalLight fill;
fill.type = raisin::LightType::DIRECTIONAL;
fill.direction = glm::normalize(glm::vec3(0.4f, -0.2f, -0.8f));
fill.diffuse = glm::vec3(0.10f, 0.12f, 0.16f);
viewer.addAdditionalLight(fill);

raisin::RayraiWindow::AdditionalLight spot;
spot.type = raisin::LightType::SPOT;
spot.position = glm::vec3(1.8f, -1.6f, 2.6f);
spot.direction = glm::normalize(glm::vec3(-1.4f, 1.0f, -1.8f));
spot.diffuse = glm::vec3(0.18f, 0.42f, 1.0f);
spot.spotInnerCos = std::cos(glm::radians(14.0f));
spot.spotOuterCos = std::cos(glm::radians(28.0f));
viewer.addAdditionalLight(spot);

raisin::RayraiWindow::AdditionalLight area;
area.type = raisin::LightType::AREA;
area.position = glm::vec3(0.0f, 1.8f, 2.1f);
area.diffuse = glm::vec3(0.55f, 0.65f, 0.42f);
area.radius = 1.4f;
area.areaSize = glm::vec2(1.8f, 0.9f);
viewer.addAdditionalLight(area);

viewer.clearAdditionalLights();

Shadow update cost is configurable. Dynamic scenes can update shadows every frame; static visual scenes can bake shadow maps at startup or refresh them only when light/object placement changes.

auto quality = raisin::RayraiWindow::defaultRenderQualitySettings(
  raisin::RayraiWindow::RenderQualityPreset::Ultra);
quality.updateShadowsEveryFrame = false;       // startup/on-demand shadow bake
quality.maxAdditionalLightsPerFrame = 12;      // light evaluation budget
quality.maxAdditionalShadowLights = 4;         // 2D shadow-map budget
quality.maxPointShadowLights = 2;              // cubemap shadow budget
quality.additionalShadowResolutionScale = 0.5f;
quality.pointShadowResolutionScale = 0.5f;
viewer.setRenderQualitySettings(quality);

Use lower budgets for interactive editing or RL throughput. Use higher budgets for offline screenshots, inspection, or demos where visual fidelity is more important than frame time.

HDR, image-based lighting, and reflections

rayrai supports HDR equirectangular environments for real-time PBR preview and inspection. The HDR path is not a ray tracer; it precomputes cubemap data for environment background, diffuse irradiance, specular prefiltering, and a split-sum BRDF lookup table, then samples those textures in the PBR shader.

The typical setup is:

const char* hdr = "/path/to/small_harbour_sunset_1k.hdr";
unsigned int env = raisin::RayraiWindow::loadHdrEquirectangularCubemap(hdr, 128, true);
unsigned int irradiance = raisin::RayraiWindow::createHdrIrradianceCubemap(hdr, 32, 64);
unsigned int prefiltered =
  raisin::RayraiWindow::createHdrPrefilteredEnvironmentCubemap(hdr, 128, 5, 64);
unsigned int brdf = raisin::RayraiWindow::createSplitSumBrdfLut(128, 128);

visual->setPbrEnvironment(env, irradiance, prefiltered, brdf, 1.0f);

Use HDR environments with visible features when inspecting reflective materials. A featureless sky or uniform studio HDR can make it hard to tell whether reflections are working. rayrai_pbr_material_grid and rayrai_pbr_texture_maps use image-based lighting and PBR texture maps so metallic and glossy surfaces show visible reflections while non-metallic assets remain mostly diffuse.

For scene-wide reflections, rayrai also has static reflection probe capture and planar ground reflection support. These are real-time approximation tools: they improve visual fidelity without enabling path tracing or other slow offline rendering mechanisms. Choose lower environment resolution, fewer prefilter samples, and fewer reflection updates for fast interactive runs; increase those values for screenshots or inspection.

Render-quality controls

rayrai keeps RL throughput and visual fidelity separate. The Fast preset keeps reflections, high-fidelity PBR, FXAA, and extra expensive viewer effects off by default. High and Ultra enable the quality-oriented path, including high-fidelity PBR, reflective ground, planar reflection sampling, stronger shadow filtering, FXAA, and depth-of-field postprocessing.

Use presets for common cases:

viewer.setRenderQualityPreset(raisin::RayraiWindow::RenderQualityPreset::Fast);
viewer.setRenderQualityPreset(raisin::RayraiWindow::RenderQualityPreset::Ultra);

Use explicit settings when you need runtime control:

auto quality = raisin::RayraiWindow::defaultRenderQualitySettings(
  raisin::RayraiWindow::RenderQualityPreset::Ultra);
quality.fxaaEnabled = true;
quality.depthOfFieldEnabled = true;
quality.depthOfFieldFocusDistance = 5.0f;
quality.depthOfFieldFocusRange = 8.0f;
quality.depthOfFieldMaxRadius = 1.25f;
quality.reflectiveGround = true;
quality.addViewerFillLights = false;
viewer.setRenderQualitySettings(quality);

The shipped rayrai_feature_showcase target exercises these controls and writes offscreen images and reports.

PBR materials

rayrai supports a lightweight glTF-style metallic-roughness PBR path in addition to the existing simple Phong-style renderer. Simple color and legacy textured meshes stay on the fast path; PBR shader work is used only for meshes whose material requests PBR features or PBR texture maps.

Supported material inputs include:

• base color factor and base color texture

• metallic and roughness factors

• metallic-roughness texture

• normal texture

• occlusion texture

• emissive factor and emissive texture

Lighting is based on rayrai’s main light, optional additional lights, shadow maps, HDR/image-based lighting when configured, and optional reflection probes. Color textures are uploaded as sRGB; data maps such as normal, metallic-roughness, and AO remain linear. Normal maps require tangent data; glTF assets usually provide it, and rayrai generates or imports tangent data where possible. The PBR path is suitable for preview, data generation, and asset inspection; it is not an offline path tracer.

The shipped PBR examples and tools are:

• rayrai_pbr_material_grid: Khronos glTF material-grid asset inspection.

• rayrai_pbr_texture_maps: base-color, normal, metallic-roughness, occlusion, and emissive texture-map coverage.

• KHR_lights_punctual from the glTF/GLB file for directional, point, and spot lights.

• *.rayrai_lights.json for Blender area lights with size, direction, color, and energy.

For best results, keep the authored scene in metric scale, keep Z as up, and prefer glTF/GLB over OBJ. OBJ is useful for simple geometry interchange, but it loses too much of the scene-level material and light data needed for high-quality rendering.

Material import details

The Assimp/glTF importer is asset-agnostic. It does not special-case the blue-wall scene or material names. It follows this priority:

• Use explicit material texture slots from the source asset when present.

• Treat base-color and emissive maps as color textures.

• Treat normal, metallic-roughness, occlusion, masks, and other data maps as linear data.

• Preserve normal-map scale and detect common OpenGL-vs-DirectX normal-map naming.

• Use embedded glTF textures when available.

• Search sibling texture files for common PBR map names when the source material omits a slot but the files are packaged next to the asset.

• Keep simple solid-color materials on the simple path unless normal maps or PBR features require the PBR shader.

This fallback behavior is meant to support real downloadable assets whose Blender, glTF, FBX, DAE, and OBJ exports often disagree about how texture slots are authored. If an asset renders white or flat, first check the import report/debug output for which texture slots were found and whether the file paths exist next to the scene.

Visual assets and collision assets

rayrai visual meshes are renderer assets. URDF models can define separate visual and collision meshes, and standalone rayrai visuals can use glTF material and texture data for inspection or presentation without becoming collision geometry in raisim::World. Keep this separation when an asset has high-detail visual triangles, PBR materials, or texture maps.

Use this pattern when you want realistic visuals with collision meshes tuned for physics:

auto* robot = world.addArticulatedSystem("anymal_c/urdf/anymal.urdf");
auto* object = world.addArticulatedSystem("ycb/002_master_chef_can.urdf");

Only call World::addMesh for collision when the mesh is intentionally part of the physics model. The current textured glTF and imported scene examples keep renderer assets separate from collision geometry; the physics model still comes from the URDF or explicit collision objects.

CoACD mesh approximation visualization

The rayrai_coacd_mesh_approximation example visualizes collision convexification. It shows the source mesh and the generated convex parts side by side. The decomposition side uses per-part colors so individual convex bodies can be inspected.

This example uses real meshes from rsc such as YCB and Minitaur assets. Some robot visual meshes are intentionally not used because they are non-manifold visual shells and are rejected by the minimal CoACD integration.

Examples

Rayrai examples are documented in Examples. Each example page includes a short explanation and installed executable usage.

Quick map to the current rayrai-related targets:

• rayrai_basic_scene: minimal in-process rayrai rendering.

• rayrai_complete_showcase: broad rayrai feature overview with sensors and visuals.

• rayrai_pbr_material_grid: glTF material-grid inspection.

• rayrai_pbr_texture_maps: PBR texture-map import and inspection.

• rayrai_coacd_mesh_approximation: visual comparison of original meshes and CoACD convex parts generated through World::addMesh.

• rayrai_tcp_viewer: TCP viewer for raisim::RaisimServer scenes.

ImGui integration (SDL2 + OpenGL)

rayrai is designed to be embedded in custom UI. The repository ships a minimal SDL2/ImGui helper in rayrai/example_common.hpp. The core idea is:

1. Create an OpenGL context (SDL2 here).

2. Render the RayraiWindow texture into an ImGui Image.

3. Forward hover and cursor data to RayraiWindow::update.

Minimal pattern (trimmed from the examples):

ExampleApp app;
if (!app.init("rayrai_example", 1280, 720))
  return -1;

auto world = std::make_shared<raisim::World>();
auto viewer = std::make_shared<raisin::RayraiWindow>(world, 1280, 720);

while (!app.quit) {
  app.processEvents();
  world->integrate();

  app.beginFrame();
  app.renderViewer(*viewer); // ImGui::Image + viewer.update(...)
  app.endFrame();
}

The renderViewer helper uses ImGui::IsItemHovered() and mouse positions to drive camera interaction and picking. Current offscreen examples exercise the camera paths used by rayrai rendering and validation.

Headless/offscreen OpenGL context

If you already manage an OpenGL context (or want a headless one), use the static helpers to create and bind a hidden SDL window context:

SDL_Window* window = nullptr;
SDL_GLContext glContext = nullptr;
raisin::RayraiWindow::createOffscreenGlContext(window, glContext, "rayrai_offscreen");
raisin::RayraiWindow::makeOffscreenContextCurrent(window, glContext);

auto world = std::make_shared<raisim::World>();
raisin::RayraiWindow viewer(world, 640, 480);

The offscreen path does not require an ImGui context. If an application embeds rayrai without ImGui, pass false for hover/click arguments to RayraiWindow::update or drive camera state explicitly. The renderer guards ImGui input access so headless tests and batch image generation can run without creating ImGui state.

Custom visuals and instancing

rayrai renders two categories of content:

• RaiSim objects: The renderer mirrors the objects already in the world.

• Custom visuals: Extra visuals you add explicitly (spheres, boxes, meshes, etc.).

Custom visuals are created through RayraiWindow and returned as Visuals:

auto box = viewer.addVisualBox("marker", 0.1, 0.1, 0.1, 0.2f, 0.6f, 1.0f, 1.0f);
box->setPosition(1.0, 0.0, 0.5);

For repeated geometry, use InstancedVisuals to reduce draw overhead:

auto instanced = viewer.addInstancedVisuals(
  "boxes", raisim::Shape::Box, glm::vec3(0.1f, 0.1f, 0.1f),
  glm::vec4(1.f, 0.2f, 0.2f, 1.f), glm::vec4(0.2f, 0.2f, 1.f, 1.f));
instanced->addInstance(glm::vec3(0.0f, 0.0f, 0.1f), 0.0f);
instanced->addInstance(glm::vec3(0.2f, 0.0f, 0.1f), 1.0f);

If you want to load meshes once and share them across visuals, use raisin::RayraiGlobalAsset and addVisualCustomMesh:

auto assets = std::make_shared<raisin::RayraiGlobalAsset>();
auto meshes = assets->getMeshes("/path/to/model.obj");
auto custom = viewer.addVisualCustomMesh("custom", meshes, glm::vec4(0.9f, 0.9f, 1.0f, 1.0f));
custom->setPosition(0.0, 1.0, 0.5);

Point clouds and coordinate frames

Point clouds and coordinate frames are lightweight debug aids:

auto cloud = viewer.addPointCloud("scan");
auto frame = viewer.addCoordinateFrame("robot_frame");

These objects are rendered alongside the world and can be updated every frame. Call updatePointBuffer() after changing point cloud data.

cloud->positions = {glm::vec3(0, 0, 1), glm::vec3(0.2f, 0.1f, 1.1f)};
cloud->colors = {glm::vec4(0, 1, 0, 1), glm::vec4(1, 0, 0, 1)};
cloud->updatePointBuffer();

Camera control and picking

RayraiWindow manages an internal camera for offscreen rendering. You can access or override it through getCamera() or by using renderWithExternalCamera when you want explicit control.

Picking is available through pickWithExternalCamera. It renders a selection pass and returns the encoded object id for a pixel.

The internal update call drives camera input and picking:

// Cursor is in framebuffer coordinates of the render area
viewer.update(width, height, isHovered, cursorX, cursorY, shouldClick);

Sensor alignment

rayrai can align rendering to RaiSim camera sensors. Use syncRaisimCameraPose and renderWithExternalCamera to ensure the render camera matches the sensor pose and intrinsics. RaiSim sensor rendering is world-object-only: custom visuals, instanced visuals, point clouds, coordinate frames, and other viewer-only helpers are excluded from RGB, depth, and LiDAR data-generation passes. Use generic external-camera rendering only when you intentionally want a viewer render that includes visualization objects. Note: syncRaisimCameraPose updates Camera::position/front/up directly; avoid calling Camera::update() immediately afterward unless you also update yaw/pitch.

For complete runnable examples, see Rayrai RGB camera, Rayrai depth camera, and Rayrai LiDAR point cloud. The broader examples/src/rayrai/getting_started/rayrai_complete_showcase.cpp source combines RGB, depth, raw buffer readback, LiDAR visualization, and camera frustums in one scene. The sensor overview in Sensors includes a longer RGB/depth readback example.

RGB/Depth camera workflow (manual source + external camera):

auto rgbCam = anymal->getSensorSet("d455_front")->getSensor<raisim::RGBCamera>("color");
auto depthCam = anymal->getSensorSet("d455_front")->getSensor<raisim::DepthCamera>("depth");

rgbCam->setMeasurementSource(raisim::Sensor::MeasurementSource::MANUAL);
depthCam->setMeasurementSource(raisim::Sensor::MeasurementSource::MANUAL);

raisin::Camera rgbCamera(*rgbCam);
raisin::Camera depthCamera(*depthCam);

viewer.renderWithExternalCamera(*rgbCam, rgbCamera, {});
viewer.renderWithExternalCamera(*depthCam, depthCamera, {});
viewer.renderDepthPlaneDistance(*depthCam, depthCamera);

You can read back the camera buffers on CPU:

const auto& prop = rgbCam->getProperties();
const int width = std::max(1, prop.width);
const int height = std::max(1, prop.height);
std::vector<char> rgba(size_t(width) * size_t(height) * 4);
rgbCamera.getRawImage(*rgbCam, raisin::Camera::SensorStorageMode::CUSTOM_BUFFER,
  rgba.data(), rgba.size(), /*flipVertical=*/false);

Depth uses a float buffer with width * height entries.

TCP viewer protocol

The rayrai TCP viewer protocol is explicitly versioned. The current viewer sends a protocol header with feature bits before each request, and the server replies with the negotiated feature set. A viewer rejects newer unsupported protocol versions with a clear error instead of attempting to parse an incompatible stream.

Current feature bits cover the explicit header and deformable delta streaming. Deformable objects send mesh topology during initialization or topology changes; ordinary update frames send vertex positions only. This keeps dynamic cloth/cube streaming cheaper while avoiding binary compression until network bandwidth is measured as a bottleneck.

Depth and LiDAR

The renderer supports a linear depth plane and a GPU-assisted LiDAR pass. These sensor passes render RaiSim world objects only; visualization-only objects are intentionally ignored so they cannot leak into training observations.

• renderDepthPlaneDistance renders a linear depth texture.

• measureSpinningLidarSingleDrawGPU renders a LiDAR slice using a spherical chunk shader.

You can retrieve the depth texture via getDepthPlaneTexture().

LiDAR usage has two paths. Prefer the rayrai GPU path when rayrai is available:

1. GPU slice rendering via measureSpinningLidarSingleDrawGPU for fast incremental updates.

2. CPU-based scan via RaiSim (SpinningLidar::update), then visualize with a point cloud, only when rayrai is unavailable or deterministic CPU ray-query behavior is required.

GPU slice example:

lidar->updatePose();
const glm::dvec3 posW = raisin::toGlm(lidar->getPosition());
const glm::dmat3 rotW = raisin::toGlm(lidar->getOrientation());
viewer.measureSpinningLidarSingleDrawGPU(*lidar, posW, rotW);

Performance notes

rayrai is intended to stay single-threaded so it can be embedded in simulation, replay, and data-generation tools without adding renderer worker synchronization. The renderer keeps the main performance controls explicit:

• Disable shadows for high-throughput camera observations when shadows are not part of the desired output. Use RenderOverrides::doShadows = false or disable the light’s shadow map globally.

• Prefer addInstancedVisuals for repeated primitives or meshes. A single instanced visual renders many copies with one instance buffer instead of many separate visual objects.

• Opaque RaiSim single-body primitives are internally batched on non-shadowed color renders. This helps high-throughput observation rendering while keeping shadowed rendering on the conservative per-object path.

• Use InstancedVisuals::setMaxRenderedInstances and PointCloud::setMaxRenderedPoints as simple LOD caps for debug overlays, particles, scans, or dense markers that do not need full density in every frame.

• Keep non-visible debug geometry outside the camera frustum when possible. rayrai performs coarse frustum culling for RaiSim objects, custom visuals, instanced visuals, and point clouds, so off-camera content is skipped before draw submission.

• If the world topology is stable, repeated updateObjectLists() calls are cheap: rayrai refreshes appearances without rebuilding the object cache unless objects were added or removed.

• RGB/depth CPU readback supports an optional PBO-backed asynchronous path. The synchronous path remains the deterministic default for simple RL loops.

• Dynamic point clouds and instanced visuals support partial buffer updates for streaming changes.

• Deformable TCP streaming sends topology/indices only during initialization or topology changes; normal frames send vertex positions only.

Additional tips

• If you add/remove RaiSim objects after constructing the viewer, call RayraiWindow::updateObjectLists() so the renderer refreshes its cache.

• Use setShowCollisionBodies(true) for debug visualization of collision shapes.

• If you want rayrai overlays to use your ImGui font, pass it via setExternalFont.

API

Core types

class RayraiWindow

Offscreen renderer and visual scene manager for raisim.

RayraiWindow maintains an internal Camera, shader set, and a set of visual objects that can be rendered into an offscreen texture. It can render raisim world objects, custom visuals, point clouds, and coordinate frames.

Public Types

enum class RenderQualityPreset

Viewer-quality policy presets.

These presets affect interactive viewer rendering only. They do not change sensor world-object filtering or make sensor outputs include viewer-only visuals.

Values:

enumerator Fast

enumerator Balanced

enumerator High

enumerator Ultra

enumerator Custom

Public Functions

RayraiWindow(std::shared_ptr<raisim::World> world, int width = 300, int height = 300)

Construct an offscreen renderer tied to a raisim world.

Parameters:

• world – Shared world pointer used for pulling object state.

• width – Initial render target width.

• height – Initial render target height.

RayraiWindow(raisim::World &world, int width = 300, int height = 300)

Construct an offscreen renderer tied to a raisim world.

Parameters:

• world – World reference used for pulling object state.

• width – Initial render target width.

• height – Initial render target height.

void setAsyncMeshLoadingEnabled(bool enabled)

Enable or disable async mesh file import for future mesh visuals/world assets.

Async loading parses mesh files on a worker thread, skips drawing pending assets, and finalizes ready GPU resources on the render thread. Disable it when a caller needs mesh assets to be available immediately after addVisualMesh()/importVisualScene().

bool asyncMeshLoadingEnabled() const

True when future mesh file imports are started asynchronously.

size_t pollAsyncMeshLoads(size_t maxAssets = 1)

Finalize ready async mesh imports on the render thread.

Parameters:

maxAssets – Maximum number of assets to finalize.

Returns:

Number of assets finalized.

size_t pendingAsyncMeshLoadCount() const

Number of async mesh imports still waiting for CPU load or GPU finalization.

void resize(int newWidth, int newHeight)

Resize the internal camera render target.

Parameters:

• newWidth – New framebuffer width in pixels.

• newHeight – New framebuffer height in pixels.

void update(int w, int h, bool isHovering, int cX, int cY, bool shouldClick)

Render one frame into the internal camera’s offscreen texture.

Parameters:

• w – Target width.

• h – Target height.

• isHovering – Whether the mouse is over the render area (for input).

• cX – Cursor x position inside the render area.

• cY – Cursor y position inside the render area.

• shouldClick – Whether mouse click selection should be processed.

unsigned int getImageTexture() const

Return the GL texture id of the internal camera output.

Returns:

Texture id for the most recent rendered frame.

void setCameraFixedDistance(bool fixed)

Keep the camera distance constant when orbiting a target.

Parameters:

fixed – True to lock distance.

void setCameraFixedTarget(bool fixed)

Keep the camera target fixed in world space.

Parameters:

fixed – True to lock the target position.

void setBackgroundColor(const glm::vec4 &color)

Set background clear color using legacy 0..255 RGBA channels.

Prefer setBackgroundColorRgb255() or setBackgroundColorLinear() in new code.

Parameters:

color – RGBA color where each channel is in [0, 255].

void setBackgroundColorRgb255(const glm::vec4 &color)

Set background clear color using 0..255 RGBA channels.

Parameters:

color – RGBA color where each channel is in [0, 255].

void setBackgroundColorLinear(const glm::vec4 &color)

Set background clear color using linear 0..1 RGBA channels.

Parameters:

color – Linear RGBA color where each channel is in [0, 1].

inline void setEnvironmentBackground(unsigned int environmentMap, float exposure = 1.0f)

Render a cubemap behind the interactive viewer camera.

Parameters:

• environmentMap – Cubemap texture id, or 0 to disable the environment background.

• exposure – Multiplier applied while drawing the environment background.

inline void clearEnvironmentBackground()

Disable the interactive viewer environment background.

void setFogDensity(float density)

Set scene fog density used by the mesh shader.

Parameters:

density – Exponential fog density.

float getFogDensity() const

Get current fog density.

Returns:

Fog density value.

Camera &getCamera()

Access the internal camera for direct manipulation.

Returns:

Reference to the internal camera.

inline const Visuals *getTargetVisual() const

Get the currently selected target visual, if any.

Returns:

Pointer to the target visual or nullptr.

void setTargetVisual(Visuals *visual)

Set the target visual for camera tracking.

Parameters:

visual – Visual to target, or nullptr to clear.

void setShadowOrtho(float halfSize, float zNear, float zFar)

Set the directional shadow box (orthographic) parameters.

Parameters:

• halfSize – Half-size of the ortho box.

• zNear – Near plane.

• zFar – Far plane.

void setShadowCenterOffset(float meters)

Set the shadow center offset distance along the camera forward vector.

Parameters:

meters – Distance ahead of the camera.

void requestShadowUpdate()

Mark viewer shadow maps stale so they are rebuilt on the next viewer frame.

This is useful when RenderQualitySettings::updateShadowsEveryFrame is false and the application moves static-looking lights or scene geometry manually.

void setUpdateShadowsEveryFrame(bool enabled)

Runtime shortcut for RenderQualitySettings::updateShadowsEveryFrame.

Parameters:

enabled – True to refresh shadow maps every viewer frame, false to update on demand.

inline bool updateShadowsEveryFrame() const

True when viewer shadows are refreshed every frame.

inline float getShadowCenterOffset() const

Get the current shadow center offset distance.

Returns:

Distance ahead of the camera.

void renderWithExternalCamera(raisim::RGBCamera &sensorCam, Camera &externalCam, const RenderOverrides &ov)

Render using a raisim RGB camera and an external render camera.

Parameters:

• sensorCam – Raisim RGB camera providing projection parameters and pose.

• externalCam – Render camera to use for rendering.

• ov – Optional shader/toggle overrides.

void renderWithExternalCamera(raisim::DepthCamera &sensorCam, Camera &externalCam, const RenderOverrides &ov)

Render using a raisim depth camera and an external render camera.

Parameters:

• sensorCam – Raisim depth camera providing projection parameters and pose.

• externalCam – Render camera to use for rendering.

• ov – Optional shader/toggle overrides.

void renderWithExternalCamera(Camera &externalCam, const RenderOverrides &ov)

Render using an external render camera.

Parameters:

• externalCam – Render camera to use for rendering.

• ov – Optional shader/toggle overrides.

unsigned int captureReflectionProbeCubemap(const glm::vec3 &position)

Capture a static local reflection probe into a cubemap.

This is a raster, load-time/manual operation. It renders six 90-degree faces from position and returns an OpenGL cubemap id with mipmaps. The caller owns the returned texture and may bind it to PBR materials through Visuals or Material.

Returns:

OpenGL cubemap id, or 0 on failure.

uint32_t pickWithExternalCamera(Camera &externalCam, int x, int y, Shader *pickingOverride = nullptr)

Run the picking pass and return object id at a pixel.

Parameters:

• externalCam – Render camera used for the picking pass.

• x – Pixel x coordinate in framebuffer space.

• y – Pixel y coordinate in framebuffer space.

• pickingOverride – Optional picking shader override.

Returns:

Encoded object id, or 0 if nothing is picked.

std::shared_ptr<PointCloud> addPointCloud(const std::string &name)

Create a named point cloud.

Parameters:

name – Unique name for the point cloud.

Returns:

Shared pointer to the point cloud instance.

void removePointCloud(const std::string &name)

Remove a point cloud by name.

Parameters:

name – Name of the point cloud to remove.

std::shared_ptr<CoordinateFrame> addCoordinateFrame(const std::string &name)

Create a named coordinate frame set.

Parameters:

name – Unique name for the coordinate frame set.

Returns:

Shared pointer to the coordinate frame instance.

void removeCoordinateFrame(const std::string &name)

Remove a coordinate frame set by name.

Parameters:

name – Name of the coordinate frame set to remove.

std::shared_ptr<Visuals> addVisualSphere(const std::string &name, double radius, float colorR = 1.f, float colorG = 1.f, float colorB = 1.f, float colorA = 1.f)

Add a visual sphere (not part of raisim::World).

Parameters:

• name – Visual name.

• radius – Sphere radius.

• colorR – Red channel.

• colorG – Green channel.

• colorB – Blue channel.

• colorA – Alpha channel.

Returns:

Shared pointer to the created visual.

std::shared_ptr<Visuals> addVisualBox(const std::string &name, double xLength, double yLength, double zLength, float colorR = 1.f, float colorG = 1.f, float colorB = 1.f, float colorA = 1.f)

Add a visual box (not part of raisim::World).

Parameters:

• name – Visual name.

• xLength – Box size in X.

• yLength – Box size in Y.

• zLength – Box size in Z.

• colorR – Red channel.

• colorG – Green channel.

• colorB – Blue channel.

• colorA – Alpha channel.

Returns:

Shared pointer to the created visual.

std::shared_ptr<Visuals> addVisualCylinder(const std::string &name, double radius, double height, float colorR = 1.f, float colorG = 1.f, float colorB = 1.f, float colorA = 1.f)

Add a visual cylinder (not part of raisim::World).

Parameters:

• name – Visual name.

• radius – Cylinder radius.

• height – Cylinder height.

• colorR – Red channel.

• colorG – Green channel.

• colorB – Blue channel.

• colorA – Alpha channel.

Returns:

Shared pointer to the created visual.

std::shared_ptr<Visuals> addVisualCapsule(const std::string &name, double radius, double height, float colorR = 1.f, float colorG = 1.f, float colorB = 1.f, float colorA = 1.f)

Add a visual capsule (not part of raisim::World).

Parameters:

• name – Visual name.

• radius – Capsule radius.

• height – Capsule height.

• colorR – Red channel.

• colorG – Green channel.

• colorB – Blue channel.

• colorA – Alpha channel.

Returns:

Shared pointer to the created visual.

std::shared_ptr<Visuals> addVisualMesh(const std::string &name, const std::string &meshFile, double scaleX = 1.0, double scaleY = 1.0, double scaleZ = 1.0, float colorR = 1.f, float colorG = 1.f, float colorB = 1.f, float colorA = 1.f)

Add a visual mesh (not part of raisim::World).

Parameters:

• name – Visual name.

• meshFile – Mesh file path.

• scaleX – Mesh scale in X.

• scaleY – Mesh scale in Y.

• scaleZ – Mesh scale in Z.

• colorR – Red channel.

• colorG – Green channel.

• colorB – Blue channel.

• colorA – Alpha channel.

Returns:

Shared pointer to the created visual.

std::shared_ptr<Visuals> addVisualPlane(const std::string &name, double width, double height, float colorR = 1.f, float colorG = 1.f, float colorB = 1.f, float colorA = 1.f)

Add a custom visual plane backed by the shared primitive plane mesh.

std::shared_ptr<Visuals> addVisualCustomMesh(const std::string &name, const std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> &meshes, const glm::vec4 &color = glm::vec4(1.f, 1.f, 1.f, 1.f))

Add a visual from custom OpenGL meshes.

Parameters:

• name – Visual name.

• meshes – Mesh list to render.

• color – Default color for non-textured meshes.

Returns:

Shared pointer to the created visual.

std::shared_ptr<Visuals> getVisualObject(const std::string &name)

Look up a visual by name.

Parameters:

name – Visual name.

Returns:

Shared pointer to the visual, or nullptr if not found.

void removeVisualObject(const std::string &name)

Remove a visual object by name.

Parameters:

name – Visual name to remove.

std::shared_ptr<InstancedVisuals> addInstancedVisuals(const std::string &name, raisim::Shape::Type type, const glm::vec3 &size, const glm::vec4 &color1, const glm::vec4 &color2, const std::string &meshFile = "")

Create instanced visuals for repeated shapes or meshes.

Parameters:

• name – Visual name.

• type – Shape type.

• size – Base size for the instances.

• color1 – Primary instance color.

• color2 – Secondary instance color.

• meshFile – Mesh file for mesh instances.

Returns:

Shared pointer to the created instanced visuals.

std::shared_ptr<InstancedVisuals> getInstancedVisuals(const std::string &name)

Look up instanced visuals by name.

Parameters:

name – Visual name.

Returns:

Shared pointer to the instanced visuals, or nullptr if not found.

void removeInstancedVisuals(const std::string &name)

Remove instanced visuals by name.

Parameters:

name – Visual name to remove.

std::shared_ptr<CameraFrustum> addCameraFrustum(const std::string &name, const glm::vec4 &color = glm::vec4(1.f, 1.f, 1.f, 1.f))

Create a camera frustum visualization.

Parameters:

• name – Visual name.

• color – Frustum color.

Returns:

Shared pointer to the frustum helper.

inline void setGroundPatternResourcePath(const std::string &path)

Set a tiled ground texture for halfspace visuals.

Parameters:

path – Path to the texture image.

inline std::string getGroundPatternResourcePath()

Get the current ground texture path.

Returns:

Ground texture path string.

void updateObjectLists()

Refresh internal lists of raisim objects.

void orthogonalViewSwitch()

Toggle between perspective and orthographic view.

void renderDepthPlaneDistance(raisim::DepthCamera &sensorCam, Camera &cam, Shader *depthPlaneOverride = nullptr)

Render a linear depth plane for a raisim depth camera.

Parameters:

• sensorCam – Raisim depth camera.

• cam – Render camera to use.

• depthPlaneOverride – Optional shader override.

void renderDepthPlaneDistance(Camera &cam, Shader *depthPlaneOverride = nullptr)

Render a linear depth plane with a given camera.

Parameters:

• cam – Render camera to use.

• depthPlaneOverride – Optional shader override.

inline unsigned int getDepthPlaneTexture() const

Get the linear depth texture id.

Returns:

OpenGL texture id for R32F linear depth.

void measureSpinningLidarSingleDrawGPU(raisim::SpinningLidar &lidar, const glm::dvec3 &sensorPosW, const glm::dmat3 &sensorRotW, const raisim::Object *objectToExclude = nullptr)

Render a single-draw incremental GPU LiDAR slice.

Renders the largest contiguous yaw span for this timestep using a spherical chunk shader.

Parameters:

• lidar – Raisim spinning lidar to update.

• sensorPosW – LiDAR position in world coordinates.

• sensorRotW – LiDAR orientation in world coordinates.

• objectToExclude – Optional object to exclude from rendering (e.g., the carrier).

inline Light &getLight()

Access the main scene light.

Returns:

Reference to the main light.

void addAdditionalLight(const AdditionalLight &light)

Add a non-shadow-casting light for visual quality.

Parameters:

light – Additional light. At most kMaxAdditionalLights are used.

void clearAdditionalLights()

Remove all non-shadow-casting additional lights.

inline size_t additionalLightCount() const

Return the current number of additional lights.

Returns:

Number of non-shadow-casting additional lights.

bool promoteDominantAdditionalLightToMainShadowCaster(bool preferSpotLights = true, bool removePromotedAdditionalLight = true)

Promote the dominant imported/local light to the single shadow-casting main light.

This is intended for authored glTF scenes that have a clear lamp/spot light. It keeps low-quality rendering fast because no additional shadow maps are created.

Parameters:

• preferSpotLights – Prefer spot lights over other light types when possible.

• removePromotedAdditionalLight – Remove the promoted light from the non-shadow list to avoid double-counting its direct contribution.

Returns:

True if a suitable light was promoted.

size_t importSceneLights(const std::string &sceneFile, float intensityScale = 1.0f)

Import authored scene lights from a mesh/scene file through Assimp.

Imported lights are appended as bounded, non-shadow-casting viewer lights. This is explicit so low-quality rendering does not pay for scene lights unless an application imports them.

Returns:

Number of lights appended, capped by kMaxAdditionalLights.

void setRenderQualityPreset(RenderQualityPreset preset)

Apply a named viewer-quality preset at runtime.

Parameters:

preset – Quality preset to apply.

inline RenderQualityPreset getRenderQualityPreset() const

Return the active named quality preset, or Custom after explicit settings.

void setRenderQualitySettings(const RenderQualitySettings &settings)

Apply explicit viewer-quality settings at runtime.

Parameters:

settings – Settings to apply.

inline const RenderQualitySettings &getRenderQualitySettings() const

Return the active explicit viewer-quality settings.

void printRenderDiagnostics(std::ostream &out = std::cout) const

Print active viewer-quality, light, shadow, and post-process diagnostics.

void writeRenderDiagnosticsJson(std::ostream &out, const SceneImportReport *importReport = nullptr) const

Write structured render diagnostics as JSON.

The JSON is intended for regression tests, issue reports, and imported-scene debugging. It contains render-quality settings, scene/resource counts, lights, shadow selection state, postprocess state, and an optional scene-import report.

std::string renderDiagnosticsJson(const SceneImportReport *importReport = nullptr) const

Return structured render diagnostics as a JSON string.

inline void setDOFEnabled(bool e)

Enable or disable depth of field post-processing.

Parameters:

e – True to enable.

inline void setDOFParams(float focusDist, float focusRange, float maxRadius)

Set depth of field parameters.

Parameters:

• focusDist – Focus distance in view space.

• focusRange – Focus range around the focus distance.

• maxRadius – Max blur radius in pixels.

inline void setShowCollisionBodies(bool show)

Toggle rendering of collision shapes for articulated systems.

Parameters:

show – True to render collision bodies.

inline bool getShowCollisionBodies() const

Return collision-body rendering state.

Returns:

True if collision bodies are shown.

void setExternalFont(ImFont *font)

Provide an ImGui font to use for the viewer overlay.

Parameters:

font – Pointer to an ImGui font.

inline ImFont *getExternalFont() const

Retrieve the external ImGui font pointer.

Returns:

Font pointer or nullptr.

Public Static Functions

static unsigned int loadTextureWithTiling(const char *path, bool useSrgbFormat = false)

Load an image file into an OpenGL texture with repeat tiling.

Parameters:

• path – Path to an image file supported by stb_image.

• useSrgbFormat – If true, RGB/RGBA textures are uploaded as sRGB color textures. Use true for albedo/emissive color maps and false for data maps such as normal, metallic-roughness, AO, depth, and masks.

Returns:

OpenGL texture id, or 0 on failure.

static unsigned int loadHdrEquirectangularCubemap(const char *path, int faceSize = 128, bool generateMipmaps = true)

Load a Radiance .hdr equirectangular environment into a cubemap texture.

This is intended for High/Ultra viewer PBR workflows. It does not allocate anything unless the application explicitly calls it.

Parameters:

• path – Path to a .hdr image supported by stb_image.

• faceSize – Cubemap face size in pixels.

• generateMipmaps – Whether to generate mip levels for roughness sampling.

Returns:

OpenGL cubemap id, or 0 on failure.

static unsigned int createHdrIrradianceCubemap(const char *path, int faceSize = 32, int sampleCount = 256)

Generate a diffuse irradiance cubemap from a Radiance .hdr equirectangular environment.

This is CPU-side cosine-hemisphere integration intended for High/Ultra PBR setup. Creation is explicit so Fast/Balanced RL rendering does not pay the cost.

Parameters:

• path – Path to a .hdr image supported by stb_image.

• faceSize – Irradiance cubemap face size in pixels.

• sampleCount – Number of hemisphere samples per texel.

Returns:

OpenGL cubemap id, or 0 on failure.

static unsigned int createHdrPrefilteredEnvironmentCubemap(const char *path, int faceSize = 128, int mipLevels = 5, int sampleCount = 256)

Generate a prefiltered specular environment cubemap from a Radiance .hdr equirectangular environment.

Mip level 0 is sharp and higher mip levels are GGX-prefiltered with increasing roughness. Use this as Material::prefilteredEnvironmentMap together with createSplitSumBrdfLut().

Parameters:

• path – Path to a .hdr image supported by stb_image.

• faceSize – Base cubemap face size in pixels.

• mipLevels – Number of mip levels to allocate and integrate.

• sampleCount – Number of GGX samples per texel.

Returns:

OpenGL cubemap id, or 0 on failure.

static unsigned int createSplitSumBrdfLut(int size = 256, int sampleCount = 512)

Create a numerically integrated split-sum BRDF LUT.

Use this as Material::brdfLut for High/Ultra PBR. Creation is explicit so Fast/Balance paths do not pay for it.

static void setTextureAnisotropy(unsigned int target, unsigned int textureId, float requestedLevel)

Apply anisotropic filtering to an existing 2D or cubemap texture if supported.

Parameters:

• target – Texture target such as GL_TEXTURE_2D or GL_TEXTURE_CUBE_MAP.

• textureId – OpenGL texture id.

• requestedLevel – Requested anisotropy level. Values <= 1 leave the texture unchanged.

static void setTextureSamplerState(unsigned int target, unsigned int textureId, int wrapS, int wrapT, int minFilter, int magFilter)

Apply sampler wrap/filter state to an existing texture.

static inline unsigned int loadColorTextureWithTiling(const char *path)

Load an albedo/emissive color texture as sRGB.

Parameters:

path – Path to an image file supported by stb_image.

Returns:

OpenGL texture id, or 0 on failure.

static inline unsigned int loadDataTextureWithTiling(const char *path)

Load a data texture as linear values.

Parameters:

path – Path to an image file supported by stb_image.

Returns:

OpenGL texture id, or 0 on failure.

static void createOffscreenGlContext(SDL_Window *&window, SDL_GLContext &glContext, const char *title = "rayrai_offscreen")

Create a hidden SDL window and an offscreen OpenGL context.

Parameters:

• window – Output pointer for the created SDL window.

• glContext – Output handle for the OpenGL context.

• title – Window title for the hidden SDL window.

static void makeOffscreenContextCurrent(SDL_Window *window, SDL_GLContext glContext)

Make an existing offscreen context current and update glbinding.

Parameters:

• window – SDL window owning the OpenGL context.

• glContext – OpenGL context to make current.

static RenderQualitySettings defaultRenderQualitySettings(RenderQualityPreset preset)

Return the default settings for a named preset.

Parameters:

preset – Preset to query.

struct AdditionalLight

Lightweight imported or viewer light used for extra fill/rim lighting.

Extra lights are shading-only by default and are capped to keep the fast one-light path available. One of them can be promoted to the main shadow-casting light.

struct ReflectionProbe

struct ReflectionProbeCaptureSettings

Explicit static reflection-probe capture settings.

Probe capture is never automatic. Applications call captureReflectionProbeCubemap() when they want a bounded real-time local reflection approximation.

struct RenderOverrides

Override shaders and rendering toggles when using external cameras.

Public Members

Shader *mesh

Mesh shader override (defaults to shaders_[“mesh”]).

Shader *pointcloud

Point cloud shader override (defaults to shaders_[“pointcloud”]).

Shader *picking

Picking shader override (defaults to shaders_[“picking”]).

Shader *post

Post-process shader override (defaults to shaders_[“post”]).

bool doShadows

Enable shadow passes.

bool drawCoordinateFrames

Draw coordinate frames.

bool drawPointClouds

Draw point clouds.

bool drawVisualizationObjects

Draw visualization-only custom visuals and instanced visuals.

bool skipGroundObjects

Skip halfspace and heightmap ground objects in scene-color rendering.

unsigned int environmentBackgroundMap

Optional cubemap rendered as a viewer background before scene geometry.

float environmentBackgroundExposure

Exposure applied to environmentBackgroundMap when rendered as background.

bool postProcess

Apply external-camera postprocessing after rendering.

struct RenderQualitySettings

Explicit runtime quality settings for viewer rendering.

struct SceneImportReport

class Camera

Camera for offscreen rendering and interaction.

The camera owns its render targets (scene color/depth, post-process, linear depth, and picking). It can be driven by user input or by raisim sensor parameters.

Public Types

enum class ProjectionMode

Projection mode for view generation.

Values:

enumerator PERSPECTIVE

enumerator ORTHOGRAPHIC

enum class SensorStorageMode

Storage mode for extracting rendered images.

Values:

enumerator SENSOR_SETTER

enumerator CUSTOM_BUFFER

Public Functions

explicit Camera(glm::vec3 position = glm::vec3(5.0f, 4.0f, 5.0f), glm::vec3 up = glm::vec3(0.0f, 0.0f, 1.0f), float yaw = -135.0f, float pitch = -30.0f)

Construct a camera with a position, up vector, and yaw/pitch.

Parameters:

• position – Camera position in world coordinates.

• up – World up vector.

• yaw – Initial yaw angle in degrees.

• pitch – Initial pitch angle in degrees.

explicit Camera(raisim::RGBCamera &rgbCam)

Construct a camera from a raisim RGB camera (and allocate targets).

Parameters:

rgbCam – Raisim RGB camera to mirror.

explicit Camera(const raisim::RGBCamera &rgbCam)

Construct a camera from a const raisim RGB camera.

Parameters:

rgbCam – Raisim RGB camera to mirror.

explicit Camera(raisim::DepthCamera &depthCam)

Construct a camera from a raisim depth camera (and allocate targets).

Parameters:

depthCam – Raisim depth camera to mirror.

explicit Camera(const raisim::DepthCamera &depthCam)

Construct a camera from a const raisim depth camera.

Parameters:

depthCam – Raisim depth camera to mirror.

void update(bool processKeyboard = true)

Update camera orientation and handle input.

Parameters:

processKeyboard – If true, WASD + vertical keys affect position.

glm::mat4 getViewMatrix() const

Get the view matrix for the current camera pose.

Returns:

View matrix in column-major GLM format.

glm::mat4 getProjectionMatrix() const

Get the projection matrix for the current camera state.

Returns:

Projection matrix in column-major GLM format.

inline glm::vec3 getPosition() const

Get the current camera position.

Returns:

Position in world coordinates.

void mouseDelta(float yawOffset, float pitchOffset)

Apply mouse delta to yaw/pitch or pan in orthographic mode.

Parameters:

• yawOffset – Delta yaw in degrees (screen-space delta).

• pitchOffset – Delta pitch in degrees (screen-space delta).

void moveForward(float steps)

Move forward/backward or zoom in orthographic mode.

Parameters:

steps – Positive moves forward/zooms in; negative moves back/zooms out.

void setCameraFixedDistance(bool fixed)

Lock camera distance while orbiting.

Parameters:

fixed – True to keep distance constant.

void setCameraFixedTarget(bool fixed)

Lock the target position to a fixed point.

Parameters:

fixed – True to lock to the current target.

void setTargetObject(raisim::Object *target)

Set a raisim object to track as the camera target.

Parameters:

target – Raisim object to follow, or nullptr to clear.

void setProjectionMode(ProjectionMode mode)

Set projection mode.

Parameters:

mode – Projection mode (perspective or orthographic).

ProjectionMode getProjectionMode() const

Get the active projection mode.

Returns:

Current projection mode.

glm::vec3 getTargetPosition() const

Get the current target position for orbiting.

Returns:

Target position in world coordinates.

void ensureRenderTargets(int width, int height)

Ensure render targets match the given size.

Parameters:

• width – Render target width.

• height – Render target height.

inline void resizeRenderTargets(int width, int height)

Resize render targets (alias for ensureRenderTargets).

Parameters:

• width – Render target width.

• height – Render target height.

void ensurePostRenderTarget()

Ensure the post-process render target exists.

void ensureLinearDepthRenderTarget()

Ensure the linear-depth render target exists.

void bindSceneFbo()

Bind the scene framebuffer for rendering.

void bindPostFbo()

Bind the post-process framebuffer for rendering.

void bindLinearDepthFbo()

Bind the linear-depth framebuffer for rendering.

inline int rtWidth() const

Current render target width.

Returns:

Width in pixels.

inline int rtHeight() const

Current render target height.

Returns:

Height in pixels.

void beginSceneFboAndClear()

Bind scene FBO and clear with background color.

void postProcessDoF(Shader &postShader, bool fxaaEnabled = false, bool bloomEnabled = false, float bloomThreshold = 0.82f, float bloomStrength = 0.18f, float bloomRadius = 4.0f, float bloomKnee = 0.22f, int bloomQuality = 1, bool screenSpaceAoEnabled = false, float screenSpaceAoRadius = 2.0f, float screenSpaceAoStrength = 0.0f, float screenSpaceAoBias = 0.02f)

Apply viewer post-processing.

Parameters:

• postShader – Shader configured for DoF, FXAA, and bloom.

• fxaaEnabled – True to apply FXAA even when DoF is disabled.

void ensurePickFramebuffer()

Ensure the picking framebuffer is created.

uint32_t readPickIdAt(int x, int y) const

Read the encoded pick id at a pixel coordinate.

Parameters:

• x – X coordinate in framebuffer space.

• y – Y coordinate in framebuffer space.

Returns:

Encoded object id.

inline unsigned int getPickFbo() const

Get the picking FBO id.

Returns:

OpenGL framebuffer id.

unsigned int getFinalTexture() const

Get the final texture to display (post or debug).

Returns:

Texture id for display.

inline unsigned int getSceneColorTex() const

Get the scene color texture id.

Returns:

Color texture id.

inline unsigned int getSceneDepthTex() const

Get the scene depth texture id.

Returns:

Depth texture id.

inline unsigned int getLinearDepthTexture() const

Get the linear depth texture id (R32F).

Returns:

Linear depth texture id.

void getRawImage(raisim::RGBCamera &sensor, SensorStorageMode mode = SensorStorageMode::SENSOR_SETTER, char *customBuffer = nullptr, size_t customBufferSize = 0, bool flipVertical = true) const

Read the color buffer into a raisim RGB camera or custom buffer.

Parameters:

• sensor – Raisim RGB camera to fill when using SENSOR_SETTER.

• mode – Storage mode (sensor setter or custom buffer).

• customBuffer – Optional custom buffer for raw RGBA data.

• customBufferSize – Size of custom buffer in bytes.

• flipVertical – Whether to flip rows from GL origin to image origin.

void getRawImage(raisim::DepthCamera &sensor, SensorStorageMode mode = SensorStorageMode::SENSOR_SETTER, float *customBuffer = nullptr, size_t customBufferLength = 0, bool flipVertical = true) const

Read the linear depth buffer into a raisim depth camera or custom buffer.

Parameters:

• sensor – Raisim depth camera to fill when using SENSOR_SETTER.

• mode – Storage mode (sensor setter or custom buffer).

• customBuffer – Optional custom buffer for float depth values.

• customBufferLength – Length of custom buffer in floats.

• flipVertical – Whether to flip rows from GL origin to image origin.

void setAsyncReadbackEnabled(bool enabled, size_t ringSize = 3)

Configure optional PBO-backed readback helpers.

Sensor getRawImage() remains synchronous by default. The async helpers below return the most recent completed frame and therefore may lag by one or more frames.

inline void setBackgroundColor(const glm::vec4 &c)

Set the background clear color using legacy 0..255 RGBA channels.

Prefer setBackgroundColorRgb255() in new code to make the range explicit.

Parameters:

c – RGBA color where each channel is in [0, 255].

inline void setBackgroundColorRgb255(const glm::vec4 &c)

Set the background clear color using 0..255 RGBA channels.

Parameters:

c – RGBA color where each channel is in [0, 255].

inline void setBackgroundColorLinear(const glm::vec4 &c)

Set the background clear color using linear 0..1 RGBA channels.

Parameters:

c – Linear RGBA color where each channel is in [0, 1].

inline glm::vec4 getBackgroundColor() const

Get the background clear color.

Returns:

Legacy RGBA color in [0, 255].

inline glm::vec4 getBackgroundColorLinear() const

Get the background clear color in linear 0..1 channels.

Returns:

Linear RGBA color in [0, 1].

inline void setGamma(float g)

Set gamma correction value.

Parameters:

g – Gamma value.

inline float getGamma() const

Get gamma correction value.

Returns:

Gamma value.

inline void setDOFEnabled(bool e)

Enable or disable DoF.

Parameters:

e – True to enable.

inline bool getDOFEnabled() const

Get DoF enabled state.

Returns:

True if DoF is enabled.

inline void setDOFParams(float focusDist, float focusRange, float maxRadius)

Set DoF parameters.

Parameters:

• focusDist – Focus distance.

• focusRange – Focus range around focus distance.

• maxRadius – Max blur radius.

inline void setDebugShowPick(bool e)

Toggle rendering of the pick buffer for debugging.

Parameters:

e – True to show pick buffer.

inline bool isDebugShowingPick() const

Check if the pick buffer is shown for debugging.

Returns:

True if pick buffer is displayed.

Public Members

glm::vec3 position

Camera pose and view parameters (public for quick access).

raisim::Object *targetObject

Current target raisim object (if any).

Public Static Functions

static void unbindFbo()

Unbind any framebuffer (bind default).

class RayraiGlobalAsset

A caching asset manager for 3D models and primitive meshes.

This class loads models on demand and keeps them in memory as long as they are being used. Once all references to a model are released, its memory can be reclaimed. It also owns reusable meshes for primitives such as cubes, spheres, cylinders, and capsules.

Public Functions

RayraiGlobalAsset()

Construct the asset cache and generate primitive meshes.

~RayraiGlobalAsset()

Destroy the asset cache.

std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> getMeshes(const std::string &name)

Retrieves a model’s meshes by its asset name (usually file path).

Parameters:

name – The unique identifier for the asset, typically the file path.

Returns:

A shared_ptr to a vector of OpenGLMeshes. Returns nullptr if loading fails.

size_t pollAsyncLoads(size_t maxAssets = 1)

Finalize any async mesh imports whose CPU-side Assimp load has completed.

This must be called on the OpenGL/render thread because it creates textures, vertex buffers, and vertex array objects. Returns the number of assets finalized.

inline size_t pendingAsyncLoadCount() const

Number of mesh file imports that have not been finalized yet.

inline void setAsyncMeshLoadingEnabled(bool enabled)

Enable or disable async mesh file import for future getMeshes() calls.

inline bool asyncMeshLoadingEnabled() const

Check whether future mesh file imports start on a worker thread.

std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> getCubeMesh()

Get a unit cube mesh.

Returns:

Shared mesh list for a cube.

std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> getCylinderMesh()

Get a unit cylinder mesh.

Returns:

Shared mesh list for a cylinder.

std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> getCapsuleMesh()

Get a unit capsule mesh.

Returns:

Shared mesh list for a capsule.

std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> getSphereMesh()

Get a unit sphere mesh.

Returns:

Shared mesh list for a sphere.

std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> getPlaneMesh()

Get a unit plane mesh.

Returns:

Shared mesh list for a plane.

Visuals and geometry

class Visuals

Renderable visual object not owned by raisim::World.

Visuals encapsulate geometry, transform, and appearance for custom renderable shapes or meshes.

Public Functions

Visuals(raisim::Shape::Type type, const std::string &name, const std::shared_ptr<RayraiGlobalAsset> &assets, const std::string &meshFile = "")

Construct a visual from a built-in shape or mesh file.

Parameters:

• type – Raisim shape type to render.

• name – Visual name.

• assets – Global asset cache for primitive meshes.

• meshFile – Mesh file path when type is Mesh.

Visuals(const std::string &name, const std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> &meshes)

Construct a visual from custom OpenGL meshes.

Parameters:

• name – Visual name.

• meshes – Mesh list to render.

void setSphereSize(double radius)

Set sphere radius (uniform scale).

Parameters:

radius – Sphere radius.

void setBoxSize(double x, double y, double z)

Set box dimensions.

Parameters:

• x – Box size in X.

• y – Box size in Y.

• z – Box size in Z.

void setCylinderSize(double radius, double height)

Set cylinder dimensions.

Parameters:

• radius – Cylinder radius.

• height – Cylinder height.

void setCapsuleSize(double radius, double height)

Set capsule dimensions.

Parameters:

• radius – Capsule radius.

• height – Capsule height.

void setMeshScale(double x, double y, double z)

Set mesh scale factors.

Parameters:

• x – Scale in X.

• y – Scale in Y.

• z – Scale in Z.

void setPosition(double x, double y, double z)

Set position from components.

Parameters:

• x – X position.

• y – Y position.

• z – Z position.

void setPosition(const glm::vec3 &pos)

Set position from a vector.

Parameters:

pos – Position vector.

void setOrientation(double w, double x, double y, double z)

Set orientation from wxyz quaternion.

Parameters:

• w – Quaternion w.

• x – Quaternion x.

• y – Quaternion y.

• z – Quaternion z.

void setOrientation(const glm::vec4 &ori)

Set orientation from wxyz quaternion vector.

Parameters:

ori – Quaternion (w, x, y, z).

void setColor(double r, double g, double b, double a)

Set object color from components.

Parameters:

• r – Linear red channel in [0, 1].

• g – Linear green channel in [0, 1].

• b – Linear blue channel in [0, 1].

• a – Alpha channel in [0, 1].

void setColor(const glm::vec4 &color)

Set object color from vector.

Parameters:

color – Linear RGBA color in [0, 1].

inline void setUseMeshColor(bool use)

Use mesh-provided colors instead of object color.

Parameters:

use – True to use mesh colors.

inline bool usesMeshColor() const

Check if mesh colors are used.

Returns:

True if mesh colors are used.

inline void setFlatShading(bool enabled)

Force flat shading for this visual.

Parameters:

enabled – True to use flat shading.

inline bool isFlatShading() const

Check if flat shading is forced.

Returns:

True if flat shading is used.

inline void setTwoSided(bool enabled)

Disable face culling while drawing this visual.

Parameters:

enabled – True to draw both sides of mesh triangles.

inline bool isTwoSided() const

Check if face culling is disabled for this visual.

Returns:

True if both sides are drawn.

inline void setPbrEnvironment(unsigned int environmentMap, unsigned int brdfLut = 0, float strength = 1.0f)

Apply an image-based lighting environment to all PBR submeshes.

inline void setPbrEnvironment(unsigned int environmentMap, unsigned int irradianceMap, unsigned int prefilteredEnvironmentMap, unsigned int brdfLut, float strength = 1.0f)

Apply split image-based lighting maps to all PBR submeshes.

This mirrors the real-time IBL layout used by engines such as Filament: a display/environment cubemap, diffuse irradiance, prefiltered specular radiance, and a split-sum BRDF LUT. Supplying the full set gives whole-scene imported visuals the same high-fidelity path used by hand-built PBR meshes.

inline void setCategory(VisualCategory category)

Set visual category for detection or filtering.

Parameters:

category – Category value.

inline void setDetectable(bool detectable)

Convenience helper to set detectability.

Parameters:

detectable – True to mark as detectable.

inline glm::vec3 getPosition() const

Get current position.

Returns:

Position vector.

inline glm::vec4 getColor() const

Get current color.

Returns:

RGBA color.

inline VisualCategory getCategory() const

Get visual category.

Returns:

Category value.

inline bool isDetectable() const

Check if visual is marked detectable.

Returns:

True if detectable.

inline void setCastsShadows(bool castsShadows)

Enable or disable rendering this visual into shadow maps.

Large receiver-only helper surfaces can keep color rendering and shadow receiving while staying out of the shadow caster pass.

inline bool castsShadows() const

Check if this visual should cast shadows.

inline bool isTransparent() const

Check if visual is translucent based on alpha.

Returns:

True if alpha < 1.0.

inline bool isValid() const

Check if the visual has valid mesh data.

Returns:

True if valid.

inline const std::string &getName() const

Get the visual name used by the renderer.

float approximateRadius() const

Conservative radius around getPosition() for coarse culling.

bool approximateBounds(glm::vec3 &center, float &radius) const

Conservative world-space bounds for coarse culling.

Parameters:

• center – Output sphere center.

• radius – Output sphere radius.

Returns:

True if bounds are available.

bool hasTextures() const

Check if any mesh has textures bound.

Returns:

True if a mesh has textures.

void draw(Shader &shader, OpenGLMesh::RenderMode mode = OpenGLMesh::RenderMode::All) const

Draw the visual using a shader.

Parameters:

shader – Shader to use for drawing.

void drawSorted(Shader &shader, const glm::vec3 &viewPos, OpenGLMesh::RenderMode mode = OpenGLMesh::RenderMode::All) const

Draw submeshes back-to-front relative to a camera position.

This is intended for transparent mesh visuals. It is more expensive than draw(), so rayrai only uses it on the transparent visual path.

uint64_t materialSortKey(OpenGLMesh::RenderMode mode) const

Compact key for sorting opaque visual draws by material/texture state.

void setMeshes(const std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> &meshes)

Replace mesh data for this visual.

Parameters:

meshes – New mesh list to render.

bool updateSingleMeshVertices(const std::vector<Vertex> &vertices)

Update the first mesh’s vertex buffer while preserving its index buffer.

Parameters:

vertices – New vertex data for the existing mesh.

Returns:

True when a single mesh was present and updated.

class InstancedVisuals

Public Functions

InstancedVisuals(raisim::Shape::Type type, const std::string &name, const glm::vec3 &size, const glm::vec4 &color1, const glm::vec4 &color2, const std::shared_ptr<RayraiGlobalAsset> &assets, const std::string &meshFile = "")

Construct an instanced visual for repeated shapes/meshes.

Parameters:

• type – Shape type to instance.

• name – Visual name.

• size – Base size for instances.

• color1 – Primary color.

• color2 – Secondary color.

• assets – Global asset cache for primitive meshes.

• meshFile – Mesh file path when type is Mesh.

inline size_t count() const

Get number of instances.

Returns:

Instance count.

void clearInstances()

Clear all instances.

void resize(size_t count)

Resize instance storage.

Parameters:

count – New instance count.

void addInstance(const glm::vec3 &pos, const glm::vec4 &ori, const glm::vec3 &scale, float colorWeight = 0.f)

Add a fully specified instance.

Parameters:

• pos – Position.

• ori – Orientation quaternion (wxyz).

• scale – Per-instance scale.

• colorWeight – Blend between color1 and color2.

void addInstance(const glm::vec3 &pos, const glm::vec4 &ori, float colorWeight = 0.f)

Add an instance with default scale.

Parameters:

• pos – Position.

• ori – Orientation quaternion (wxyz).

• colorWeight – Blend between color1 and color2.

void addInstance(const glm::vec3 &pos, float colorWeight = 0.f)

Add an instance at position with identity orientation.

Parameters:

• pos – Position.

• colorWeight – Blend between color1 and color2.

void setPosition(size_t id, const glm::vec3 &pos)

Update position of an instance.

Parameters:

• id – Instance index.

• pos – New position.

void setOrientation(size_t id, const glm::vec4 &ori)

Update orientation of an instance.

Parameters:

• id – Instance index.

• ori – Orientation quaternion (wxyz).

void setScale(size_t id, const glm::vec3 &scale)

Update scale of an instance.

Parameters:

• id – Instance index.

• scale – New scale.

void setColor(const glm::vec4 &color1, const glm::vec4 &color2)

Update the two base colors used for blending.

Parameters:

• color1 – Primary linear RGBA color in [0, 1].

• color2 – Secondary linear RGBA color in [0, 1].

void setColorWeight(size_t id, float weight)

Set per-instance blend weight.

Parameters:

• id – Instance index.

• weight – Blend weight (0..1).

inline void setCategory(VisualCategory category)

Set visual category for detection or filtering.

Parameters:

category – Category value.

inline void setDetectable(bool detectable)

Convenience helper to set detectability.

Parameters:

detectable – True to mark as detectable.

glm::vec3 getPosition(size_t id) const

Get instance position.

Parameters:

id – Instance index.

Returns:

Position vector.

glm::vec4 getOrientation(size_t id) const

Get instance orientation (wxyz).

Parameters:

id – Instance index.

Returns:

Quaternion vector.

glm::vec3 getScale(size_t id) const

Get instance scale.

Parameters:

id – Instance index.

Returns:

Scale vector.

inline const glm::vec4 &getColor1() const

Get primary color.

Returns:

Primary color.

inline const glm::vec4 &getColor2() const

Get secondary color.

Returns:

Secondary color.

inline VisualCategory getCategory() const

Get visual category.

Returns:

Category value.

inline bool isDetectable() const

Check if visuals are marked detectable.

Returns:

True if detectable.

inline bool isTransparent() const

Check if any color is translucent.

Returns:

True if alpha < 1.0 for any color.

inline bool isValid() const

Check if meshes are valid.

Returns:

True if valid.

bool approximateBounds(glm::vec3 &center, float &radius) const

Conservative bounding sphere for all instances.

Parameters:

• center – Output center.

• radius – Output radius.

Returns:

True if there are instances to bound.

inline void setMaxRenderedInstances(size_t maxInstances)

Cap drawn instances for simple single-threaded LOD. 0 means draw all.

void setRenderedInstanceStride(size_t stride)

Set a stride for simple per-camera-independent LOD. 1 draws every instance.

void setProjectedLodPolicy(bool enabled, float minProjectedRadiusPixels = 2.0f, size_t maxStride = 8)

Enable camera-projected LOD for instanced visuals.

When enabled, the renderer increases the effective draw stride for far-away instances whose approximate per-instance radius projects below minProjectedRadiusPixels. The explicit rendered stride remains a lower bound, so user-requested LOD is preserved.

void setDoubleBufferedInstanceUploads(bool enabled)

Enable double-buffering for full instance-buffer uploads.

Dirty-range updates remain on the single-buffer subdata path. Double-buffering helps when full rebuilds are frequent, for example after sorting or projected LOD stride changes.

inline void setSortTransparentInstances(bool enabled)

Enable optional back-to-front CPU sorting for transparent instances.

This is disabled by default to keep RL visual workloads on the fast path.

void draw(Shader &shader)

Draw all instances using instanced rendering.

Parameters:

shader – Shader to use for drawing.

void draw(Shader &shader, const glm::vec3 &viewPos, float verticalFovDeg, int viewportHeight)

Draw all instances using camera-aware projected LOD.

class OpenGLMesh

Simple OpenGL mesh with indexed geometry and optional textures.

Public Functions

OpenGLMesh(const std::vector<Vertex> &vertices, const std::vector<unsigned int> &indices, const std::vector<Texture> &textures = {}, const glm::vec4 &baseColor = glm::vec4(1.0f, 1.0f, 1.0f, 1.0f))

Construct a mesh and upload it to the GPU.

Parameters:

• vertices – Vertex array.

• indices – Index array.

• textures – Optional texture list.

• baseColor – Base color used when textures are absent.

void draw(const Shader &shader) const

Draw the mesh using the provided shader.

Parameters:

shader – Shader with expected texture uniforms.

void draw(const Shader &shader, RenderMode mode) const

Draw the mesh if it matches the requested material render mode.

Parameters:

• shader – Shader with expected material uniforms.

• mode – Material filter for multi-pass rendering.

void drawInstanced(const Shader &shader, int instanceCount) const

Draw the mesh with instancing.

Parameters:

• shader – Shader with expected texture uniforms.

• instanceCount – Number of instances to draw.

bool usesPbrShading() const

True when this mesh should be rendered by the PBR shader.

uint64_t materialSortKey(RenderMode mode = RenderMode::All) const

Compact key for ordering opaque draws by material/texture state.

void updateMesh(const std::vector<Vertex> &vertices, const std::vector<unsigned int> &indices)

Update mesh data on the GPU.

Parameters:

• vertices – New vertex data.

• indices – New index data.

void updateVertices(const std::vector<Vertex> &vertices)

Update only vertex data on the GPU, keeping the existing index buffer.

Parameters:

vertices – New vertex data.

void uploadVertices(const Vertex *vertices, size_t count)

Upload vertex data without copying it into the CPU-side mesh cache.

Parameters:

• vertices – Pointer to vertex data.

• count – Number of vertices.

Public Members

std::vector<Vertex> vertices

Vertex list for the mesh.

std::vector<unsigned int> indices

Index list for the mesh.

std::vector<Texture> textures

List of textures associated with the mesh.

glm::vec4 baseColor

Base color used when textures are not present.

raisin::Material material

Material data used by the optional PBR path.

std::string debugSourceMeshName

Source mesh name and imported primitive counts used by optional diagnostics.

unsigned int VAO

OpenGL vertex array object handle.

namespace assimp

Functions

static inline void trimWhitespace(std::string &s)

static inline std::string lowerCopy(std::string value)

static inline bool isUsdModelFile(const std::string &path)

static bool hasImageExtension(const std::filesystem::path &path)

static inline std::string sanitizeTextureReference(std::string value)

static bool filenameMatchesTexturePattern(const std::filesystem::path &candidate, const std::string &requestedFilename)

static bool resolveExistingTexturePath(const std::string &directory, const std::string &rawPath, std::string &fullPath)

static bool materialTexturePath(aiMaterial *material, aiTextureType type, const std::string &directory, std::string &fullPath, unsigned int textureIndex = 0)

static unsigned int uploadDecodedTexture(const unsigned char *data, int width, int height, int components, bool useSrgbFormat)

static std::unordered_map<std::string, unsigned int> &embeddedTextureCache()

static unsigned int loadEmbeddedTexture(const aiScene *scene, const std::string &rawPath, bool useSrgbFormat)

static const aiTexture *embeddedTextureFromReference(const aiScene *scene, const std::string &rawPath)

static std::string embeddedTextureNameFromReference(const aiScene *scene, const std::string &rawPath)

static unsigned int loadEmbeddedTextureByIndex(const aiScene *scene, unsigned int index, bool useSrgbFormat)

static std::string textureFamilyStem(std::string stem)

static bool filenameLooksLikeMap(const std::string &lowerName, const char *kind)

static bool filenameLooksLikeRoughnessOnlyMap(const std::string &lowerName)

static bool filenameLooksLikeMetallicOnlyMap(const std::string &lowerName)

static std::string materialTextureDebugName(const aiScene *scene, aiMaterial *material, aiTextureType type, const std::string &directory, unsigned int textureIndex = 0)

static unsigned int loadSiblingEmbeddedTexture(const aiScene *scene, const std::string &referenceName, const char *kind, bool useSrgbFormat, std::string *matchedName = nullptr)

static unsigned int loadSiblingDirectoryTexture(const std::string &directory, const std::string &referenceName, const char *kind, bool useSrgbFormat, std::string *matchedName = nullptr)

static unsigned int loadMaterialTextureAtIndex(const aiScene *scene, aiMaterial *material, aiTextureType type, const std::string &directory, bool useSrgbFormat, unsigned int textureIndex, std::string *resolvedPath = nullptr)

static unsigned int loadMaterialTexture(const aiScene *scene, aiMaterial *material, aiTextureType type, const std::string &directory, bool useSrgbFormat)

static raisin::Material::NormalMapConvention detectNormalMapConventionFromPath(const std::string &path)

static bool preferredLdrTexturePath(const std::filesystem::path &candidate, const std::string &current)

static DirectoryPbrTextures findDirectoryPbrTextures(const std::string &directory)

static bool isUnambiguousDirectoryPbrSet(const DirectoryPbrTextures &textures)

static std::string materialTextureKey(std::string value)

static bool filenameStartsWithMaterialKey(const std::filesystem::path &path, const std::string &materialKey)

static DirectoryPbrTextures findDirectoryPbrTexturesForMaterial(const std::string &directory, const std::string &materialName)

static bool materialTextureLoggingEnabled()

static bool sceneLightLoggingEnabled()

static bool meshBoundsLoggingEnabled()

static bool lightNamePassesDebugFilter(const std::string &nodeName, const std::string &lightName)

static const char *loadedLabel(unsigned int textureId)

static std::string textureSizeLabel(unsigned int textureId)

static void logMaterialTextureStatus(const aiMaterial *sourceMaterial, const aiMesh *sourceMesh, const raisin::Material &materialInfo, const std::vector<Texture> &legacyTextures)

static void logMeshBoundsStatus(const aiMesh *sourceMesh, const raisin::Material &materialInfo, const std::vector<Vertex> &vertices, size_t indexCount)

static int glWrapMode(aiTextureMapMode mode)

static int glFilterModeFromGltf(int filter, bool minFilter)

static void applyTextureSamplerState(aiMaterial *material, aiTextureType type, unsigned int textureId)

static bool applyUvTransform(aiMaterial *material, aiTextureType type, raisin::Material &materialInfo, std::ostringstream &report)

static std::string parseMtlTexturePath(const std::string &line)

static bool loadMtlFallback(const std::filesystem::path &mtlPath, FallbackMaterial &out)

static void processNode(aiNode *node, const aiScene *scene, std::vector<std::shared_ptr<OpenGLMesh>> &meshes, const std::string &directory, aiMatrix4x4 parentTransform, const FallbackMaterial *fallback)

Recursively process an Assimp node hierarchy.

Recursively processes nodes in the Assimp scene graph.

Parameters:

• node – Current node.

• scene – Source Assimp scene.

• meshes – Output mesh list.

• directory – Base directory for texture loading.

• parentTransform – Accumulated transform from parent nodes.

static std::shared_ptr<OpenGLMesh> processMesh(aiMesh *mesh, const aiScene *scene, const std::string &directory, aiMatrix4x4 transform, const FallbackMaterial *fallback)

Convert a single Assimp mesh into an OpenGLMesh.

Processes an aiMesh and converts it into our OpenGLMesh format.

Parameters:

• mesh – Assimp mesh to convert.

• scene – Source Assimp scene.

• directory – Base directory for texture loading.

• transform – Accumulated transform for this mesh.

Returns:

Shared OpenGLMesh instance.

Returns:

An OpenGLMesh object.

static std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> loadUsdModel(const std::string &path)

static AsyncModelLoadResult importModelScene(const std::string &path)

static std::future<AsyncModelLoadResult> loadModelSceneAsync(const std::string &path)

static std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> finishImportedModelScene(AsyncModelLoadResult &imported)

static std::shared_ptr<std::vector<std::shared_ptr<OpenGLMesh>>> loadModel(const std::string &path)

Loads a model file and all its submeshes.

Parameters:

path – The file path to the 3D model.

Throws:

std::runtime_error – if the file cannot be loaded.

Returns:

A shared_ptr to a vector containing all loaded OpenGLMesh objects.

static glm::vec3 toGlmVec3(const aiVector3D &v)

static glm::vec3 transformPoint(const aiMatrix4x4 &transform, const aiVector3D &point)

static glm::vec3 transformDirection(const aiMatrix4x4 &transform, const aiVector3D &direction)

static void collectNodeTransforms(aiNode *node, aiMatrix4x4 parentTransform, std::unordered_map<std::string, aiMatrix4x4> &out)

static bool readGltfJsonText(const std::string &path, std::string &jsonText)

static glm::vec3 jsonVec3(const rapidjson::Value &value, const glm::vec3 &fallback)

static glm::quat jsonQuatXyzw(const rapidjson::Value &value)

static glm::vec2 jsonVec2(const rapidjson::Value &value, const glm::vec2 &fallback)

static glm::mat4 jsonNodeLocalTransform(const rapidjson::Value &node)

static void collectGltfLightNodes(const rapidjson::Value &nodes, unsigned int nodeIndex, const glm::mat4 &parentTransform, const std::vector<GltfPunctualLight> &lightDefs, float intensityScale, std::vector<raisin::RayraiWindow::AdditionalLight> &out)

static std::vector<raisin::RayraiWindow::AdditionalLight> loadGltfPunctualLightNodes(const std::string &path, float intensityScale)

static std::vector<std::filesystem::path> rayraiLightSidecarCandidates(const std::string &path)

static std::vector<raisin::RayraiWindow::AdditionalLight> loadRayraiLightSidecar(const std::string &path, float intensityScale)

static std::vector<raisin::RayraiWindow::AdditionalLight> loadSceneLights(const std::string &path, float intensityScale = 1.0f)

Import scene lights through Assimp and convert them into bounded RayRai viewer lights.

This reads authored punctual/area lights from formats that preserve them (for example glTF with KHR_lights_punctual). It does not add the lights automatically; callers explicitly decide when to pay the shading cost.

struct FallbackMaterial

#include <AssimpMeshLoader.hpp>

struct DirectoryPbrTextures

#include <AssimpMeshLoader.hpp>

struct AsyncModelLoadResult

#include <AssimpMeshLoader.hpp>

struct GltfPunctualLight

#include <AssimpMeshLoader.hpp>

Scene helpers

class PointCloud

Simple point cloud renderer.

Stores positions and colors and uploads them to the GPU when updated.

Public Functions

PointCloud()

Construct an empty point cloud.

~PointCloud()

Destroy the point cloud and release GPU resources.

inline void setCategory(VisualCategory category)

Set the visual category for detection or filtering.

Parameters:

category – Category value.

inline void setDetectable(bool detectable)

Convenience helper to set detectability.

Parameters:

detectable – True to mark as detectable.

inline VisualCategory getCategory() const

Get current visual category.

Returns:

Category value.

inline bool isDetectable() const

Check if point cloud is detectable.

Returns:

True if detectable.

void updatePointBuffer()

Upload point data to the GPU (or clear if empty).

void updatePointRange(size_t begin, size_t count)

Upload a changed point range without reallocating the whole buffer when possible.

void draw(Camera &camera, Shader &shader)

Draw the point cloud.

Parameters:

• camera – Active camera (unused but kept for consistency).

• shader – Shader to use for drawing.

bool approximateBounds(glm::vec3 &center, float &radius) const

Conservative bounding sphere for all points.

inline void setMaxRenderedPoints(size_t maxPoints)

Cap drawn points for simple single-threaded LOD. 0 means draw all.

void setRenderedPointStride(size_t stride)

Draw every Nth point for cheap point-cloud LOD. 1 draws contiguous points.

inline void enable(bool enable)

Enable or disable rendering.

Parameters:

enable – True to enable.

Public Members

std::vector<glm::vec3> positions

Point positions in world space.

std::vector<glm::vec4> colors

Per-point colors (must match positions size).

float pointSize

Rendered point size in pixels.

class CoordinateFrame

Render helper for visualizing multiple coordinate frames.

Public Functions

CoordinateFrame(std::shared_ptr<RayraiGlobalAsset> &asset)

Construct a coordinate frame visualizer.

Parameters:

asset – Shared asset cache for retrieving the axis mesh.

void draw(Shader &shader)

Draws all coordinate frames in the ‘poses’ vector.

Parameters:

shader – The shader program to use for rendering.

inline void enable(bool enable)

Enables or disables rendering of this coordinate frame set.

Parameters:

enable – True to enable, false to disable.

Public Members

std::vector<Pose> poses

List of poses to render.

double frameSize

Scale factor for axis length.

struct Pose

Pose definition for a single coordinate frame.

class CameraFrustum

Visual helper that renders a camera frustum volume.

The frustum mesh is updated from either a rayrai Camera or a raisim DepthCamera and rendered using a Visuals instance.

Public Functions

CameraFrustum(std::shared_ptr<Visuals> visual, std::shared_ptr<OpenGLMesh> mesh)

Construct a frustum helper with a visual and mesh.

Parameters:

• visual – Visuals object used for rendering.

• mesh – OpenGL mesh representing the frustum volume.

void updateFromCamera(const Camera &camera)

Update frustum mesh from a rayrai camera.

Parameters:

camera – Source camera.

void updateFromDepthCamera(raisim::DepthCamera &depthCam)

Update frustum mesh from a raisim depth camera.

Parameters:

depthCam – Source depth camera.

void setColor(const glm::vec4 &color)

Set the frustum color.

Parameters:

color – RGBA color.

void setDetectable(bool detectable)

Set detectability flag on the frustum visual.

Parameters:

detectable – True to mark as detectable.

inline std::shared_ptr<Visuals> getVisual() const

Get the underlying Visuals object.

Returns:

Shared pointer to the visual.

enum class raisin::VisualCategory

Category tag for visuals and point clouds.

Values:

enumerator NotDetectable

enumerator Detectable

Rendering and materials

class Light

Simple light with optional shadow map support.

Supports directional and point lighting models and owns the shadow-map framebuffer used by the renderer.

Public Functions

inline Light()

Default constructor creates a directional light and shadow resources.

inline void setAsDirectional(const glm::vec3 &dir, const glm::vec3 &diffColor = {0.85f, 0.85f, 0.85f})

Configure as a directional light.

Parameters:

• dir – Direction pointing from the light toward the scene.

• diffColor – Diffuse color.

inline void setAsPoint(const glm::vec3 &pos, const glm::vec3 &diffColor = {0.85f, 0.85f, 0.85f})

Configure as a point light.

Parameters:

• pos – Light position.

• diffColor – Diffuse color.

bool ensureShadowFramebuffer()

Create or resize the shadow framebuffer and depth texture.

bool shadowPrePass()

Prepare for shadow depth rendering (bind FBO, set viewport).

bool pointShadowPrePassFace(int face)

Prepare one face of a point-light cubemap shadow render.

Parameters:

face – Cubemap face index in OpenGL order [0, 5].

void shadowPostPass()

Restore state after shadow rendering.

void setShadowResolution(int res)

Set shadow-map resolution.

Parameters:

res – Shadow map size in pixels.

inline void setShadowsEnabled(bool e)

Enable or disable shadow sampling.

Parameters:

e – True to enable shadows.

inline void setShadowParams(float bias, float strength, float pcfRadius)

Set shadow bias and filtering parameters.

Parameters:

• bias – Depth bias to reduce acne.

• strength – Shadow darkening strength.

• pcfRadius – PCF sampling radius.

inline void setShadowOrtho(float halfSize, float zNear, float zFar)

Configure orthographic shadow projection for directional lights.

Parameters:

• halfSize – Half-size of the ortho box.

• zNear – Near plane.

• zFar – Far plane.

inline void setShadowCenter(const glm::vec3 &center)

Set the center point used for directional shadow projection.

Parameters:

center – World-space center.

inline void setShadowPosition(const glm::vec3 &pos, bool enable = true)

Override the light position used for directional shadow projection.

Parameters:

• pos – World-space light position.

• enable – True to use the custom position, false to revert to auto position.

Public Members

LightType type

Active light type.

glm::vec3 ambient

Ambient, diffuse, and specular color terms.

glm::vec3 position

Light position (point lights only).

glm::vec3 direction

Light direction (directional lights only).

float constant

Attenuation parameters for point lights.

unsigned int shadowFbo_

Shadow framebuffer and depth texture identifiers.

class Material

A general-purpose material class supporting PBR, textured, and simple color models.

This class uses a unified PBR model. Simpler material types are emulated by setting PBR properties to default values (e.g., metallic=0, roughness=0.8 for a dielectric).

Public Types

enum class Type

Defines the user-facing type of material for easier configuration. The shader itself will always use the PBR rendering path.

Values:

enumerator SIMPLE_COLOR

Basic diffuse material with a solid color.

enumerator TEXTURED

A simple textured material (uses albedoMap).

enumerator PBR

Full PBR material with metallic/roughness properties.

enum class NormalMapConvention

Tangent-space normal-map convention.

Values:

enumerator OpenGL

+Y/green-up normal maps.

enumerator DirectX

-Y/green-down normal maps, common in Unreal assets.

enum class AlphaMode

glTF-style alpha handling for imported materials.

Values:

enumerator Opaque

enumerator Mask

enumerator Blend

Public Functions

Material(std::string name = "DefaultMaterial")

Construct a new Material object.

Parameters:

name – A unique name for the material.

void applyTo(Shader &shader) const

Applies the material’s properties to a shader.

This method binds the necessary textures and sets all material-related uniforms on the provided shader before a draw call.

Parameters:

shader – The shader to which the properties will be applied.

bool usesPbrShading() const

True when this material needs the PBR shader path.

Public Members

std::string name

Material name used for debugging or lookup.

Type type

Material type hint (PBR path is always used in shaders).

glm::vec4 baseColorFactor

Base color factor (albedo tint).

This is the base color if no texture is used, or a tint factor otherwise.

float metallicFactor

Metallic factor (0 for dielectric, 1 for metal).

float roughnessFactor

Roughness factor (0 smooth, 1 rough).

float ao

Ambient occlusion strength.

glm::vec3 emissiveFactor

Emissive color factor in linear RGB.

float iblStrength

Strength of the simple procedural image-based lighting approximation.

float environmentMapStrength

Blend amount for the optional environment cubemap.

float normalStrength

Strength of tangent-space normal-map perturbation.

float normalMapYScale

Tangent-space normal-map Y scale. Use -1 for DirectX-style normal maps.

NormalMapConvention normalMapConvention

Named normal-map convention. This is applied before normalMapYScale.

AlphaMode alphaMode

Material alpha mode. Masked and fully transparent texels discard before depth writes.

float alphaCutoff

Alpha cutoff used for masked materials.

float aoStrength

Ambient-occlusion texture strength. 0 ignores the AO map, 1 uses it fully.

float emissiveStrength

Multiplier for emissive texture/factor contribution.

glm::vec4 uvScaleOffset

UV transform as scale.xy and offset.zw.

float uvRotation

UV rotation in radians.

float clearcoatFactor

Optional clearcoat layer strength.

float clearcoatRoughnessFactor

Optional clearcoat roughness.

bool usePlanarReflection

Whether this material samples the renderer’s planar reflection texture.

float planarReflectionStrength

Blend strength for planar reflections on this material.

bool doubleSided

Disable back-face culling for this material.

bool decalDepthBias

Render after ordinary opaque surfaces with decal-style depth handling.

bool tinySurfaceDetail

Very small albedo-only imported surface detail such as book covers.

unsigned int albedoMap

Base color texture (OpenGL id, 0 if unused).

unsigned int normalMap

Tangent-space normal map (OpenGL id, 0 if unused).

unsigned int metallicMap

Metallic map (OpenGL id, 0 if unused).

unsigned int roughnessMap

Roughness map (OpenGL id, 0 if unused).

unsigned int metallicRoughnessMap

Combined glTF metallic-roughness map (G=roughness, B=metallic, 0 if unused).

unsigned int aoMap

Ambient occlusion map (OpenGL id, 0 if unused).

unsigned int emissiveMap

Emissive color texture (OpenGL id, 0 if unused).

unsigned int clearcoatMap

Optional clearcoat intensity texture, red channel.

unsigned int clearcoatRoughnessMap

Optional clearcoat roughness texture, red channel.

unsigned int environmentMap

Optional radiance/environment cubemap (OpenGL cubemap id, 0 if unused).

unsigned int irradianceMap

Optional diffuse irradiance cubemap for high-fidelity IBL.

unsigned int prefilteredEnvironmentMap

Optional prefiltered specular cubemap for high-fidelity IBL.

unsigned int brdfLut

Optional split-sum BRDF LUT texture for high-fidelity IBL.

std::string importReport

Human-readable import notes for debugging material differences.

RaiSim integration

class RaisimObject

Render wrapper for a raisim::Object.

RaisimObject caches meshes and per-shape scales for a raisim object and exposes a draw method that respects appearance and collision-visual toggles.

Public Functions

explicit RaisimObject(raisim::Object *object, RayraiWindow *parent, std::shared_ptr<RayraiGlobalAsset> asset)

Construct a render wrapper for a raisim object.

Parameters:

• object – Raisim object to wrap.

• parent – Owning RayraiWindow (for shared assets/settings).

• asset – Asset cache to load primitive/mesh geometry.

void draw(Shader &shader, bool showCollisionBodies, OpenGLMesh::RenderMode mode = OpenGLMesh::RenderMode::All, uint64_t renderSerial = 0)

Draw the object using the provided shader.

Parameters:

• shader – Shader used for rendering.

• showCollisionBodies – True to render collision geometry when available.

bool hasPbrMeshes() const

True when any cached mesh needs the PBR shader.

bool isSelectable() const

Check if this object can be selected in the picker.

Returns:

True if selectable.

void updateAppearance()

Refresh appearance (color) from the underlying raisim object.

void configureGroundMaterial(bool reflective, float roughness, float metallic)

Configure ground/halfspace material. No-op for non-ground objects.

inline void setHidden(const bool h)

Hide or show this object in rendering.

Parameters:

h – True to hide.

inline bool hidden() const

Check if object is hidden.

Returns:

True if hidden.

inline glm::vec4 getColor() const

Get object color used for rendering.

Returns:

RGBA color.

float approximateRadius(bool showCollisionBodies) const

Conservative radius around the raisim object position for coarse culling.

bool getPrimitiveBatchData(bool showCollisionBodies, glm::mat4 &model, raisim::Shape::Type &shape, glm::vec4 &color) const

Return transform/color data for simple single-body primitive batching.

Returns:

True when this object can be drawn by the internal primitive batch path.

struct BufferReader

Lightweight reader for binary message buffers.

The reader maintains a cursor into a byte buffer and provides helpers for reading typed values, strings, and vectors.

Public Functions

inline explicit BufferReader(const std::vector<char> &data)

Construct a reader over an existing byte buffer.

Parameters:

data – Message payload.

inline size_t offset() const

Current offset from the start of the buffer.

Returns:

Offset in bytes.

inline size_t size() const

Total size of the buffer.

Returns:

Size in bytes.

inline size_t remaining() const

Remaining unread bytes.

Returns:

Remaining byte count.

template<typename T>
inline T read()

Read a trivially copyable value from the buffer.

Template Parameters:

T – Value type.

Returns:

Parsed value or default-constructed value on failure.

template<typename T>
inline bool peek(size_t offset, T &out) const

Peek a value at a byte offset without advancing the cursor.

Template Parameters:

T – Value type.

Parameters:

• offset – Byte offset from current cursor.

• out – Output value.

Returns:

True if the value could be read.

inline bool readBool()

Read a boolean value.

Returns:

Boolean value (false on failure).

inline std::string readString()

Read a length-prefixed string.

Returns:

Parsed string (empty on failure).

inline glm::vec3 readVec3f()

Read a vec3 of floats.

Returns:

Vector value.

inline glm::vec4 readVec4f()

Read a vec4 of floats.

Returns:

Vector value.

inline glm::vec4 readQuatWxyz()

Read a quaternion encoded as wxyz floats.

Returns:

Quaternion vector (wxyz).

inline std::vector<float> readFloatVector()

Read a vector of floats with a 32-bit length prefix.

Returns:

Parsed float vector.

inline std::vector<int32_t> readIntVector()

Read a vector of int32 with a 32-bit length prefix.

Returns:

Parsed int vector.

inline std::vector<uint8_t> readByteVector()

Read a vector of bytes with a 32-bit length prefix.

Returns:

Parsed byte vector.

inline void skipBytes(size_t count)

Skip a number of bytes.

Parameters:

count – Number of bytes to skip.

inline std::vector<raisim::ColorRGB> readColorMap()

Read a color map (length-prefixed array of raisim::ColorRGB).

Returns:

Parsed color map.

inline void skipColorMap()

Skip a color map payload (length-prefixed).