72956 segmentation fault

[zichunxx]

Hi! Thanks for your great sharing!

I met the 72956 segmentation fault when I tried to train the task with Pixels suffix like FrankaPickPixels.

Besides, I have finished the training successfully with the task without the Pixels suffix. It seems that the segmentation fault is not triggered by pytorch.

I'm using Isaac Gym Preview 4 on Ubuntu 20.04.

Here is the output after running python tools/train_ppo.py task=FrankaPickPixels

Importing module 'gym_37' (/home/xzc/Downloads/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/xzc/Downloads/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.10.0
Device count 1
/home/xzc/Downloads/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/xzc/.cache/torch_extensions/py37_cu113 as PyTorch extensions root...
Emitting ninja build file /home/xzc/.cache/torch_extensions/py37_cu113/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
/home/xzc/Downloads/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/torch_utils.py:135: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  def get_axis_params(value, axis_idx, x_value=0., dtype=np.float, n_dims=3):
tools/train_ppo.py:13: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_name="config", config_path="../configs/ppo")
/home/xzc/mambaforge/envs/mvp/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/home/xzc/mambaforge/envs/mvp/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:415: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/defaults_list_override for more information.

  deprecation_warning(msg)
/home/xzc/mambaforge/envs/mvp/lib/python3.7/site-packages/hydra/_internal/hydra.py:127: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  configure_logging=with_log_configuration,
task: 
    name: FrankaPick
    env: 
        numEnvs: 256
        envSpacing: 1.5
        episodeLength: 500
        object_pos_init: [0.5, 0.0]
        object_pos_delta: [0.1, 0.2]
        goal_height: 0.8
        obs_type: pixels
        im_size: 224
        cam: 
            w: 298
            h: 224
            fov: 120
            ss: 2
            loc_p: [0.04, 0.0, 0.045]
            loc_r: [180, -90.0, 0.0]
        dofVelocityScale: 0.1
        actionScale: 7.5
        objectDistRewardScale: 0.08
        liftBonusRewardScale: 4.0
        goalDistRewardScale: 1.28
        goalBonusRewardScale: 4.0
        actionPenaltyScale: 0.01
        asset: 
            assetRoot: assets
            assetFileNameFranka: urdf/franka_description/robots/franka_panda.urdf
    sim: 
        substeps: 1
        physx: 
            num_threads: 4
            solver_type: 1
            num_position_iterations: 12
            num_velocity_iterations: 1
            contact_offset: 0.005
            rest_offset: 0.0
            bounce_threshold_velocity: 0.2
            max_depenetration_velocity: 1000.0
            default_buffer_size_multiplier: 5.0
            always_use_articulations: False
    task: 
        randomize: False
train: 
    seed: 0
    torch_deterministic: False
    encoder: 
        name: vits-mae-hoi
        pretrain_dir: /home/xzc/Documents/mvp/tmp/pretrained
        freeze: True
        emb_dim: 128
    policy: 
        pi_hid_sizes: [256, 128, 64]
        vf_hid_sizes: [256, 128, 64]
    learn: 
        agent_name: franka_ppo
        test: False
        resume: 0
        save_interval: 50
        print_log: True
        max_iterations: 2000
        cliprange: 0.1
        ent_coef: 0
        nsteps: 32
        noptepochs: 10
        nminibatches: 4
        max_grad_norm: 1
        optim_stepsize: 0.001
        schedule: cos
        gamma: 0.99
        lam: 0.95
        init_noise_std: 1.0
        log_interval: 1
physics_engine: physx
pipeline: gpu
sim_device: cuda:0
rl_device: cuda:0
graphics_device_id: 0
num_gpus: 1
test: False
resume: 0
logdir: /home/xzc/Documents/mvp/tmp/debug
cptdir: 
headless: True
Wrote config to: /home/xzc/Documents/mvp/tmp/debug/config.yaml
Setting seed: 0
Setting sim options
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
num franka bodies:  11
num franka dofs:  9
[1]    72956 segmentation fault  python tools/train_ppo.py task=FrankaPickPixels

Looking forward to any comments! Thanks!

[Virlus]

Exactly the same issue. I have tried solutions from nvidia forum but they didn't work. Besides, I have tried to change the config as follows:

physics_engine: "physx"
pipeline: "gpu"
sim_device: "cuda:1"
rl_device: "cuda:1"
graphics_device_id: 1
num_gpus: 1

And another error occurs:

[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101
*** Can't create empty tensor

Hence I referred to this post for solution. But it fails as well. Hopefully the authors can come up with a workaround. Thanks!

[orrkrup]

Any luck solving this issue?

I had working code and then updated my Nvidia driver from 515 to 550, just to get this error when trying to instantiate camera sensors.

[qpwo]

There's a blog post in chinese

https://github.com/misaka0502/misaka0502.github.io/blob/6baa4b9c7ca34a007cbdc3cac0d4dd467c980839/source/_posts/fix-isaac-gym-bug.md?plain=1#L2

Here is an automatic translation:

title: Documenting a Bug Encountered While Running Isaac Gym and the Debugging Process: cudaExternamMemoryGetMappedBuffer failed on…

The Bug Appears

Today, when running the project, I encountered an error:

Traceback (most recent call last):
	...
	color_obs = torch.stack(color_obs)[..., :-1] #RGBA->RGB
TypeError: expected Tensor as elment 0 in argument 0, but got NoneType

The general situation is that during environment reset, the camera images cannot be read; they are all None. However, this error never occurred before, and no modifications were made to the environment code since the last run.

Then, I tried outputting relevant information during environment initialization and found some strange ERROR messages:

[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101
*** Can't create empty tensor

The general meaning is that cudaExternalMemoryGetMappedBuffer failed when trying to map CUDA memory for RGB images, depth maps, segmentation maps, and optical flow maps, with error code 101, and filling the rigid body tensor also failed.

At this time, I was using GPU 9 (RTX3090). When I switched to GPU 3, which had no processes running, it worked normally again. The reason was temporarily unknown. After several attempts, I found that this error would occur on certain GPUs, while the code could run normally on others.

After some time, I found that all GPUs on the server reported this error, and my code could no longer run at all 🙃.

Currently, I still haven't found the specific cause or a fundamental solution. After searching online for a long time, I found that many people have encountered this bug, but there's no common explanation or fix. I can only speculate that it might be related to some underlying issues with Isaac Gym and CUDA. However, I did find a way to work around this problem.

The Debugging Process

Finally, the problem was narrowed down to two points.

First, it was discovered during the creation of the camera sensor, where the camera properties were set as follows:

camera_cfg = gymapi.CameraProperties()
camera_cfg.enable_tensors = True
camera_cfg.width = self.img_size[0]
camera_cfg.height = self.img_size[1]
camera_cfg.near_plane = 0.001
camera_cfg.far_plane = 2.0
camera_cfg.horizontal_fov = 40.0 if self.resize_img else 69.4
self.camera_cfg = camera_cfg

The problem appeared in the line camera_cfg.enable_tensors = True. This line instructs Isaac Gym to directly output the image data from this camera as GPU tensors, instead of the default CPU NumPy array format. As long as this value was set to True, the error would definitely occur, but setting it to False prevented the error.

Furthermore, when enable_tensors=True, running the following statement:

self.isaac_gym.prepare_sim(self.sim)

would also report Failed to fill rigid body state tensor (I'm not sure why filling the rigid body tensor is related to the camera settings). However, after setting enable_tensors=False, this error no longer occurred.

Second, the original method for acquiring images used the following approach:

tensor = gymtorch.wrap_tensor(
    self.isaac_gym.get_camera_image_gpu_tensor(
        self.sim, env, handle, render_type
    )
)

The original code obtained tensors directly from the camera sensor in this way, which was intended to improve efficiency. However, due to the aforementioned problem during camera creation, the images could naturally not be obtained here.

Given this, I decided not to use the original camera sensor setup and instead adopted a different approach, rewriting the way the environment acquires camera images.

First, change camera_cfg.enable_tensors = False during camera creation. Then, when getting the observation in the step function, add the following lines first:

self.isaac_gym.fetch_results(self.sim, True)
self.isaac_gym.step_graphics(self.sim)
self.isaac_gym.render_all_camera_sensors(self.sim)

These three lines are important to ensure the camera images are acquired. Then, use the gym.get_camera_image() method to get the image. Here's an example:

class env(gym.Env):
		# Create environment
		self.isaac_gym = gymapi.acquire_gym()
		sim = self.isaac_gym.create_sim(
			compute_device_id,
		    graphics_device_id,
		    gymapi.SimType.SIM_PHYSX,
		    sim_config["sim_params"],
		)
		self.env = self.isaac_gym.create_env(self.sim, ...)
		# Set up camera
		camera_cfg = gymapi.CameraProperties()
		camera_cfg.enable_tensors = False # Set to False
		self.camera = self.isaac_gym.create_camera_sensor(env, camera_cfg)
		... # Other settings

		def step(self, action):
			...
			obs = self.get_observation() # Renamed for clarity

		def get_observation(self):
			# Important: Fetch results, step graphics, render sensors
			self.isaac_gym.fetch_results(self.sim, True)
			self.isaac_gym.step_graphics(self.sim)
			self.isaac_gym.render_all_camera_sensors(self.sim)

			# Get color image using get_camera_image
			image_handle = self.camera # Assuming self.camera holds the handle
			image = self.isaac_gym.get_camera_image(self.sim, self.env, image_handle, gymapi.IMAGE_COLOR)

			# Convert from NumPy array (CPU) to PyTorch tensor (GPU)
			# Note: image returned is a flat C-style array (width * height * 4)
			image = np.frombuffer(image, dtype=np.uint8) # Interpret buffer as uint8
			image = image.reshape(self.img_size[1], self.img_size[0], 4) # Reshape to H x W x 4 (RGBA)
			image = image[..., :3] # RGBA -> RGB

			# Convert to torch tensor and move to device
			image_tensor = torch.from_numpy(image).to(self.device)

			# Get other observations
			...
			# Return combined observations including the image_tensor
			return combined_obs

After making these changes, there were temporarily no more problems, and the code could finally run again. 🤧