How to replace SMPL-X's local wrist rotatoin with MANO hand's global orientation prediction

[hongsukchoi]

Hi!

Is there any way to replace SMPL-X's local wrist rotatoin with MANO hand's global orientation prediction?

I have SMPL-X prediction converted from SMPL prediction. Since the wrist rotations are not accurate inn SMPL prediction, especially yaw and pitch, I want to replace them with MANO predictions' global orientation from WiLoR: https://github.com/rolpotamias/WiLoR

What I am trying is like this:

I think this makes sense and my implementation is like this, but I don't know the "offset rotation" between MANO and SMPLX coordinate frames. Could you guide me through this?

converted_betas = conv_body_shape(smpl_params['betas'], smpl_to_smplx_shape_converter)
  converted_body_pose = conv_smpl_to_smplx_body_pose(smpl_params['body_pose']) # (21, 3, 3)
  converted_global_orient = smpl_params['global_orient']

  # 21 joints' parent joint index
  smpl_parent_joint_idx = smplx_layer.parents[:22] # [-1,  0,  0,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  9,  9, 12, 13, 14, 16, 17, 18, 19]
  # body_pose: (21, 3, 3) are local rotation matrices respect to the parent joints
  # Get global joint rotations for wrists using forward kinematics
  global_rotations = torch.eye(3).expand(22, 3, 3)  # Initialize identity matrices for all joints
  global_rotations[0] = torch.from_numpy(converted_global_orient[0])  # Set root orientation

  # Forward kinematics to compute global rotations
  for joint_idx in range(1, 22):  # Skip root (idx 0)
      parent_idx = smpl_parent_joint_idx[joint_idx]
      if parent_idx >= 0:  # Skip root which has -1 as parent
          local_rot = torch.from_numpy(converted_body_pose[joint_idx-1])  # -1 since body_pose doesn't include root
          global_rotations[joint_idx] = torch.matmul(global_rotations[parent_idx], local_rot)
  
  # Get left and right wrist global rotations (indices 20 and 21)
  left_wrist_global_rot = global_rotations[20].numpy()  # Left wrist
  right_wrist_global_rot = global_rotations[21].numpy()  # Right wrist

  left_wrist_global_rot_mano = left_hand_mano_data['global_orient'] # (1, 3)
  right_wrist_global_rot_mano = right_hand_mano_data['global_orient'] # (1, 3)
  # convert to rotation matrix
  left_wrist_global_rot_mano = R.from_rotvec(left_wrist_global_rot_mano).as_matrix()[0]
  right_wrist_global_rot_mano = R.from_rotvec(right_wrist_global_rot_mano).as_matrix()[0]

  # !!!! TEMP: random search the offset rotation matrices !!!! #
  for rot_idx, rot_matrix in enumerate(rot_matrices):
      right_mano_smpl_offset_rot = rot_matrix
      left_mano_smpl_offset_rot = rot_matrix

      # TEMP; assign 180 degree rotation around z axis to right hand
      # right_mano_smpl_offset_rot = rot_x_90 @ rot_y_neg90 @ rot_z_neg90

      # Get the new local rotation matrix for wrists
      left_wrist_local_rot_mano = left_wrist_global_rot.T @ left_wrist_global_rot_mano @ left_mano_smpl_offset_rot
      right_wrist_local_rot_mano = right_wrist_global_rot.T @ right_wrist_global_rot_mano @ right_mano_smpl_offset_rot

      # pelvis offset; -1
      converted_body_pose[20 - 1] = left_wrist_local_rot_mano
      converted_body_pose[21 - 1] = right_wrist_local_rot_mano