Question about Teacher-Student Policy Training and Obstacle Crossing

[PHZ-897044002]

Dear Author,

I hope this email finds you well.
I am currently reproducing your code and have encountered some issues that I would like to ask for your advice. First of all, thank you very much for sharing this project. It has been very helpful for my learning.
I have successfully trained the BBC controller and tested it, and the performance looks very good. After that, I followed the README to train the teacher and student policies. During teacher policy training, the success rate fluctuates around 34–40. However, when training the student policy, I found that the overall success rate remains at 0, although the reward curve seems to converge relatively normally.
During testing, I also noticed that both the teacher and student policies tend to get stuck in the circular obstacle scenario. Specifically, after the robot’s front legs step over the obstacle, the rear legs often get stuck and the robot cannot pass through successfully.
I would like to ask whether this situation is expected during reproduction, or whether some reward terms or training settings may need to be modified.
Thank you very much for your time and help.

The figure below shows the training curve of my BBC controller. I would like to ask whether this result indicates that the model has reached its final expected performance.

[tRNAoO]

Thank you for your interest in our work and for the detailed description of the issue.

In our experiments, when a well-trained BBC controller is used for teacher policy training, the teacher policy is expected to achieve a success rate higher than 90%. Therefore, the success rate of around 34–40% that you observed suggests that there may be an issue in the reproduced BBC controller, the training configuration, or the downstream teacher/student training setup.

As a first step, we recommend using the pretrained BBC controller provided in our repository and then training the teacher policy again with the same settings described in the README. This can help determine whether the problem comes from the BBC controller itself or from other parts of the training pipeline.

If the pretrained BBC controller leads to the expected success rate, then the issue is likely related to the reproduced BBC training process or its checkpoint quality. In that case, you may want to compare the BBC training configuration, motion data, checkpoint loading, and environment settings with the provided setup.

If the issue still occurs even when using the pretrained BBC controller, please feel free to contact us again with more details, such as the training configuration, command lines, logs, and checkpoint information. We would be happy to help further investigate the problem.

[PHZ-897044002]

Thank you for your reply. After receiving your response, we used the BBC pretrained model you provided, which was originally located in the tsc/weights/bbc directory, and then trained the teacher policy. However, the success rate did not reach the expected performance and fluctuated around 15%.

Throughout the entire training process, I strictly followed the instructions in the README. The following is my training result. Could you please help me check it?

he device I used is an RTX 3090 GPU. When running the code, we only used device 0 for training, and we did not make any modifications to the code.

If you need any additional runtime information to further check the issue, please let me know.

Thank you again for your reply and for helping me reproduce this project.

[tRNAoO]

We trained using exactly the same repository, and the success rate can quickly rise to above 80% on an RTX 4080. Based on the training curve you shared, mean_episode_length seems to be too high (it should not be that high if randomize_start is enabled). Please check whether you trained with the recommended script:
python train.py --proj_name agility --exptid base --device_id 0 --randomize_start --randomize_base_mass --randomize_base_com --push_robots

[JiayingST]

Hi, I have a similar with TSC-teacher training.

Setup: RTX 6000 Ada (48GB), CUDA 11.8, pretrained BBC from tsc/weights/bbc/model.pt, trained with the exact command from the README.

Symptom: Train/mean_success_rate is stuck at ~0.14–0.15 from step 0 through 50k iterations with no improvement. The curriculum (curr_threshold=0.8) never advances. Meanwhile mean_reward rises normally (~50→97), suggesting the policy learns locomotion but fails to complete the full obstacle course.

Observation: During play, the robot successfully crosses the first two obstacles, but consistently fails at the tunnel — it approaches the entrance but cannot navigate through.

[tRNAoO]

@JiayingST Hi, could you share your TSC-teacher training script?

[JiayingST]

My training script is:

python train.py --proj_name agility --exptid base --device_id 0
--randomize_start --randomize_base_mass --randomize_base_com --push_robots

I also restarted the training just now, and it looks same issue:

[PHZ-897044002]

Following your suggestions, I re-ran the experiments. However, the issue remains the same: the reward increases rapidly, but the success rate plateaus at around 0.15 and does not show any further significant improvement. I also tried re-downloading the project and training the teacher policy directly from scratch, but the results were still unchanged.

[JiayingST]

Hi, I also reran the training, but I'm still seeing the same issue.

The episode length is still consistently above 1000, and the success rate is only around 0.14.

This is the training script I'm using:

python train.py --proj_name agility --exptid base2 --device_id 0
--randomize_start --randomize_base_mass --randomize_base_com --push_robots
--max_iterations 30000

[tRNAoO]

On our three machines (4080, 4090, and 5060), we created fresh conda environments, re-cloned the repository, and retrained the TSC-Teacher. In all cases, it converged normally, so we were unable to reproduce the issue.

We suggest checking the following:

1. Whether the BBC checkpoint is being loaded correctly. You can also try training without --headless to inspect the BBC behavior, or keep one TSC-Teacher action fixed and see whether the BBC behavior looks normal.
2. Whether the command-line arguments are actually taking effect, e.g. randomize_start.
3. Compare your setup with our local agilit conda environment dependencies.
   agility.yml

[PHZ-897044002]

Thank you very much for your suggestions.

Following your advice, I re-cloned the repository and re-ran the TSC-Teacher training using the recommended command. However, the training result is still not satisfactory, and the success rate does not converge to the expected level.

I also carefully checked the issues you mentioned. The command-line arguments appear to be taking effect correctly: randomize_start, randomize_base_mass, randomize_base_com, and push_robots are all set to True, and env_cfg.obstacle.randomize_start is also True. In addition, I checked the BBC checkpoint loading process. The checkpoint can be loaded successfully, and the actor_critic, estimator, and discriminator are loaded without missing keys or size mismatch errors.

Although these checks seem normal, the training result is still far from the expected performance. Since I am very interested in your work and would sincerely like to use this project as the base framework for my own research, may I ask whether it would be possible to add your contact information, such as email or WeChat, for further communication? I would greatly appreciate the opportunity to discuss this issue with you in more detail.

Thank you again for your time and help.

[tRNAoO]

Hi @PHZ-897044002, thank you for the detailed follow-up and for trying the recommended steps. If you'd like to discuss the issue further, please feel free to reach out to me at hqfu@smail.nju.edu.cn.

[JiayingST]

May I ask what software environment you used on the RTX 5090? In particular:

PyTorch version
CUDA Toolkit version
NVIDIA driver version (if relevant)

If possible, could you also share the output of conda env export (or the environment YAML file)? That would be really helpful for reproducing the setup.

Thanks!

[tRNAoO]

@JiayingST Sorry, there was a mistake here. We actually tested on 4080, 4090, and 5060, not a 5090. Below is the exported conda YAML file from the 4090 environment (Driver Version: 580.159.03, CUDA Version: 13.0):
env.yml
and the 5060 environment (Driver Version: 580.159.03, CUDA Version: 13.0):
env.yml

[JiayingST]

Hi, thanks for the previous reply. We've done further diagnosis.

Setup:

• Server: RTX 6000 Ada (48GB, SM89), CUDA 11.8, PyTorch 2.3.1+cu118, numpy 1.23.5
• Using the pretrained BBC model with a fresh clone of the repository.
• Problem: mean_success_rate remains around 0.13–0.15 even after 50k iterations.

Diagnosis via play.py:

We watched your demo video and noticed that the robot uses canter → jump → canter to enter the tunnel.

However, in our trained policy (30k iterations), the terminal consistently shows:

cur_obst_type: tunnel
latent_c: 2 or 3

Specifically, latent_c = 3 only appeared once at the very beginning of training. After that, it always remained at latent_c = 2. We have never observed latent_c = 4 (jump) at the tunnel.

It seems that the policy never learns to switch to the jump gait when approaching the tunnel. Do you have any idea what could cause this behavior? For example, could it be related to the discrete gait exploration, latent initialization, reward design, curriculum, or some other training setting? Is there anything you would recommend checking or modifying to help the policy discover the jump behavior?

[tRNAoO]

@PHZ-897044002 @JiayingST Thanks for the discussion and the detailed reports.

After running additional experiments on several different hardware setups, we found that the original hyperparameters can be sensitive to stochastic factors such as seed, GPU/runtime differences, and training dynamics. In some configurations, the policy may get stuck in a suboptimal behavior and fail to converge reliably.

A more robust setting in our experiments is to increase tracking_goal_vel from 0.4 to 1.2, and reduce tracking_yaw from 2.0 to 1.0:

python train.py \
  --proj_name agility \
  --exptid base \
  --device_id 0 \
  --randomize_start \
  --randomize_base_mass \
  --randomize_base_com \
  --push_robots \
  --tracking_goal_vel 1.2 \
  --tracking_yaw 1.0

This setting was more stable across multiple seeds and machines in our tests. If convergence is still difficult, another more conservative option is to further increase reach_goal to 10:

python train.py \
  --proj_name agility \
  --exptid base \
  --device_id 0 \
  --randomize_start \
  --randomize_base_mass \
  --randomize_base_com \
  --push_robots \
  --tracking_goal_vel 1.2 \
  --tracking_yaw 1.0 \
  --reach_goal 10