model training fails

[sumwailiu]

Here is my log of running python -u /root/dreamer/dreamer.py --logdir /root/dreamer/logdir/dmc_walker_walk/dreamer/1 --task dmc_walker_walk:

Prefill dataset with 0 steps.
Simulating agent for 4995000 steps.
Found 5296820 model parameters.
Found 578412 actor parameters.
Found 413601 value parameters.
Start evaluation.
Training for 100 steps.
[5000] expl_amount 0.0 / model_grad_norm nan / value_grad_norm nan / actor_grad_norm nan / prior_ent nan / post_ent nan / image_loss inf / reward_loss nan / div nan / model_loss nan / value_loss nan / actor_loss nan / action_ent nan
Test episode of length 1000 with return 45.2.
Start collection.
Train episode of length 1000 with return 45.8.
Training for 100 steps.
[6000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm nan / actor_grad_norm nan / prior_ent nan / post_ent nan / image_loss inf / reward_loss nan / div nan / model_loss nan / value_loss nan / actor_loss nan / action_ent nan / fps 0.7
Train episode of length 1000 with return 4.9.
Training for 100 steps.
[7000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm nan / actor_grad_norm nan / prior_ent nan / post_ent nan / image_loss inf / reward_loss nan / div nan / model_loss nan / value_loss nan / actor_loss nan / action_ent nan / fps 21.3
Train episode of length 1000 with return 8.9.
Training for 100 steps.
[8000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm nan / actor_grad_norm nan / prior_ent nan / post_ent nan / image_loss inf / reward_loss nan / div nan / model_loss nan / value_loss nan / actor_loss nan / action_ent nan / fps 22.3
Train episode of length 1000 with return 46.8.
Training for 100 steps.
[9000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm 9.2 / actor_grad_norm 0.0 / prior_ent 35.6 / post_ent 52.0 / image_loss inf / reward_loss inf / div inf / model_loss inf / value_loss 1.2 / actor_loss -12.3 / action_ent inf / fps 28.4
Train episode of length 1000 with return 5.2.

...
[162000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm nan / actor_grad_norm nan / prior_ent nan / post_ent nan / image_loss inf / reward_loss nan / div nan / model_loss nan / value_loss nan / actor_loss nan / action_ent nan / fps 20.6
Train episode of length 1000 with return 6.3.
Training for 100 steps.
[163000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm nan / actor_grad_norm nan / prior_ent nan / post_ent nan / image_loss inf / reward_loss nan / div nan / model_loss nan / value_loss nan / actor_loss nan / action_ent nan / fps 20.9
Train episode of length 1000 with return 47.6.
Training for 100 steps.
[164000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm nan / actor_grad_norm nan / prior_ent nan / post_ent nan / image_loss inf / reward_loss nan / div nan / model_loss nan / value_loss nan / actor_loss nan / action_ent nan / fps 20.6
Train episode of length 1000 with return 4.9.
Start evaluation.
Training for 100 steps.
[165000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm nan / actor_grad_norm nan / prior_ent nan / post_ent nan / image_loss inf / reward_loss nan / div nan / model_loss nan / value_loss nan / actor_loss nan / action_ent nan / fps 20.8
WARNING:absl:Nan, Inf or huge value in CTRL at ACTUATOR 0. The simulation is unstable. Time = 0.0000.
Traceback (most recent call last):
  File "/root/dreamer2/dreamer.py", line 468, in <module>
    main(parser.parse_args())
  File "/root/dreamer2/dreamer.py", line 448, in main
    functools.partial(agent, training=False), test_envs, episodes=1)
  File "/root/dreamer2/tools.py", line 124, in simulate
    obs, _, done = zip(*[p()[:3] for p in promises])
  File "/root/dreamer2/tools.py", line 124, in <listcomp>
    obs, _, done = zip(*[p()[:3] for p in promises])
  File "/root/dreamer2/wrappers.py", line 350, in step
    obs, reward, done, info = self._env.step(action)
  File "/root/dreamer2/wrappers.py", line 162, in step
    obs, reward, done, info = self._env.step(action)
  File "/root/dreamer2/wrappers.py", line 211, in step
    obs, reward, done, info = self._env.step(action)
  File "/root/dreamer2/wrappers.py", line 267, in step
    return self._env.step(original)
  File "/root/dreamer2/wrappers.py", line 239, in step
    obs, reward, done, info = self._env.step(action)
  File "/root/dreamer2/wrappers.py", line 45, in step
    time_step = self._env.step(action)
  File "/root/miniconda3/envs/dreamer/lib/python3.7/site-packages/dm_control/rl/control.py", line 106, in step
    self._physics.step(self._n_sub_steps)
  File "/root/miniconda3/envs/dreamer/lib/python3.7/site-packages/dm_control/mujoco/engine.py", line 176, in step
    mujoco.mj_step(self.model.ptr, self.data.ptr, nstep)
  File "/root/miniconda3/envs/dreamer/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/root/miniconda3/envs/dreamer/lib/python3.7/site-packages/dm_control/mujoco/engine.py", line 344, in check_invalid_state
    raise _control.PhysicsError(message)
dm_control.rl.control.PhysicsError: Physics state is invalid. Warning(s) raised: mjWARN_BADCTRL

It seems that models were not well trained because of abnormal gradients. Specifically, paritial results of post, prior = self._dynamics.observe(embed, data['action']) are nan:

prior['deter']
Out[23]: 
<tf.Tensor: shape=(25, 50, 200), dtype=float16, numpy=
array([[[ 0.00545, -0.0837 , -0.0209 , ...,  0.06793, -0.1763 ,
          0.0151 ],
        [ 1.     , -1.     , -0.0209 , ..., -1.     , -1.     ,
          0.0151 ],
        [-1.     , -1.     , -0.0209 , ..., -1.     ,  1.     ,
          0.0151 ],
        ...,
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan]],
       [[ 0.00545, -0.0837 , -0.0209 , ...,  0.06793, -0.1763 ,
          0.0151 ],
        [ 0.00545, -0.0837 , -0.0209 , ...,  0.06793,  1.     ,
          0.0151 ],
        [ 0.00545, -0.0837 , -0.0209 , ..., -1.     ,  1.     ,
          0.0151 ],
        ...,
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan]],
       [[ 0.00545, -0.0837 , -0.0209 , ...,  0.06793, -0.1763 ,
          0.0151 ],
        [ 1.     , -0.0837 , -0.0209 , ...,  0.06793, -0.1763 ,
          0.0151 ],
        [ 1.     , -0.0837 ,  1.     , ...,  0.06793, -0.1763 ,
          0.0151 ],
        ...,
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan]],
       ...,
       [[ 0.00545, -0.0837 , -0.0209 , ...,  0.06793, -0.1763 ,
          0.0151 ],
        [-1.     , -0.0837 , -0.0209 , ...,  1.     , -1.     ,
         -1.     ],
        [-1.     , -1.     ,  1.     , ...,  1.     ,  1.     ,
         -1.     ],
        ...,
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan]],
       [[ 0.00545, -0.0837 , -0.0209 , ...,  0.06793, -0.1763 ,
          0.0151 ],
        [ 1.     , -0.0837 , -1.     , ...,  0.06793,  1.     ,
          0.0151 ],
        [ 1.     , -0.0837 ,  1.     , ...,  0.06793,  1.     ,
          0.0151 ],
        ...,
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan]],
       [[ 0.00545, -0.0837 , -0.0209 , ...,  0.06793, -0.1763 ,
          0.0151 ],
        [-1.     , -0.0837 , -0.0209 , ...,  0.06793,  1.     ,
          0.0151 ],
        [ 1.     , -1.     , -0.0209 , ...,  0.06793,  1.     ,
          0.0151 ],
        ...,
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan],
        [     nan,      nan,      nan, ...,      nan,      nan,
              nan]]], dtype=float16)>

I tried to set the config.precision to be 32. Here is the log:

Simulating agent for 4995000 steps.
Found 5296820 model parameters.
Found 578412 actor parameters.
Found 413601 value parameters.
Start evaluation.
Training for 100 steps.
[5000] expl_amount 0.0 / model_grad_norm nan / value_grad_norm 40.3 / actor_grad_norm 0.3 / prior_ent 61.4 / post_ent 43.3 / image_loss 11650.7 / reward_loss 1.2 / div 77.2 / model_loss 5864.5 / value_loss 6.4 / actor_loss -6.9 / action_ent -23.6
Test episode of length 1000 with return 29.2.
Start collection.
Train episode of length 1000 with return 13.9.
Training for 100 steps.
[6000] expl_amount 0.3 / model_grad_norm 4468751873343488.0 / value_grad_norm 61.0 / actor_grad_norm 0.1 / prior_ent 73.1 / post_ent 62.4 / image_loss 304112.1 / reward_loss 3.8 / div 234.1 / model_loss 152175.0 / value_loss 6.3 / actor_loss -14.7 / action_ent -51.5 / fps 1.0
Train episode of length 1000 with return 19.1.
Training for 100 steps.
[7000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm 81.3 / actor_grad_norm 0.1 / prior_ent 76.6 / post_ent 68.2 / image_loss 8496855334057326588896827408384.0 / reward_loss 1067895404697208119929243435008.0 / div inf / model_loss inf / value_loss 5.0 / actor_loss -25.3 / action_ent -63.6 / fps 9.3
Train episode of length 1000 with return 27.9.
Training for 100 steps.
[8000] expl_amount 0.3 / model_grad_norm 3648154079592448.0 / value_grad_norm 54.8 / actor_grad_norm 0.0 / prior_ent 86.7 / post_ent 75.9 / image_loss 11476.0 / reward_loss 2.2 / div 188.8 / model_loss 5833.5 / value_loss 3.2 / actor_loss -13.2 / action_ent -69.6 / fps 9.3
Train episode of length 1000 with return 29.9.
Training for 100 steps.
[9000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm 347.4 / actor_grad_norm 0.2 / prior_ent 106.9 / post_ent 84.1 / image_loss 4700695580858688274767510765568.0 / reward_loss 472017832713614273488978182144000.0 / div inf / model_loss inf / value_loss 33.1 / actor_loss 19.2 / action_ent -77.6 / fps 9.3
Train episode of length 1000 with return 15.8.
Training for 100 steps.
[10000] expl_amount 0.3 / model_grad_norm inf / value_grad_norm 688.1 / actor_grad_norm 1.6 / prior_ent 99.8 / post_ent 82.3 / image_loss 11431.8 / reward_loss 9.1 / div 230.2 / model_loss 5835.5 / value_loss 55.7 / actor_loss 47.7 / action_ent -69.7 / fps 9.3
Train episode of length 1000 with return 5.8.
...
[981000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm 80.4 / actor_grad_norm 0.0 / prior_ent 109.3 / post_ent 99.2 / image_loss 181376736.0 / reward_loss 1.0 / div 411.2 / model_loss 90688576.0 / value_loss 0.5 / actor_loss 85.6 / action_ent -84.5 / fps 21.8
Train episode of length 1000 with return 33.0.
Training for 100 steps.
[982000] expl_amount 0.3 / model_grad_norm 3922079252480.0 / value_grad_norm 76.0 / actor_grad_norm 0.0 / prior_ent 109.3 / post_ent 99.2 / image_loss 184545072.0 / reward_loss 1.1 / div 410.5 / model_loss 92272736.0 / value_loss 0.5 / actor_loss 85.6 / action_ent -84.5 / fps 23.0
Train episode of length 1000 with return 11.2.
Training for 100 steps.
[983000] expl_amount 0.3 / model_grad_norm nan / value_grad_norm 89.9 / actor_grad_norm 0.0 / prior_ent 109.3 / post_ent 99.1 / image_loss 262517074079239897088.0 / reward_loss 9597141123072.0 / div 28321263910912.0 / model_loss 131258545835712970752.0 / value_loss 0.6 / actor_loss 84.9 / action_ent -84.5 / fps 23.5
Train episode of length 1000 with return 31.0.
Training for 100 steps.

There is no significant improvent after 0.98 million trainning steps, and both the model losses and gradients are not stable. Here are some intermediate results:

likes
Out[12]: 
{'image': <tf.Tensor: shape=(), dtype=float32, numpy=-11762.033>,
 'reward': <tf.Tensor: shape=(), dtype=float32, numpy=-0.95620227>}
 
model_loss
Out[13]: <tf.Tensor: shape=(), dtype=float32, numpy=5882.9946>

model_norm
Out[14]: <tf.Tensor: shape=(), dtype=float32, numpy=nan>

The python packages of my env are as follows:

Package                 Version
----------------------- -----------
absl-py                 2.1.0
astunparse              1.6.3
backcall                0.2.0
cachetools              4.2.4
certifi                 2022.12.7
charset-normalizer      3.4.0
cloudpickle             1.3.0
cycler                  0.11.0
decorator               5.1.1
dm-control              1.0.13
dm-env                  1.6
dm-tree                 0.1.8
gast                    0.3.3
glfw                    2.8.0
google-auth             1.35.0
google-auth-oauthlib    0.4.6
google-pasta            0.2.0
grpcio                  1.62.3
Gymnasium               0.26.3
gymnasium-notices       0.0.1
h5py                    2.10.0
idna                    3.10
imageio                 2.31.2
imageio-ffmpeg          0.5.1
importlib-metadata      6.7.0
ipython                 7.34.0
jedi                    0.19.2
Keras-Preprocessing     1.1.2
kiwisolver              1.4.5
labmaze                 1.0.6
lxml                    5.3.0
Markdown                3.4.4
MarkupSafe              2.1.5
matplotlib              3.1.1
matplotlib-inline       0.1.6
mujoco                  2.3.6
numpy                   1.18.5
nvidia-ml-py            12.535.161
nvitop                  1.3.2
oauthlib                3.2.2
opt-einsum              3.3.0
parso                   0.8.4
pexpect                 4.9.0
pickleshare             0.7.5
Pillow                  9.5.0
pip                     22.3.1
prompt_toolkit          3.0.48
protobuf                3.20.3
psutil                  6.1.0
ptyprocess              0.7.0
pyasn1                  0.5.1
pyasn1-modules          0.3.0
Pygments                2.17.2
PyOpenGL                3.1.7
pyparsing               3.1.4
python-dateutil         2.9.0.post0
requests                2.31.0
requests-oauthlib       2.0.0
rsa                     4.9
scipy                   1.4.1
setuptools              65.6.3
six                     1.17.0
tensorboard             2.2.2
tensorboard-data-server 0.6.1
tensorboard-plugin-wit  1.8.1
tensorflow              2.2.0
tensorflow-estimator    2.2.0
tensorflow-probability  0.10.0
termcolor               2.3.0
tqdm                    4.67.1
traitlets               5.9.0
typing_extensions       4.7.1
urllib3                 2.0.7
wcwidth                 0.2.13
Werkzeug                2.2.3
wheel                   0.38.4
wrapt                   1.16.0
zipp                    3.15.0