Why does Turtlebot4 need external time server? Why not the RPi?

[slowrunner]

Robot Model

Turtlebot4 Lite

ROS distro

Jazzy

Networking Configuration

Discovery Server

OS

Ubuntu 24.04

Built from source or installed?

Built from Source

Package version

turtlebot4/turtlebot4 dc744fe
turtlebot4/turtlebot4_robot 28e115f

Type of issue

Navigation (SLAM, Nav2 etc.)

Expected behaviour

Expect turtlebot4 installation/setup would configure Create3 to sync time with Raspberry Pi.

(interesting - curling the turtlebot4-setup script to bash and running it did not setup chrony, but installing sudo apt install ros-jazzy-turtlebot4-setup did setup and start the chronyd)

Why not run chrony on the RPi?

A long time ago the Create3 folk suggested setting up chrony on the Raspberry Pi but the Issue 530 suggests an external time server. Why not use the turtlebot4 setup of chrony on the RPi?

ubuntu@TB5WaLI:~/TB5-WaLI$ sudo chronyc clients
Hostname                      NTP   Drop Int IntL Last     Cmd   Drop Int  Last
===============================================================================
192.168.186.2                   3      0  12   -   273       0      0   -     -
localhost                       0      0   -   -     -       1      0   -    10

Actual behaviour

Create3 and Raspberry Pi connected LIDAR topics report unsynchronized frame clock times sometimes causing terrible SLAM quality.

Error messages

Create3:

Feb  9 03:13:19 iRobot-883C456BE1E840DC89E64B28B4F75B53 user.notice ntpd: Script called in periodic mode
Feb  9 03:13:39 iRobot-883C456BE1E840DC89E64B28B4F75B53 user.notice ntpd: ntpd: bad address '3.irobot.pool.ntp.org'
Feb  9 03:13:44 iRobot-883C456BE1E840DC89E64B28B4F75B53 user.notice ntpd: ntpd: bad address '2.irobot.pool.ntp.org'
Feb  9 03:13:49 iRobot-883C456BE1E840DC89E64B28B4F75B53 user.notice ntpd: ntpd: bad address '1.irobot.pool.ntp.org'
Feb  9 03:13:54 iRobot-883C456BE1E840DC89E64B28B4F75B53 user.notice ntpd: ntpd: bad address '0.irobot.pool.ntp.org'
Feb  9 03:14:18 iRobot-883C456BE1E840DC89E64B28B4F75B53 user.notice create-platform: [INFO] [1739070858.563552425] [_do_not_use.system_health]: CPU usage: max 66 [%] mean 59 [%] RAM usage: 35/59 [MB]
Feb  9 03:15:18 iRobot-883C456BE1E840DC89E64B28B4F75B53 user.notice create-platform: [INFO] [1739070918.567063903] [_do_not_use.system_health]: CPU usage: max 67 [%] mean 59 [%] RAM usage: 35/59 [MB]
Feb  9 03:15:25 iRobot-883C456BE1E840DC89E64B28B4F75B53 user.notice ntpd: ntpd: timed out waiting for 192.168.186.1, reach 0x00, next query in 523s

To Reproduce

1. Download turtlebot4 software via turtlebot4-setup script per
   https://github.com/turtlebot/turtlebot4_setup/tree/jazzy?tab=readme-ov-file#download-and-run-the-setup-script
2. Setup turtlebot4 (and Create3) with Discovery Server
3. launch async SLAM:

ros2 launch turtlebot4_navigation slam.launch.py sync:=false params:=/opt/ros/jazzy/share/turtlebot4_navigation/config/slam.yaml

4. sudo apt install ros-jazzy-turtlebot4-setup
   Chronyd will be setup and running

Other notes

Guidance is to set up an external time server for both Create3 and Raspberry Pi: Issue 530

Default Create3 ntp.conf:

# iRobot servers
server 0.irobot.pool.ntp.org iburst
server 1.irobot.pool.ntp.org iburst
server 2.irobot.pool.ntp.org iburst
server 3.irobot.pool.ntp.org iburst
# SBC servers
server 192.168.186.1 iburst
server 192.168.186.3 iburst

/etc/turtlebot4/chrony.conf:

pool ntp.ubuntu.com        iburst maxsources 4
pool 0.ubuntu.pool.ntp.org iburst maxsources 1
pool 1.ubuntu.pool.ntp.org iburst maxsources 1
pool 2.ubuntu.pool.ntp.org iburst maxsources 2

# Enable serving time to ntp clients on 192.168.186.0 subnet.
allow 192.168.186.0/24

# Allow local sync
local stratum 10

# This directive specify the location of the file containing ID/key pairs for
# NTP authentication.
keyfile /etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate
# information.
driftfile /var/lib/chrony/chrony.drift

# Uncomment the following line to turn logging on.
#log tracking measurements statistics

# Log files location.
logdir /var/log/chrony

# Stop bad estimates upsetting machine clock.
maxupdateskew 100.0

# This directive enables kernel synchronisation (every 11 minutes) of the
# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.
rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than
# one second, but only in the first three clock updates.
makestep 1 3

RPi Time Status:

ubuntu@TB5WaLI:~/TB5-WaLI$ ps -ef | grep chron
_chrony    40614       1  0 19:03 ?        00:00:00 /usr/sbin/chronyd -F 1
_chrony    40616   40614  0 19:03 ?        00:00:00 /usr/sbin/chronyd -F 1



ubuntu@TB5WaLI:~/TB5-WaLI$ timedatectl status
               Local time: Sat 2025-02-08 21:12:57 EST
           Universal time: Sun 2025-02-09 02:12:57 UTC
                 RTC time: Sun 2025-02-09 02:12:57
                Time zone: America/New_York (EST, -0500)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

[hilary-luo]

@slowrunner this response is going to focus on the conceptual topic since I haven't looked into the details of the setup script vs setup tool yet.

I think you correctly figured out that the expected normal operation is that chrony be running on the pi (syncing with NTP servers on the internet) and that the create3 be served by the NTP server on the pi. Most people use the default images so that is probably why we haven't seen this come up before as it is set up correctly in those images. However there are limitations with this configuration. It ensures that the create3 timestamp always match with the pi timestamp and if the pi has an internet connection it also will match the internet time servers. This is sufficient when running all nodes onboard the Turtlebot4, or when working with computers where both the pi and the offboard computers have internet access and are syncing with the internet time servers.

If the system is not connected to the internet then you run into an issue where the pi4 does not have the ability to keep time while it is powered off, which can be problematic in itself if events are being scheduled or logged based on time of day. Additionally, the disconnected pi would not match the timestamp on other devices within the network. Whenever communication is intended between nodes on different computers it is important to have their time synchronized. This also becomes an issue if the offboard computers have just had time synchronization with the internet disabled. These are the situations where we recommend hosting a local server on the local network, to replace the usage of the internet time servers. Even in the case for devices like the pi5 where there is support for RTC, it is still important to have that time synchronized across devices.

[slowrunner]

Super - The rationale for some folk to establish the time server on the visualization/remote computing platform was foreign to me due to my narrow "only onboard computing" focus. Running the time server on the Pi is perfect for my use case, where my Pi is on 24/7/365 and connected to the Internet 99.x% of that.

Thanks.