Performance Issue in updateHeights with High-Resolution Custom Terrain Providers

[calogeromauceri]

What happened?

Summary

When using a high-resolution custom terrain provider (e.g., ArcGIS WorldElevation3D), updateHeights in QuadtreePrimitive struggles to keep up with entity height updates. The intersections with the terrain take significantly longer (often exceeding 50ms), while the time limit for updateHeights is only 2ms.

This results in a growing queue of tilesToUpdateHeights, leading to poor entity positioning and potential performance degradation. The issue does not occur when using Cesium.createWorldTerrainAsync().

Reproduction steps

Sandcastle example

Observations & Debugging

To analyze the issue, I added logging inside updateHeights as follows:

const currentTime = getTimestamp();
const deltaTime = currentTime - startTime; // Calculate elapsed time
if (currentTime >= endTime) {
    console.log(`Time slice exceeded: Δt = ${deltaTime} ms (limit: ${timeSlice} ms)`);
    console.log(`Queue size: tilesToUpdateHeights.length = ${tilesToUpdateHeights.length}`);
    console.log(`Last processed tile index: ${primitive._lastTileIndex}`);
    timeSliceMax = true;
    break;
}

After all tiles have been loaded, I keep seeing these logs:

Time slice exceeded: Δt = 53.59999990463257 ms (limit: 2 ms)
Queue size: tilesToUpdateHeights.length = 56
Last processed tile index: 1
Time slice exceeded: Δt = 86.2999997138977 ms (limit: 2 ms)
Queue size: tilesToUpdateHeights.length = 56
Last processed tile index: 1
...

Key Findings:

• The intersection time consistently exceeds 50ms, whereas the function is only allowed 2ms per update cycle.
• This causes tilesToUpdateHeights to grow indefinitely, preventing entity heights from updating.
• Switching to Cesium.createWorldTerrainAsync() eliminates the issue, suggesting that high-resolution terrain data slows down intersections significantly.

Sandcastle example

https://sandcastle.cesium.com/#c=rVTbbhs3EP2VgdCHla2Qko00vihGDTtNAsSxYSntQ7ZwaO5oRZRLCuSsFNXwv2f2JkuN+5CgxALkkjNnzpwZUnsXCZYGVxjgNThcwQVGUxbij3ovSXu6/r/wjpRxGNJe/zR1qZN7qaOwhofUQQsgokaHgjAENr0JfmmyCrayAFArZahD1wEV4Z8+2GzamJ/HtdNJhf0IWpGeQ8IHPvSbCCvjMr8SymKg5MvvyljMgDxYr3huIAT88lD7PH5pcFK3JyuuP8/zPOi37yfTKtobi0tFxrvprp+YBV98CjZp3AHS3pxoEU+kxM7lMBMq6NxEoX0hm6UMGElGDEvDEstai02Mw0vZhuHV+0LlOGHDSv1BE+b/Fip1O9rk1t+jyHBB8ynTPM/ZK1JLiRuFQol1G8xKpyvCkCPdKg5dXM9mESlp6QSkMjhIrhTNRagN+OgFDMXLPuzxNKzH6BSkhEmhrIXGCnyNw+yqMLruU2QFmM2FCuTzoBZzo5lLW6rt3boml5gHxJgc/SqOD14O4OCVOD46Gg1g2K+jvWnQIJZFYWgriiNDBiNDf/6Lk5z5AIlFAsM7w1OexjCq5v39NsnGz3qXGyozfOJUZ02+Y/IMf/Hktf+9hlV1NvDcGD+O3jn9B/gTfJ31msHbPuhUECrLkoem6RY+mqrYJ9ua80q5wx3FNzkNNqwHcMxl7rfdW/gM7Qk8dFemGmUwJ3x3hJD8TVSxsHipSMnaNsom4pVy7eqOl9ym95sb0Yw5mnxOtzhjLZzGDdN3u/vi4sP51c3d9Pru7e31p4+XOxiFcaYoixvzFe3E/MMgo4OjXQv1tbKYaL5mfDocbp0+1uvHTt2NjosyzpNG5frSbd05Ckr/zS9MV4LO5/Ow6r/eoDeOtLZ41gX5zRQLH4glswmLRchi8XMa5X3JOCR0jE3nAIzltus4M0sw2etnXnXQVsXIJ7PS1lmnvbOxZPvvXKuXxLj8mtvNqnVlNh+dfWg2hRBjyb/Pe5L39l6FfyF/Aw

Environment

Browser: Any
CesiumJS Version: 1.126
Operating System: Any

[ggetz]

Thanks for noting this and doing some initial exploration @calogeromauceri!

At first glance, it appears that this could be helped by moving some of these operations out of the main thread and into a worker or over to the GPU. There's been some significant brainstorming for this over in #8481.

It seems like your report here is covered by that existing issue. I'm going to close your issue to keep the discussion in one place, and if you have any further input on this, please post it there instead.

Let me know if you think this is fundamentally a different issue, and we can reopen this one.

[calogeromauceri]

Thanks for the detailed response, @ggetz!

Fixing the globe picking slowness would be ideal, as it is causing significant issues in our high-resolution terrain.

However, a quick solution for the specific issue I reported would be to avoid clearing the custom data for a tile at every render.
Currently, in QuadtreeTile.prototype._updateCustomData, the tile's custom data is cleared on each call.
Instead, restoring the tile's custom data when available resolves the problem.

Here's what I mean

QuadtreeTile.prototype._updateCustomData = function (
  frameNumber,
  added,
  removed
) {
  let customData = this.customData;

  let i;
  let rectangle;


  if (defined(added) && defined(removed)) {
    customData = customData.filter(function (value) {
      return removed.indexOf(value) === -1;
    });
    this._customData = customData;

    rectangle = this._rectangle;
    for (i = 0; i < added.length; ++i) {
      const data = added[i];
      
      // Check if the same data exists in removed
      const existingData = removed.find((d) => {
      return (
        CesiumMath.equalsEpsilon(d?.positionCartographic?.longitude, data?.positionCartographic?.longitude, CesiumMath.EPSILON13) &&
        CesiumMath.equalsEpsilon(d?.positionCartographic?.latitude, data?.positionCartographic?.latitude, CesiumMath.EPSILON13) &&
        d.level >= 0 &&
        defined(d.positionOnEllipsoidSurface));
      });

      if (existingData) {
        data.positionOnEllipsoidSurface = existingData.positionOnEllipsoidSurface;
        data.level = existingData.level;
      }
      
      if (Rectangle.contains(rectangle, data.positionCartographic)) {
        customData.push(data);
      }
    }

    this._frameUpdated = frameNumber;
  } else {
    // interior or leaf tile, update from parent
    const parent = this._parent;
    if (defined(parent) && this._frameUpdated !== parent._frameUpdated) {
      customData.length = 0;

      rectangle = this._rectangle;
      const parentCustomData = parent.customData;
      
      for (i = 0; i < parentCustomData.length; ++i) {
        const data = parentCustomData[i];
        if (Rectangle.contains(rectangle, data.positionCartographic)) {
          customData.push(data);
        }
      }

      this._frameUpdated = parent._frameUpdated;
    }
  }
};

This change ensures that tile custom data is retained when available, instead of being needlessly discarded and recomputed each frame. While optimizing globe picking would be the best long-term solution, this fix addresses the immediate issue effectively.

I'm happy to help with fixing the underlying problem as well!

[ggetz]

Thanks @calogeromauceri! We're certainly open to a shorter-term fix if it's relatively targeted and general-purpose.

I agree it seems like a major bottleneck that positionOnEllipsoidSurface is being recomputed each call.

I'm a bit hesitant to rely on the position equality checks here though, as it could be error-prone. Looking through the relevant code now, we could definitely make improvements to how the "updateHeight" callbacks work across frames. If you have the bandwidth to look through a slightly broader solution here, we'd be happy to discuss.

Perhaps it would make sense to create a cache for positionOnEllipsoidSurface based on both the level and and ID of the subscriber?

It also looks like there are some clues in #8006.