Fix vertical_transport with vanleer_limiter on GPU

[nefrathenrici]

Purpose

Closes #4046

Previously, we encountered an InvalidIRError from the van Leer limiter on CUDA when using itime. This PR fixes the issue by converting dt to an FT before passing it to ᶠlin_vanleer.
We currently convert FT(dt) at a lot of call sites, perhaps we should follow up on this PR and push these conversions into the corresponding methods.
No automated tests catch this, perhaps we should use itime more widely.

Reproducer:

using Revise, Infiltrator 
ENV["CLIMACOMMS_CONTEXT"]="SINGLETON"
ENV["CLIMACOMMS_DEVICE"]="CUDA"

import CUDA, ClimaAtmos as CA

config_dict = Dict(
    "apply_limiter" => true,
    "h_elem" => 12,
    "z_elem" => 25,
    "rayleigh_sponge" => false,
    "viscous_sponge" => false,
    "dt" => "1secs",
    "t_end" => "10secs",
    "log_progress" => false,
    "moist" => "nonequil",
    "surface_setup" => "DefaultMoninObukhov",
    "rad" => "allskywithclear",
    "vert_diff" => false,
    "precip_model" => "1M",
    "turbconv" => "diagnostic_edmfx",
    "edmfx_sgs_mass_flux" => true,
    "edmfx_sgs_diffusive_flux" => true,
    "edmfx_sgsflux_upwinding" => "vanleer_limiter",
    "edmfx_tracer_upwinding" => "vanleer_limiter",
    "use_itime" => true,

)
config = CA.AtmosConfig(config_dict)
sim = CA.get_simulation(config)

dt = sim.integrator.dt
cρ = sim.integrator.u.c.ρ
f_u3 = sim.integrator.p.precomputed.ᶠu³
cχ = sim.integrator.u.c.ρq_tot ./ sim.integrator.u.c.ρ

Base.Broadcast.materialize(CA.vertical_transport(cρ, f_u3, cχ, dt, Val(:vanleer_limiter)))

Content

---

• I have read and checked the items on the review checklist.

[trontrytel]

Is it a fix to this issue: #4046? The error looks different?

[nefrathenrici]

Is it a fix to this issue: #4046? The error looks different?

This PR should fix #4046, here is the error log from the current main.

[trontrytel]

Should we maybe change one of the GPU CI cases to run with vanleer to test this?

[nefrathenrici]

Should we maybe change one of the GPU CI cases to run with vanleer to test this?

Yes, do you have one in mind? We need to use itime with vanleer to test this.

[trontrytel]

I don't know where itime is used or what it is? Any GPU spherical prognostic edmf 1M simulation should do. Maybe we can add one to the CI or switch one of the 0M ones. It will probably not run for too long right now, but at least we will be testing if its compiling

[nefrathenrici]

I don't know where itime is used or what it is? Any GPU spherical prognostic edmf 1M simulation should do. Maybe we can add one to the CI or switch one of the 0M ones. It will probably not run for too long right now, but at least we will be testing if its compiling

ITime is another way of representing time in a simulation. @ph-kev wrote it so he can correct me but I think the most important advantage of itime over a float64 is to prevent floating point errors, particularly in long simulations.

The issue here was that dt is an ITime, not a float, so it needed to be converted before being passed to the limiter. The FT(dt) pattern to fix this should be cleaned up in a future PR, since this happens in many places, not just for the vanleer limiter.

I think we can just set use_itime: true in an existing GPU simulation. I can choose one and test it to ensure using itime breaks on main.

[trontrytel]

Sound good. Not sure if we have a

• GPU
• prognostic EDMF
• 1M
• vanleer transport for tracers
  in the CI. But of not, we should definitely have one

[szy21]

I think this fixes the diagnostic + 1M + vanleer. It probably also fixes prognostic + 1M + vanleer. The problem with testing prognostic + 1M on gpu on ci is that it runs out of parameter memory on the central gpu.