1370-2 Add support for reading templated SECIR models from JSON data #1375

julianlitz · 2025-09-15T14:33:46Z

Changes and Information

This is part 2 of adding support for reading templated SECIR models from JSON data.

In PR #1374, we moved the IO functions from the source file to the header to enable their implementation. This branch builds on that work, so its initial commits are identical to the changes introduced there. Subsequent commits focus on addressing AD compatibility and directly modifying the implementation.

Please briefly list the changes (main added features, modified items, or fixed issues):

Use FP instead of double in IO functions
Ensure to correctly convert AD types to integers
Added ad::value() to convert a FP to ScalarType for the ParameterDistribution
Data is read using H5T_NATIVE_DOUBLE and then converted to FP later

If need be, add additional information and what the reviewer should look out for in particular:

Merge Request - Guideline Checklist

Please check our git workflow. Use the draft feature if the Pull Request is not yet ready to review.

Checks by code author

Every addressed issue is linked (use the "Closes #ISSUE" keyword below)
New code adheres to coding guidelines
No large data files have been added (files should in sum not exceed 100 KB, avoid PDFs, Word docs, etc.)
Tests are added for new functionality and a local test run was successful (with and without OpenMP)
Appropriate documentation within the code (Doxygen) for new functionality has been added in the code
Appropriate external documentation (ReadTheDocs) for new functionality has been added to the online documentation
Proper attention to licenses, especially no new third-party software with conflicting license has been added
(For ABM development) Checked benchmark results and ran and posted a local test above from before and after development to ensure performance is monitored.

Checks by code reviewer(s)

Corresponding issue(s) is/are linked and addressed
Code is clean of development artifacts (no deactivated or commented code lines, no debugging printouts, etc.)
Appropriate unit tests have been added, CI passes, code coverage and performance is acceptable (did not decrease)
No large data files added in the whole history of commits(files should in sum not exceed 100 KB, avoid PDFs, Word docs, etc.)
On merge, add 2-5 lines with the changes (main added features, changed items, or corrected bugs) to the merge-commit-message. This can be taken from the briefly-list-the-changes above (best case) or the separate commit messages (worst case).

Closes #1370, #1382, #1383, #1384

codecov · 2025-09-15T16:52:11Z

Codecov Report

❌ Patch coverage is 97.94521% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.29%. Comparing base (71833e4) to head (11bb56c).

Files with missing lines	Patch %	Lines
cpp/memilio/io/mobility_io.h	92.59%	2 Missing ⚠️
cpp/models/ode_secir/parameters_io.h	96.00%	2 Missing ⚠️
cpp/models/ode_secirts/parameters_io.h	96.15%	2 Missing ⚠️
cpp/models/ode_secirvvs/parameters_io.h	98.95%	2 Missing ⚠️
...memilio/mobility/metapopulation_mobility_instant.h	83.33%	1 Missing ⚠️

Additional details and impacted files

@@                      Coverage Diff                       @@
##           1370-move-source-to-header    #1375      +/-   ##
==============================================================
- Coverage                       97.31%   97.29%   -0.03%     
==============================================================
  Files                             171      171              
  Lines                           15213    15194      -19     
==============================================================
- Hits                            14805    14783      -22     
- Misses                            408      411       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

julianlitz · 2025-09-16T08:48:17Z

cpp/memilio/ad/ad.h

 #include <cmath>
 #include <limits>

+// Extend automatic differentiation (AD) library to support std::round.


We extend AD support for std::round and std::trunc. We used the same structure as from std::floor, provided in ad/include/ad.hpp.
Note that we don't modify the original ad/include/ad.hpp library.

julianlitz · 2025-09-16T08:49:22Z

cpp/memilio/data/analyze_result.h

                               });
                std::sort(single_element_ensemble.begin(), single_element_ensemble.end());
-                percentile[node][time][elem] = single_element_ensemble[static_cast<size_t>(num_runs * p)];
+                percentile[node][time][elem] = single_element_ensemble[static_cast<size_t>(floor(num_runs * p))];


To cast a FP to size_t, we need to call either floor, ceil, round or trunc.

julianlitz · 2025-09-16T08:52:11Z

cpp/memilio/mobility/graph.h

+        tnt_value.set_distribution(
+            mio::ParameterDistributionUniform(0.8 * ad::value(tnt_capacity), 1.2 * ad::value(tnt_capacity)));


ParameterDistributionUniform takes in ScalarType as an argument.
To cast FP to ScalarType we need to call ad::value.

julianlitz · 2025-09-16T08:52:37Z

cpp/memilio/mobility/graph.h

+                compartment_value.set_distribution(mio::ParameterDistributionUniform(
+                    0.9 * ad::value(compartment_value.value()), 1.1 * ad::value(compartment_value.value())));


ParameterDistributionUniform takes in ScalarType as an argument.
To cast FP to ScalarType we need to call ad::value.

julianlitz · 2025-09-16T08:58:50Z

cpp/models/ode_secirvvs/parameters_io.h

        if (it != vregion.end()) {
            auto region_idx = size_t(it - vregion.begin());
-            if (rki_entry.date == offset_date_by_days(date, int(-delay))) {
+            if (rki_entry.date == offset_date_by_days(date, int(trunc(-delay)))) {


int(-x) seems to be equivalent to int(std::trunc(-x)).
This is needed for AD to convert a FP to int correctly.

julianlitz · 2025-09-16T09:00:43Z

cpp/models/ode_secirvvs/parameters_io.h

+                round(static_cast<FP>(model[county].parameters.template get<TimeExposed<FP>>()[(AgeGroup)group]))));
+            t_InfectedNoSymptoms[county].push_back(static_cast<int>(round(static_cast<FP>(
+                model[county].parameters.template get<TimeInfectedNoSymptoms<FP>>()[(AgeGroup)group] * reduc_t))));
+            t_InfectedSymptoms[county].push_back(static_cast<int>(round(static_cast<FP>(


Cast UncertainValue<FP> to FP.
Then call round to be able to cast to an integer.
We can't call round directly on an Uncertain Value.

julianlitz · 2025-09-26T14:19:19Z

cpp/memilio/mobility/metapopulation_mobility_instant.h

+using test_commuters_expr_t = decltype(test_commuters<FP>(
+    std::declval<Sim&>(), std::declval<Eigen::Ref<Eigen::VectorX<FP>>>(), std::declval<FP>()));


Metaprogramming uses incorrect function declaration.
const& must be removed.

julianlitz · 2025-09-29T11:29:36Z

cpp/memilio/mobility/metapopulation_mobility_instant.h

 }
 template <typename FP, class Sim,
-          std::enable_if_t<is_expression_valid<get_mobility_factors_expr_t, Sim>::value, void*> = nullptr>
+          std::enable_if_t<is_expression_valid<get_mobility_factors_expr_t, FP, Sim>::value, void*> = nullptr>


Important to add a FP in the metaprogramming as well. Otherwise the function can't be found.

julianlitz · 2025-09-29T11:30:15Z

cpp/memilio/mobility/metapopulation_mobility_instant.h

 auto get_mobility_factors(const SimulationNode<FP, Sim>& node, FP t, const Eigen::Ref<const Eigen::VectorX<FP>>& y)
 {
-    return get_mobility_factors<FP, Sim>(node.get_simulation(), t, y);
+    return get_mobility_factors<FP>(node.get_simulation(), t, y);


Using Sim here seems to cause issues. Better remove it and let the compiler choose it for us.

Max: I dont understand how using Sim causes problems at this point. Can u elaborate on that.

Julian: I am not sure why, but using <FP, Sim> instead of in some places connected with the metaprograming, it wont be able to find the overloading function of the specialized model. It is really strange. I think in the apply_mobility function, it is fine to use <FP, Sim> but I wouldn't risk it in the other functions above which transfer to the overloaded function, as it doesn't seem to work properly then.
Better not risk it here and let the compiler decide which template to use as it can be deduced from the arguments anyway.

I am not sure why, but using <FP, Sim> instead of <FP> in some places connected with the metaprograming, it wont be able to find the overloading function of the specialized model. It is really strange. I think in the apply_mobility function, it is fine to use <FP, Sim> but I wouldn't risk it in the other functions above which transfer to the overloaded function, as it doesn't seem to work properly then.
Better not risk it here and let the compiler decide which template to use as it can be deduced from the arguments anyway.

MaxBetzDLR

Changes requested

cpp/memilio/ad/ad.h

MaxBetzDLR · 2025-09-29T13:05:49Z

cpp/memilio/mobility/metapopulation_mobility_instant.h

 auto get_mobility_factors(const SimulationNode<FP, Sim>& node, FP t, const Eigen::Ref<const Eigen::VectorX<FP>>& y)
 {
-    return get_mobility_factors<FP, Sim>(node.get_simulation(), t, y);
+    return get_mobility_factors<FP>(node.get_simulation(), t, y);


Max: I dont understand how using Sim causes problems at this point. Can u elaborate on that.

Julian: I am not sure why, but using <FP, Sim> instead of in some places connected with the metaprograming, it wont be able to find the overloading function of the specialized model. It is really strange. I think in the apply_mobility function, it is fine to use <FP, Sim> but I wouldn't risk it in the other functions above which transfer to the overloaded function, as it doesn't seem to work properly then.
Better not risk it here and let the compiler decide which template to use as it can be deduced from the arguments anyway.

MaxBetzDLR · 2025-09-29T13:13:58Z

cpp/tests/test_odesecir.cpp

    std::string path = mio::path_join(pydata_dir_Germany, "county_current_population.json");
    const std::vector<int> region{1002};
-    auto population_data = mio::read_population_data(path, region).value();
+    auto population_data = mio::read_population_data<double>(path, region).value();


Use a consistent type in all tests. Until now it seems, that ScalarType wasnt used and instead double was picked as FP type.

At the moment the tests use a wild mix of doubles and ScalarType.

The plan is to progress in three steps:

Template missing functionalities (Done)
Here we had a large PR which also removed doubles from the memilio/ and models/ folder.

Template IO Functions (This PR)

Switch from doubles to ScalarType everywhere. These mostly effect the tests and the python binding. (Future PR)

I dont think tests are suppose to use ScalarType. The type should be fixed to produce consistent results

The MEmilio software is currently compiled with a single, fixed floating-point type, which is defined as:

using ScalarType = double;

However, our goal is to support alternative types, such as:

using ScalarType = float;

If we keep double in the tests while compiling the library with a different ScalarType, the build will fail because of type mismatches. Therefore, the tests must also consistently use ScalarType to ensure compatibility across different configurations.

Only the occurrences of double that interact with MEmilio’s numeric types or APIs should be replaced with ScalarType. Plain double usage that is unrelated to MEmilio internals does not necessarily need to be changed.

I think there are very few instances where we use thresholds e.g. 1e-12 which are tuned for doubles. You are right, that these would have to be addressed.

You are right for variables that expect ScalarType, but all these instances, where you added ScalarType in the tests, are as a template instatiation of FP. And in those cases they SHOULD also work with dissimilar types.
So please change that.

I don’t think this PR is the right place to fully sort out the floating-point type policy. Let’s wrap this one up focusing only on adding the missing templates in the IO functions.

About the tests: I’m not sure yet if keeping double everywhere will even compile once ScalarType changes (e.g. to float). My gut feeling is that using whatever ScalarType the library was built with is the safer option — but I’d rather test it properly before we decide.

So I suggest:
Finish this PR first (IO templating only and some bug fixes).
Afterwards I’ll check which tests actually need ScalarType and where double still works.
Then we can discuss it again (maybe with René) and agree on a consistent rule.

Does that sound okay?

MaxBetzDLR · 2025-09-29T13:37:07Z

cpp/models/ode_secir/parameters_io.h

-    std::vector<std::vector<double>>& vnum_InfectedSymptoms, std::vector<std::vector<double>>& vnum_InfectedSevere,
-    std::vector<std::vector<double>>& vnum_icu, std::vector<std::vector<double>>& vnum_death,
-    std::vector<std::vector<double>>& vnum_rec, const std::vector<std::vector<int>>& vt_Exposed,
+    std::vector<std::vector<FP>>& vnum_Exposed, std::vector<std::vector<FP>>& vnum_InfectedNoSymptoms,


Should those values really be FP or ScalarType? The IO could work a lot without FP and just cast to that type when writing into the model. Similar with input parameters like scaling_factor_inf

I would argue that templating the read_confirmed_cases_data<FP> is the straight forward and an easy way to do it.
Manually casting lots of std::vectors to FP in some intermediate step where I don't know where this should even happen doesn't sound great.

I like the current approach where we read in something from a file and directly convert that value to float, double or AD.

Add support for reading templated SECIR models from JSON

8e9f700

julianlitz commented Sep 16, 2025

View reviewed changes

Safer FP to int conversion

ccddcd1

julianlitz commented Sep 16, 2025

View reviewed changes

julianlitz self-assigned this Sep 16, 2025

julianlitz added loc::backend This issue concerns the C++ backend implementation. class::improvement Cleanup that doesn't affect functionality labels Sep 16, 2025

julianlitz requested review from MaxBetzDLR and reneSchm September 16, 2025 09:38

reneSchm changed the base branch from main to 1370-move-source-to-header September 16, 2025 09:45

julianlitz added 2 commits September 16, 2025 09:49

std::round and static_cast

5641fbd

Fix FP metaprogramming madness.

052e31f

julianlitz commented Sep 26, 2025

View reviewed changes

julianlitz added 2 commits September 26, 2025 19:40

FP fixes in Graph_Simulation.h

31e299a

Resolve Issue 1382, 1383, 1384

c20d181

This was linked to issues Sep 29, 2025

examples/ode_secir_graph.cpp does not use test_commuters overload #1382

Closed

Type mismatch in metaprogramming for test_commuters #1383

Closed

Missing template parameter FP in the metaprogramming validation #1384

Closed

julianlitz commented Sep 29, 2025

View reviewed changes

MaxBetzDLR requested changes Sep 29, 2025

View reviewed changes

Implement suggestion to merge round and trunc namespace region

11bb56c

julianlitz requested a review from MaxBetzDLR October 1, 2025 14:20

MaxBetzDLR added this to MEmilio: Equation-Based-Models Development Oct 9, 2025

MaxBetzDLR moved this to 👀 Review in MEmilio: Equation-Based-Models Development Oct 9, 2025

		tnt_value.set_distribution(
		mio::ParameterDistributionUniform(0.8 * ad::value(tnt_capacity), 1.2 * ad::value(tnt_capacity)));

		compartment_value.set_distribution(mio::ParameterDistributionUniform(
		0.9 * ad::value(compartment_value.value()), 1.1 * ad::value(compartment_value.value())));

		using test_commuters_expr_t = decltype(test_commuters<FP>(
		std::declval<Sim&>(), std::declval<Eigen::Ref<Eigen::VectorX<FP>>>(), std::declval<FP>()));

1370-2 Add support for reading templated SECIR models from JSON data #1375

Are you sure you want to change the base?

1370-2 Add support for reading templated SECIR models from JSON data #1375

Uh oh!

Conversation

julianlitz commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes and Information

Merge Request - Guideline Checklist

Checks by code author

Checks by code reviewer(s)

Uh oh!

codecov bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

julianlitz Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxBetzDLR Sep 29, 2025 • edited by julianlitz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julianlitz Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxBetzDLR left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MaxBetzDLR Sep 29, 2025 • edited by julianlitz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julianlitz Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julianlitz Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julianlitz Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

julianlitz commented Sep 15, 2025 •

edited

Loading

codecov bot commented Sep 15, 2025 •

edited

Loading

julianlitz Sep 16, 2025 •

edited

Loading

MaxBetzDLR Sep 29, 2025 •

edited by julianlitz

Loading

julianlitz Sep 29, 2025 •

edited

Loading

MaxBetzDLR Sep 29, 2025 •

edited by julianlitz

Loading

julianlitz Sep 29, 2025 •

edited

Loading

julianlitz Sep 30, 2025 •

edited

Loading

julianlitz Sep 29, 2025 •

edited

Loading