[EPIC] TTE code sharing and new python packages organization proposal

[margauxraguenel]

What is the requested feature?

Various Python tools were developed by TotalEnergies to visualize and process GEOS inputs/outputs. These codes may however benefit to the whole GEOS community (e.g., #50) and can be shared. Some common needs (e.g., #16) may also be addressed once instead of multiple times. To this end, this epic aims to share in the GEOS repository TotalEnergies Python tools and reorganize/refactor python projects to merge duplicated codes.

---

Is your request related to a specific problem?

• Need to share TotalEnergies Python tools that will benefit to the whole GEOS community,
• Reorganized and refactor python projects, to avoid duplicated codes and improve dependency management (cf. proposed organisation)

---

Describe the solution you'd like

We propose first to integrate TotalEnergies code to geosPythonPackages as they currently are, then a refactoring of the whole project to merge duplicated codes and update package dependencies.

The proposed organisation is the following, as described in the attached document:

• Data structures and processing packages:
  • geos-ats:
  • hdf5-wrapper:
  • geos-xml-tools: reader, writer, split or merge tools, check tools for geos-xml file.
  • geos-mesh: IO, statistics, procesing tools of meshes based on vtk data structure.
  • geomechanics: Data structures and geomechanical properties calculation functions
• API packages:
  • xml-vtk: Convert geos-xml to vtk objects and conversely.
  • geos-prop: Pre-procssing tools as vtk filters including mesh-doctor features
  • geos-posp: Post-processing tools as vtk filters including geos output mesh cleaning and new properties calculation filters
• Script packages:
  • geos-timehistory: plot time serie curves from geos hdf5 (and csv) files
  • geos-xml-viewer: Input object visualization tool from geos-xml based on pyvista
  • mesh-doctor: pre-processing scripts
  • geos-trame: xml interactive viewer and editor
• Paraview (PV) plugins:
  • geos-pv: paraview plugins to wrap API packages

The proposed new organisation aims at splitting processing packages from user-interface packages (i.e., script or IHM-based packages). This organisation was initially proposed by @untereiner who applied it to packages hdf5-wrapper (processing) and geos-timehistory (script).

Advantages are:

• to use same processing modules in multiple user-interface packages
• avoiding duplicated codes and easing code maintenance
• better management of package dependencies

For instance, PR #16 deals with the need to use mesh-doctor tools through Paraview in addition of current scripts. This need requires to either duplicate code or introduce a dependency of Paraview to the whole geos-mesh package, resulting in introducing a dependency of Paraview plugins to pyvista. It is not recommended since Paraview and pyvista are both user-interface modules (the former through an IHM, the latter through APIs) with similar functionalities. The integration of TotalEnergies code into geosPythonPackages will raise similar issues.

---

Proposed roadmap

1. TotalEnergies code sharing:

• First PR with TotalEnergies python code for GEOS post-processing (geos-posp), pre-processing (geos-prep), geos-xml-viewer, and geos-trame packages with minimal work to conform to CI/CD #56
• Settup Github CI/CD with unit tests, mypy and linting checks #58
• Create geos-geomechanics package with geomechanics classes and functions from geos-posp and update dependencies #59
• Create geos-utils package with logger, geos constants, geometry functions, ConnectionSet and update dependencies #63
• Move VTK functions and utilities to geos-mesh package #69
• Move vtk filter AttributeMappingFromCellCoords to geos-mesh + remove AttributeMappingFromCellId + move corresponding PV plugin to geos-pv
• Move TransferAttributesVolumeSurface to geos-mesh + move corresponding PV plugin to geos-pv
• Move VolumeSurfaceMeshMapper to geos-mesh
• Create vtk filter from PVMergeBlocksEnhanced in geos-mesh and move PV plugin to geos-pv #92
• Move GeosBlockExtractor to geos-posp
• Move GeosBlockMerge to geos-posp
• Move vtk filter SurfaceGeomechanics to geos-posp + move corresponding PV plugin to geos-pv
• Move vtk filter GeomechanicsCalculator to geos-posp + move corresponding PV plugin to geos-pv
• Create vtk filter from PVCreateConstantAttributePerRegion PV plugin in geos-mesh and refactor + move PV plugin to geos-pv #106
• Create vtk filter in geos-mesh and its associated pv plugin in geos-pv to fill partial arrays #104
• Create vtk filter from PVExtractMergeBlocksVolumeSurfaceWell PV plugin in geos-posp and refactor + move PVExtractMergeBlocksVolume* plugins to geos-pv
• Create vtk filter from PVGeomechanicsWorkflowVolumeSurfaceWell PV plugin in geos-posp and refactor + move PVGeomechanicsWorkflowVolume* plugins to geos-pv
• Create geos-pv package and add PVGeosLogReader plugin #70
• Move PVMohrCirclePlot to geos-pv
• Integration of Makutu code (Revert "feat: Python interfaces to geos from Makutu repo" #47 and Rework "feat: Python interfaces to geos from Makutu repo" #52)
• Add TotalEnergies existing mesh statistic code to geos-mesh package
• Update documentation (project README.md) to explain new project organisation and package dependencies. #61

2. Create vtk filters for mesh-doctor features #80 :

• Create vtk filter for mesh-doctor generate_global_ids in geos-mesh
• Create vtk filter for mesh-doctor collocated_nodes in geos-mesh
• Create vtk filter for mesh-doctor element_volumes in geos-mesh
• Create vtk filter for mesh-doctor fix_elements_orderings in geos-mesh
• Create vtk filter for mesh-doctor generate_fractures in geos-mesh
• Create vtk filter for mesh-doctor non_conformal in geos-mesh
• Create vtk filter for mesh-doctor self_intersecting_elements in geos-mesh
• Create vtk filter for mesh-doctor supported_elements in geos-mesh
• Create mesh-doctor package and update with vtk filters
• Create Paraview plugins for mesh-doctor vtk filters (PoC / Paraview integration for mesh_doctor  #16)
• Add mesh_stats checks and field_operations features #51 and merge with TotalEnergies codes

3. geos-xml-tools refactoring

• Gather python codes that process GEOS xml files from geos-xml-tools, geos-xml-viewer, geos-trame (Unify the XML parsing of GEOS files #76)
• Create xml-vtk package with xml-vtk conversion vtk filters
• Refactor geos-xml-viewer to update with xml-vtk package
• Refactor geos-trame to update with xml-vtk package
• (Optional) Create Paraview plugins for xml-vtk filters
• File classification  #25

---

PythonPackagesProposal.pdf
updatePythonPackagesProposal_09042025.pdf

[untereiner]

Don't hesitate if you need some help !