Skip to content

Ensure all examples are up to date and verified #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 54 commits into from
Jun 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
cc1ed99
Started examples CI
kzscisoft Apr 25, 2024
4bfd7c2
Switched to Multiparser for OpenFoam example
kzscisoft Apr 26, 2024
7466d93
Use root user in OpenFOAM container
kzscisoft Apr 29, 2024
bf7c124
Try 22.04 in OpenFoam
kzscisoft Apr 29, 2024
287673d
Tru 3.10 in OpenFOAM
kzscisoft Apr 29, 2024
29f674d
Manual Python install for OpenFOAM
kzscisoft Apr 29, 2024
3c93f4d
Add logging example
kzscisoft Apr 29, 2024
3f298cb
Added Optuna example
kzscisoft Apr 29, 2024
b04bb12
Added SU2 example to CI
kzscisoft Apr 29, 2024
384affe
Smaller sized runs for PyTorch example CI
kzscisoft Apr 29, 2024
3a533a3
Use PyTorch CPU in CI
kzscisoft Apr 29, 2024
7418e26
Add CI flags to examples
kzscisoft Apr 29, 2024
5c343f6
Added Tensorflow example
kzscisoft Apr 30, 2024
7e466b7
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Apr 30, 2024
963e5c8
Merge branch 'dev' into examples/geant4
kzscisoft May 20, 2024
7547d08
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft May 30, 2024
4fbed11
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 3, 2024
1b8686b
Install git on OPENFOAM CI
kzscisoft Jun 3, 2024
c6d0a0f
Fix retention period in demos
kzscisoft Jun 3, 2024
cfe6b34
make examples visible to tenant in CI
kzscisoft Jun 3, 2024
3d55e29
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 3, 2024
598c9d6
Fix SU2 example input file error
kzscisoft Jun 3, 2024
303f4ca
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 4, 2024
83c264a
Add example as submodule
kzscisoft Jun 5, 2024
1cea109
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 5, 2024
de2154d
Add GEANT4 example to CI
kzscisoft Jun 5, 2024
01b54b9
Run CI for examples branches
kzscisoft Jun 5, 2024
a2aff4b
Use CMAKE_PREFIX for GEANT4
kzscisoft Jun 5, 2024
ce5f71d
Fix wrong docker tag
kzscisoft Jun 5, 2024
674eaa2
Add missing submodule checkout
kzscisoft Jun 5, 2024
9bb637d
Use recursive submodules in CI
kzscisoft Jun 6, 2024
e9c5455
Added missing environment variables to CI
kzscisoft Jun 6, 2024
ceddd03
Merge branch 'examples/geant4' into 296-ensure-all-examples-are-up-to…
kzscisoft Jun 6, 2024
66e6628
Run example trial only on tags
kzscisoft Jun 6, 2024
43144b6
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 10, 2024
c53d97a
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 10, 2024
05d2d52
Run 296-* on CI
kzscisoft Jun 10, 2024
3a51582
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 10, 2024
d6b3721
Fix OpenFOAM example
kzscisoft Jun 10, 2024
49a38af
Source OpenFOAM environment before running
kzscisoft Jun 10, 2024
efc7de1
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 10, 2024
27bd307
Try . instead of source
kzscisoft Jun 10, 2024
6a9717a
Use bash shell explicitly
kzscisoft Jun 10, 2024
93989ff
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 10, 2024
373e308
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 14, 2024
7940d3a
Enable workflow dispatch for examples
kzscisoft Jun 14, 2024
46d04a2
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 14, 2024
d2d386b
Remove +/- from G4 example
kzscisoft Jun 14, 2024
593d18b
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 14, 2024
c3578ad
Revert name change in G4 particles
kzscisoft Jun 14, 2024
03b7176
Merge branch 'dev' into 296-ensure-all-examples-are-up-to-date-and-ve…
kzscisoft Jun 14, 2024
3a0b220
Fix invalid metadata names
kzscisoft Jun 14, 2024
5dd9b46
Remove square brackets before metrics log
kzscisoft Jun 14, 2024
428490d
Add sharing of G4 to CI
kzscisoft Jun 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
184 changes: 184 additions & 0 deletions .github/workflows/examples.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
name: Simvue Examples

on:
push:
tags:
- 'v*-*-rc*'
branches:
- "296-*"
workflow_dispatch:

permissions:
contents: read

jobs:
GeometryOptimisation:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Download BlueMira
uses: actions/checkout@v4
with:
repository: Fusion-Power-Plant-Framework/bluemira
ref: v1.10.0
path: blue_mira
- uses: conda-incubator/setup-miniconda@v3
with:
activate-environment: bluemira
environment-file: blue_mira/conda/environment.yml
miniforge-version: "latest"
use-only-tar-bz2: false
miniforge-variant: Mambaforge
use-mamba: true
python-version: "3.11"
- name: Install bluemira
shell: bash -l {0}
run: pip install -e blue_mira
- name: Install Simvue
shell: bash -l {0}
run: pip install -e .
- name: Run Example
shell: bash -l {0}
run: python ./examples/GeometryOptimisation/bluemira_simvue_geometry_optimisation.py
Logging:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Simvue
run: python3 -m pip install .
- name: Run Example
run: |
export SIMVUE_URL=${{ secrets.SIMVUE_URL }}
export SIMVUE_TOKEN=${{ secrets.SIMVUE_TOKEN }}
python3 examples/Logging/logging-to-simvue.py --ci
OpenFOAM:
runs-on: ubuntu-latest
container:
image: openfoam/openfoam10-paraview56
options: --user root
steps:
- uses: actions/checkout@v4
- name: Manual Python Install
run: |
add-apt-repository ppa:deadsnakes/ppa -y
apt-get update && \
apt-get install -y git wget curl python3.11-full

- name: Install dependencies
run: |
python3.11 -m ensurepip --upgrade
python3.11 -m pip install ukaea-multiparser
- name: Install Package
run: python3.11 -m pip install .
- name: Run Example
shell: bash
run: |
export SIMVUE_URL=${{ secrets.SIMVUE_URL }}
export SIMVUE_TOKEN=${{ secrets.SIMVUE_TOKEN }}
python3.11 ./examples/OpenFOAM/simvue_openfoam.py /opt/openfoam10/tutorials/incompressible/pimpleFoam/laminar/movingCone/Allrun --ci
Optuna:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Simvue
run: python3 -m pip install .
- name: Install Dependencies
run: |
python3 -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
python3 -m pip install -r examples/Optuna/PyTorch/requirements.txt
- name: Run Example
run: |
export SIMVUE_URL=${{ secrets.SIMVUE_URL }}
export SIMVUE_TOKEN=${{ secrets.SIMVUE_TOKEN }}
python3.11 ./examples/Optuna/PyTorch/simvue_optuna_pytorch.py --ci
PyTorch:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Simvue
run: python3 -m pip install .
- name: Install Dependencies
run: |
python3 -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
python3 -m pip install -r examples/PyTorch/requirements.txt
- name: Run Example
run: |
export SIMVUE_URL=${{ secrets.SIMVUE_URL }}
export SIMVUE_TOKEN=${{ secrets.SIMVUE_TOKEN }}
python3.11 ./examples/PyTorch/main.py --ci
SU2:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Download SU2
run: |
wget https://github.com/su2code/SU2/releases/download/v8.0.1/SU2-v8.0.1-linux64.zip
unzip SU2-v8.0.1-linux64.zip
- name: Install dependencies
run: |
python -m pip install ukaea-multiparser
python -m pip install .
- name: Run Example
run: |
export SIMVUE_URL=${{ secrets.SIMVUE_URL }}
export SIMVUE_TOKEN=${{ secrets.SIMVUE_TOKEN }}
python examples/SU2/SU2.py bin --config tests/example_data/SU2_inv_ONERAM6.cfg --ci
TensorFlow:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Simvue
run: python3 -m pip install .
- name: Install Dependencies
run: python3 -m pip install -r examples/Tensorflow/requirements.txt
- name: Run Example
run: |
export SIMVUE_URL=${{ secrets.SIMVUE_URL }}
export SIMVUE_TOKEN=${{ secrets.SIMVUE_TOKEN }}
python3.11 ./examples/Tensorflow/dynamic_rnn.py --ci

GEANT4:
runs-on: ubuntu-latest
container:
image: artemisbeta/geant4:11.2.1
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install Simvue
run: python3 -m pip install .
- name: Install Dependencies
run: python3 -m pip install -r examples/Geant4/requirements.txt
- name: Build Example
run: |
cmake -DCMAKE_PREFIX_PATH=/usr/local/share/geant4/install/4.11.2/ -Bbuild examples/Geant4/FixedTarget/
cmake --build build
- name: Run Example
run: |
export SIMVUE_URL=${{ secrets.SIMVUE_URL }}
export SIMVUE_TOKEN=${{ secrets.SIMVUE_TOKEN }}
python examples/Geant4/geant4_simvue.py build/MaterialTesting --ci --events 10
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -141,3 +141,6 @@ offline/

# VSCode
.vscode/

# Vagrant
Vagrantfile
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "examples/Geant4/FixedTarget"]
path = examples/Geant4/FixedTarget
url = https://github.com/artemis-beta/Geant4-Example.git
1 change: 1 addition & 0 deletions examples/Geant4/FixedTarget
Submodule FixedTarget added at 4113c4
122 changes: 122 additions & 0 deletions examples/Geant4/geant4_simvue.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
"""
Geant4 Simvue
=============

Example of repeating simulation of a proton fired at a target of beryllium
monitoring the yield of key particles of interest
"""

import multiparser
import multiparser.parsing.file as mp_file_parse
import simvue
import uproot
import multiprocessing
import typing
import click
import pathlib
import os
import tempfile

from particle import Particle


@click.command
@click.argument("g4_binary", type=click.Path(exists=True))
@click.option("--config", type=click.Path(exists=True), default=None)
@click.option("--ci", is_flag=True, default=False)
@click.option("--momentum", type=float, default=10)
@click.option("--events", type=int, default=100)
def geant4_simvue_example(
g4_binary: str, config: typing.Optional[str], ci: bool, momentum: float, events: int
) -> None:
@mp_file_parse.file_parser
def root_file_parser(
file_name: str, *_, **__
) -> tuple[dict[str, typing.Any], dict[str, typing.Any]]:
with uproot.open(file_name) as root_data:
hit_data: dict[str, uproot.TBranch]
if not (hit_data := root_data.get("Hits")):
raise RuntimeError("Expected key 'Hits' in ROOT file")

particles_of_interest = [2212, 211, 11, 22, 2112]

all_particles = hit_data["fID"].array(library="np").tolist()

out_data = {
Particle.from_pdgid(abs(identifier))
.name.replace("+", "plus")
.replace("-", "minus"): [abs(i) for i in all_particles].count(
abs(identifier)
)
for identifier in particles_of_interest
}

return {}, out_data

termination_trigger = multiprocessing.Event()

with simvue.Run() as run:
run.init(
"Geant4_simvue_demo",
folder="/simvue_client_demos",
tags=[
"Geant4",
],
description="Geant4 fixed target scenario",
retention_period="1 hour" if ci else None,
visibility="tenant" if ci else None,
)

kwargs: dict[str, typing.Any] = {}

if config:
kwargs["script"] = config
with tempfile.TemporaryDirectory() as tempd:
with multiparser.FileMonitor(
per_thread_callback=lambda metrics, *_: run.log_metrics(metrics),
exception_callback=run.log_event,
terminate_all_on_fail=False,
plain_logging=True,
flatten_data=True,
termination_trigger=termination_trigger,
) as monitor:
monitor.track(
path_glob_exprs=[f'{pathlib.Path(tempd).joinpath("*")}'],
parser_func=root_file_parser,
static=True,
)
monitor.run()

for i in range(events):
if i % 10 == 0:
click.secho(
f"Running {i+1}/{events} with momentum {momentum} GeV",
bold=True,
fg="cyan",
)
running_simulation = multiprocessing.Event()
run.add_process(
identifier=f"Geant4_simulation_{momentum}GeV_{i}",
executable=g4_binary,
momentum=momentum,
batch=True,
output=pathlib.Path(tempd).joinpath(
f"output_{momentum}GeV_{i+1}.root"
),
completion_trigger=running_simulation
if i == events - 1
else None,
**kwargs,
)

termination_trigger.set()

for file in pathlib.Path().cwd().glob("Geant4*.err"):
os.remove(file)

for file in pathlib.Path().cwd().glob("Geant4*.out"):
os.remove(file)


if __name__ in "__main__":
geant4_simvue_example()
5 changes: 5 additions & 0 deletions examples/Geant4/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
simvue
uproot
ukaea-multiparser
click
particle
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
A quick tutorial on the optimisation of geometry in bluemira
"""

import os
import logging

from bluemira.geometry.optimisation import GeometryOptimisationProblem
Expand Down Expand Up @@ -68,6 +69,9 @@
"optimiser.ftol_abs": ftol_abs,
"optimiser.ftol_rel": ftol_rel,
},
folder="/simvue_client_demos",
visibility="tenant" if os.environ.get("CI") else None,
tags=["bluemira", "simvue_client_examples"],
description="A simple GeometryOptimisationProblem, where we minimise the length of parameterised geometry using gradient-based optimisation algorithm.",
)

Expand Down
27 changes: 17 additions & 10 deletions examples/Logging/logging-to-simvue.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,24 @@
import logging
import click

from simvue import Handler, Run

if __name__ == "__main__":
run = Run()

run.init(tags=["logging"], description="Logging test")
@click.command
@click.option("--ci", is_flag=True, default=False)
def simvue_logger_demo(ci: bool) -> None:
with Run() as run:
run.init(
tags=["logging", "simvue_client_examples"],
folder="/simvue_client_demos",
description="Logging test",
retention_period="1 hour" if ci else None,
visibility="tenant" if ci else None,
)

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
sth = Handler(run)
logger.addHandler(sth)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
sth = Handler(run)
logger.addHandler(sth)

logger.info("This is a Simvue logging test")

run.close()
logger.info("This is a Simvue logging test")
11 changes: 11 additions & 0 deletions examples/OpenFOAM/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
FROM openfoam/openfoam10-paraview56
ARG simvue_file
USER root
RUN add-apt-repository ppa:deadsnakes/ppa -y
RUN apt-get update && \
apt-get install -y python3.11-full
RUN useradd simvue -m
COPY . /simvue_client
RUN python3.11 -m ensurepip --upgrade
RUN python3.11 -m pip install /simvue_client
RUN python3.11 -m pip install -r /simvue_client/examples/OpenFOAM/requirements.txt
1 change: 1 addition & 0 deletions examples/OpenFOAM/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ukaea-multiparser>=1.0.1
Loading
Loading