Skip to content

Ecoclient Simvue Python API Extension #752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
Mar 28, 2025
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
1ab36df
Remove CodeCarbon from Python API
kzscisoft Mar 6, 2025
ddabec6
Fix emission metric sending
kzscisoft Mar 7, 2025
4978446
Fix wrong variable name
kzscisoft Mar 7, 2025
3916018
Merge branch 'dev' into feature/ecoclient
kzscisoft Mar 13, 2025
333d374
Fix co2 monitor default
kzscisoft Mar 17, 2025
223d4c3
Remove extra parameter from _start
kzscisoft Mar 17, 2025
bd66480
Added path field validation
kzscisoft Mar 19, 2025
8f79810
Refactor of heartbeat code
kzscisoft Mar 19, 2025
bd10c44
Started simplifying ecoclient
kzscisoft Mar 19, 2025
3aab367
Fix emissions estimates using resource metrics CPU info
kzscisoft Mar 20, 2025
2f104e8
Add GPU metrics
kzscisoft Mar 20, 2025
08dc7a0
Format code
kzscisoft Mar 20, 2025
933ba92
Fix loop frequency to be 1s
kzscisoft Mar 20, 2025
f885caf
Fix bad refresh rate for CO2 signal and added CO2 signal mocking
kzscisoft Mar 24, 2025
5f5fcfa
Add CO2 intensity refresh to sender
kzscisoft Mar 25, 2025
6d1917d
Merge branch 'dev' into feature/ecoclient
kzscisoft Mar 25, 2025
3e08099
Remove unneeded extra thread in sender
kzscisoft Mar 25, 2025
eacbe45
Resolve stability of log_metrics test
kzscisoft Mar 25, 2025
af4d0e3
[skip ci] Updated changelog
kzscisoft Mar 25, 2025
3acce87
Merge branch 'dev' into feature/ecoclient
kzscisoft Mar 25, 2025
bae9ade
Re-write abort python on alert test
kzscisoft Mar 25, 2025
30da207
Merge branch 'dev' into feature/ecoclient
kzscisoft Mar 25, 2025
6538fc6
Fix monitor tests
kzscisoft Mar 25, 2025
4de2f5f
Simplified and fixed the emissions run tests
kzscisoft Mar 26, 2025
01f8c0b
Fix bad code inclusion of 'attach_process' for emissions monitor
kzscisoft Mar 27, 2025
b2811f0
Fix 0 GPUs bug
kzscisoft Mar 27, 2025
462e7be
Renamed resource metrics interval to system metrics interval
kzscisoft Mar 28, 2025
ebef998
Add Python3.10 support for CO2 timestamp parsing
kzscisoft Mar 28, 2025
444805d
Tidy error response from CO2 signal API
kzscisoft Mar 28, 2025
1626f17
Reactivate sorting tests
kzscisoft Mar 28, 2025
9c5317e
Fix Co2 client logic, do not get from CO2 signal if value provided
kzscisoft Mar 28, 2025
ff41785
Do not refresh if offline
kzscisoft Mar 28, 2025
b0545e5
Added local refresh and handling of complete offline running for CO2
kzscisoft Mar 28, 2025
74f00f2
Added Number of CPU cores to config
kzscisoft Mar 28, 2025
37d900b
Added intensity units to log message
kzscisoft Mar 28, 2025
2437e37
Fix refresh on first run offline
kzscisoft Mar 28, 2025
01b5ff8
Correct step 0 emissions measure
kzscisoft Mar 28, 2025
bbe4b4f
Test parallelization
kzscisoft Mar 28, 2025
d352d7e
Mock location info for international test running
kzscisoft Mar 28, 2025
4099b79
Fix handling of no local emission data
kzscisoft Mar 28, 2025
ff59423
Run non offline/online tests in CI
kzscisoft Mar 28, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
877 changes: 127 additions & 750 deletions poetry.lock

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@ dependencies = [
"gitpython (>=3.1.44,<4.0.0)",
"humanfriendly (>=10.0,<11.0)",
"randomname (>=0.2.1,<0.3.0)",
"codecarbon (>=2.8.3,<3.0.0)",
"numpy (>=2.0.0,<3.0.0)",
"flatdict (>=4.0.1,<5.0.0)",
"semver (>=3.0.4,<4.0.0)",
Expand All @@ -54,6 +53,7 @@ dependencies = [
"tenacity (>=9.0.0,<10.0.0)",
"typing-extensions (>=4.12.2,<5.0.0) ; python_version < \"3.11\"",
"deepmerge (>=2.0,<3.0)",
"geocoder (>=1.38.1,<2.0.0)",
]

[project.urls]
Expand Down
2 changes: 1 addition & 1 deletion simvue/api/objects/folder.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
import pathlib
import typing
import datetime
import json

from codecarbon.output_methods.emissions_data import json
import pydantic

from simvue.exception import ObjectNotFoundError
Expand Down
2 changes: 1 addition & 1 deletion simvue/api/request.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@
import copy
import json as json_module
import typing
import logging
import http

from codecarbon.external.logger import logging
import requests
from tenacity import (
retry,
Expand Down
2 changes: 0 additions & 2 deletions simvue/config/parameters.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,6 @@ def check_token(cls, v: typing.Any) -> str | None:

class OfflineSpecifications(pydantic.BaseModel):
cache: pathlib.Path | None = None
country_iso_code: str | None = None


class MetricsSpecifications(pydantic.BaseModel):
resources_metrics_interval: pydantic.PositiveInt | None = -1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would call this 'system_metrics_interval' as it now applies to both resources and emissions metrics

Expand Down
8 changes: 5 additions & 3 deletions simvue/config/user.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
from simvue.version import __version__
from simvue.api.request import get as sv_get
from simvue.api.url import URL
from simvue.eco.config import EcoConfig

logger = logging.getLogger(__name__)

Expand All @@ -42,14 +43,15 @@

class SimvueConfiguration(pydantic.BaseModel):
# Hide values as they contain token and URL
model_config = pydantic.ConfigDict(hide_input_in_errors=True)
model_config = pydantic.ConfigDict(hide_input_in_errors=True, revalidate_instances="always")
client: ClientGeneralOptions = ClientGeneralOptions()
server: ServerSpecifications = pydantic.Field(
..., description="Specifications for Simvue server"
)
run: DefaultRunSpecifications = DefaultRunSpecifications()
offline: OfflineSpecifications = OfflineSpecifications()
metrics: MetricsSpecifications = MetricsSpecifications()
eco: EcoConfig = EcoConfig()

@classmethod
def _load_pyproject_configs(cls) -> dict | None:
Expand Down Expand Up @@ -135,14 +137,14 @@ def write(self, out_directory: pydantic.DirectoryPath) -> None:

@pydantic.model_validator(mode="after")
@classmethod
def check_valid_server(cls, values: "SimvueConfiguration") -> bool:
def check_valid_server(cls, values: "SimvueConfiguration") -> "SimvueConfiguration":
if os.environ.get("SIMVUE_NO_SERVER_CHECK"):
return values

cls._check_server(values.server.token, values.server.url, values.run.mode)

return values

@classmethod
@sv_util.prettify_pydantic
def fetch(
Expand Down
132 changes: 0 additions & 132 deletions simvue/eco.py

This file was deleted.

14 changes: 14 additions & 0 deletions simvue/eco/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
"""
Simvue Eco
==========

Contains functionality for green IT, monitoring emissions etc.
NOTE: The metrics calculated by these methods should be used for relative
comparisons only. Any values returned should not be taken as absolute.

"""
__date__ = "2025-03-06"

from .emissions_monitor import CO2Monitor as CO2Monitor

__all__ = ["CO2Monitor"]
141 changes: 141 additions & 0 deletions simvue/eco/api_client.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
"""
CO2 Signal API Client
=====================

Provides inteface to the CO2 Signal API,
which provides real-time data on the carbon intensity of
electricity generation in different countries.
"""

__date__ = "2025-02-27"

import requests
import pydantic
import functools
import http
import logging
import datetime
import geocoder
import geocoder.location
import typing


class CO2SignalData(pydantic.BaseModel):
datetime: datetime.datetime
carbon_intensity: float
fossil_fuel_percentage: float


class CO2SignalResponse(pydantic.BaseModel):
disclaimer: str
country_code: str
status: str
data: CO2SignalData
carbon_intensity_units: str

@classmethod
def from_json_response(cls, json_response: dict) -> "CO2SignalResponse":
_data: dict[str, typing.Any] = json_response["data"]
_co2_signal_data = CO2SignalData(
datetime=datetime.datetime.fromisoformat(_data["datetime"]),
carbon_intensity=_data["carbonIntensity"],
fossil_fuel_percentage=_data["fossilFuelPercentage"],
)
return cls(
disclaimer=json_response["_disclaimer"],
country_code=json_response["countryCode"],
status=json_response["status"],
data=_co2_signal_data,
carbon_intensity_units=json_response["units"]["carbonIntensity"],
)


@functools.lru_cache()
def _call_geocoder_query() -> typing.Any:
"""Call GeoCoder API for IP location

Cached so this API is only called once per session as required.
"""
return geocoder.ip("me")


class APIClient(pydantic.BaseModel):
"""
CO2 Signal API Client

Provides an interface to the Electricity Maps API.
"""

co2_api_endpoint: pydantic.HttpUrl = pydantic.HttpUrl(
"https://api.co2signal.com/v1/latest"
)
co2_api_token: pydantic.SecretStr | None = None
timeout: pydantic.PositiveInt = 10

def __init__(self, *args, **kwargs) -> None:
"""Initialise the CO2 Signal API client.

Parameters
----------
co2_api_endpoint : str
endpoint for CO2 signal API
co2_api_token: str
RECOMMENDED. The API token for the CO2 Signal API, default is None.
timeout : int
timeout for API
"""
super().__init__(*args, **kwargs)
self._logger = logging.getLogger(self.__class__.__name__)

if not self.co2_api_token:
self._logger.warning(
"⚠️ No API token provided for CO2 Signal, it is recommended "
)

self._get_user_location_info()

def _get_user_location_info(self) -> None:
"""Retrieve location information for the current user."""
self._logger.info("📍 Determining current user location.")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably add a disclaimer somewhere that we are doing this... Also do we need to consider anything in terms of data protection for collecting this? I guess not because its staying local to their machine

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh actually no the exact lat and long get sent to the geocoder API - I think definitely need to consider whether there are GDPR etc issues there

_current_user_loc_data: geocoder.location.BBox = _call_geocoder_query()
self._latitude: float
self._longitude: float
self._latitude, self._longitude = _current_user_loc_data.latlng
self._two_letter_country_code: str = _current_user_loc_data.country # type: ignore

def get(self) -> CO2SignalResponse:
"""Get the current data"""
_params: dict[str, float | str] = {
"lat": self._latitude,
"lon": self._longitude,
"countryCode": self._two_letter_country_code,
}

if self.co2_api_token:
_params["auth-token"] = self.co2_api_token.get_secret_value()

self._logger.debug(f"🍃 Retrieving carbon intensity data for: {_params}")
_response = requests.get(f"{self.co2_api_endpoint}", params=_params)

if not _response.status_code == http.HTTPStatus.OK:
raise RuntimeError(
"Failed to retrieve current CO2 signal data for"
f" country '{self._two_letter_country_code}': {_response.text}"
)

return CO2SignalResponse.from_json_response(_response.json())

@property
def country_code(self) -> str:
"""Returns the country code"""
return self._two_letter_country_code

@property
def latitude(self) -> float:
"""Returns current latitude"""
return self._latitude

@property
def longitude(self) -> float:
"""Returns current longitude"""
return self._longitude
Loading
Loading