-
Notifications
You must be signed in to change notification settings - Fork 351
Refactor cache metrics to be homeserver-scoped #18604
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
``` synapse/replication/tcp/streams/_base.py:568: error: Cannot determine type of "_device_list_id_gen" [has-type] synapse/storage/databases/main/event_push_actions.py:256: error: Cannot determine type of "server_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/event_push_actions.py:256: error: Cannot determine type of "server_name" in base class "EventsWorkerStore" [misc] synapse/storage/databases/main/event_push_actions.py:256: error: Cannot determine type of "_instance_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/metrics.py:64: error: Cannot determine type of "server_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/metrics.py:64: error: Cannot determine type of "server_name" in base class "EventsWorkerStore" [misc] synapse/storage/databases/main/metrics.py:64: error: Cannot determine type of "_instance_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/push_rule.py:118: error: Cannot determine type of "_instance_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/push_rule.py:118: error: Cannot determine type of "server_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/push_rule.py:118: error: Cannot determine type of "server_name" in base class "EventsWorkerStore" [misc] synapse/storage/databases/main/account_data.py:60: error: Cannot determine type of "_instance_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/account_data.py:60: error: Cannot determine type of "server_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/account_data.py:60: error: Cannot determine type of "server_name" in base class "EventsWorkerStore" [misc] synapse/storage/databases/main/__init__.py:114: error: Cannot determine type of "server_name" in base class "PresenceStore" [misc] synapse/storage/databases/main/__init__.py:114: error: Cannot determine type of "server_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/__init__.py:114: error: Cannot determine type of "server_name" in base class "ClientIpWorkerStore" [misc] synapse/storage/databases/main/__init__.py:114: error: Cannot determine type of "server_name" in base class "DeviceInboxWorkerStore" [misc] synapse/storage/databases/main/__init__.py:114: error: Cannot determine type of "server_name" in base class "EventsWorkerStore" [misc] synapse/storage/databases/main/__init__.py:114: error: Cannot determine type of "_instance_name" in base class "ReceiptsWorkerStore" [misc] synapse/storage/databases/main/__init__.py:114: error: Cannot determine type of "_instance_name" in base class "DeviceInboxWorkerStore" [misc] synapse/app/generic_worker.py:117: error: Cannot determine type of "_instance_name" in base class "DeviceInboxWorkerStore" [misc] synapse/app/generic_worker.py:117: error: Cannot determine type of "_instance_name" in base class "ReceiptsWorkerStore" [misc] Found 22 errors in 7 files (checked 937 source files) ```
synapse/util/caches/__init__.py
Outdated
@@ -31,6 +31,7 @@ | |||
from prometheus_client.core import Gauge | |||
|
|||
from synapse.config.cache import add_resizable_cache | |||
from synapse.metrics import INSTANCE_LABEL_NAME |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The metrics being refactored to be homeserver scoped are in this file.
The rest of the changes are to support that change and supply the server_name
to the instance
label.
class HasServerName(Protocol): | ||
server_name: str | ||
""" | ||
The homeserver name that this cache is associated with (used to label the metric) | ||
(`hs.hostname`). | ||
""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This pattern is copied from Measure
synapse/synapse/util/metrics.py
Lines 95 to 96 in a2bee2f
class HasClock(Protocol): | |
clock: Clock |
(The Measure
pattern is also updated in #18601)
Conflicts: synapse/http/federation/matrix_federation_agent.py synapse/http/federation/well_known_resolver.py synapse/storage/_base.py synapse/storage/controllers/state.py
Refactor cache metrics to be homeserver-scoped (add
server_name
label to cache metrics).Part of #18592
This can be reviewed commit by commit to skip over some of the bulk refactor but there are some fixes down the line and I'd prefer to keep the history than clean it all up in a rebase.
Testing strategy
See behavior of previous
metrics
listenermetrics
listener in yourhomeserver.yaml
poetry run synapse_homeserver --config-path homeserver.yaml
http://localhost:9323/metrics
synapse_util_caches_cache_size
,synapse_util_caches_cache_hits
,synapse_util_caches_cache_evicted_size
, etc)See behavior of the
http
metrics
resourcemetrics
resource to a new or existinghttp
listeners in yourhomeserver.yaml
poetry run synapse_homeserver --config-path homeserver.yaml
http://localhost:9322/_synapse/metrics
(it's just aGET
request so you can even do in the browser)synapse_util_caches_cache_size
,synapse_util_caches_cache_hits
,synapse_util_caches_cache_evicted_size
, etc): example, example fromdevelop
Dev notes
LruCache
/@cached
,CacheMetric
Todo
@cached
scripts-dev/mypy_synapse_plugin.py
works correctly with cached functions@cached
more but should be fine with how we've done it.Pull Request Checklist
EventStore
toEventWorkerStore
.".code blocks
.