Skip to content

Client get_runs is extremely slow #846

@wk9874

Description

@wk9874

Description of Bug

client.get_runs() is very slow, and appears to do a GET to every run in turn

Steps To Reproduce

client = simvue.Client()

_runs = client.get_runs(
filters=["folder.path == /GS2/test_data"],
metrics=True,
count_limit=5000,
output_format='dataframe'
)

_runs.to_csv("output.csv")

This pulls 4000 runs and takes tens of minutes.

Expected behavior

Should be much faster and not doing GETs to each individual endpoint.

Actual Behaviour

Does a GET to each endpoint:

GET to: https://uk.simvue.io/api/version, Params: None
GET to: https://uk.simvue.io/api/runs, Params: {'attributes': 'null', 'filters': '["folder.path == /GS2/test_data"]', 'return_basic': True, 'return_system': False, 'return_timing': False, 'return_metrics': True, 'return_alerts': False, 'return_metadata': False, 'count': 100, 'start': 0}
GET to: https://uk.simvue.io/api/runs, Params: {'attributes': 'null', 'filters': '["folder.path == /GS2/test_data"]', 'return_basic': True, 'return_system': False, 'return_timing': False, 'return_metrics': True, 'return_alerts': False, 'return_metadata': False, 'count': 100, 'start': 100}
GET to: https://uk.simvue.io/api/runs, Params: {'attributes': 'null', 'filters': '["folder.path == /GS2/test_data"]', 'return_basic': True, 'return_system': False, 'return_timing': False, 'return_metrics': True, 'return_alerts': False, 'return_metadata': False, 'count': 100, 'start': 200}
GET to: https://uk.simvue.io/api/runs, Params: {'attributes': 'null', 'filters': '["folder.path == /GS2/test_data"]', 'return_basic': True, 'return_system': False, 'return_timing': False, 'return_metrics': True, 'return_alerts': False, 'return_metadata': False, 'count': 100, 'start': 300}
GET to: https://uk.simvue.io/api/runs, Params: {'attributes': 'null', 'filters': '["folder.path == /GS2/test_data"]', 'return_basic': True, 'return_system': False, 'return_timing': False, 'return_metrics': True, 'return_alerts': False, 'return_metadata': False, 'count': 100, 'start': 400}
GET to: https://uk.simvue.io/api/runs, Params: {'attributes': 'null', 'filters': '["folder.path == /GS2/test_data"]', 'return_basic': True, 'return_system': False, 'return_timing': False, 'return_metrics': True, 'return_alerts': False, 'return_metadata': False, 'count': 100, 'start': 500}
GET to: https://uk.simvue.io/api/runs/KrsRBi8Lj4Yi7oy3Brz4mi, Params: {}
GET to: https://uk.simvue.io/api/runs/MwYF3kAkeShXFGdtU5sKgk, Params: {}
GET to: https://uk.simvue.io/api/runs/ReFSipaLK7P6tSDKWp3Qav, Params: {}
GET to: https://uk.simvue.io/api/runs/KhjnkNGhC5o22mRWvpbfv4, Params: {}
...

Setup

  • OS: Ubuntu
  • Python version 3.10

Additional context

Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions