Slow loading /users and /assets listings

[Flix6x]

I'm noticing the /users (and the /assets) page can take seconds to load if there are many organisation accounts. Internally, the /api/v3_0/users?account_id=<id> endpoint is getting called as many times as there are organisation accounts that the user has access to, so there is great potential to speed this up. Two options that come to mind:

1. Allow passing account_id multiple times to the users listing endpoint, so we only need to call it once. This would probably require the permission_required_for_context decorator to be enhanced to support a list of accounts.
2. Alternatively, we stop relying on the internal API so much. For example, instead of users += get_users_by_account(account.id, include_inactive) (which uses the InternalAPI), use users += account.users and then filter out the inactive users if needed.

Arguably, option 2 is easier to implement, but option 1 actually improves API and auth capabilities.

[victorgarcia98]

I endorse this idea, we need to improve loading times.

In order to have a more informed process, I would like to create a script to load dummy data and perform a benchmark on this endpoint. We can try loading times with different combinations of account/users/assets. What do you think?

Among the two alternatives, I would favor more (1) as it doesn't break with the status quo.

[nhoening]

We had the idea for a benchmark script here but didn't find the right underlying approach yet.

[victorgarcia98]

Thanks for the reference! #949 apparently benchmarks fetching data from the DB (via calling show beliefs command). I was planning to create something that actually hit the endpoint to see the response time. What do you think?

[nhoening]

I agree the approach needs not to call CLI commands, also due to their initial loading time. I thought about calling code directly (and first establishing an app context), then we can call whatever function we like.

But probably simply using the API might be more straightforward.

[victorgarcia98]

I have create a WIP PR to speed up the loading times of the /users and /assets page. It relies on the fact that the we already propagate authentication to the internal API and we can use the index methods instead of calling get methods on the individual accounts, as suggested by @Flix6x .

Benchmark script

import requests
from bs4 import BeautifulSoup
import time

session = requests.Session()
session.max_redirects = 3

# get CSRF Token
response = session.get("http://localhost:5000")
soup = BeautifulSoup(response.text, "html.parser")
csrf_token = soup.find("input", {"id" : "csrf_token"})["value"]

## Login
response = session.post(
    "http://localhost:5000/login",
    data={
        "email" : "<email>",
        "password" : "<password>",
        "submit" : "Login",
        "next" : "",
        "csrf_token" : csrf_token
    },
)


for _ in range(10):
    t = time.time()
    response = session.get("http://localhost:5000/assets")
    print(time.time() - t)
    soup = BeautifulSoup(response.text, "html.parser")
    print(soup.title.get_text())  # we should see  Asset listing  - FlexMeasures