vastly improve UI testing

[lukaskollmer]

Use Case

goal: we want to have as extensive a testing system as possible

Problem

we currently have decent ui testing coverage if you look only at "lines of code covered as part of running the tests".
however, this is an inherently flawed metric, since the fact that some view body was evaluated as part of running the tests does not mean that the view's state was actually inspected and validated by the tests.
limitations within XCUITest and the iOS accessibility APIs constrain how easy it currently is for us to do ui testing, and how much further we could go wrt the extent to which we validate the app's runtime behaviour.

---

we currently run almost all ui tests in the context of the same user (the very first ui test tests the onboarding and as part of that creates and signs up a user; all subsequent tests then operate in that user's context).
this has been mostly fine so far, but it does mean that:

• the tests currently depend on an implicitly enforced order (the onboarding one always needs to happen at least once before any other tests run)
• there might be data cross-contamination between tests
• we technically don't validate the app in isolation each time, meaning that running only a subset of the tests might result in a different app/user context being present when the tests are run

---

we currently have multiple different approaches for allowing the tests to control the app beyond actions that are exposed as part of the regular ui:

• we have a ton of testing-only launch options that are passed to the app as CLI arguments and parsed at runtime
• we have hidden testing-only debug menus in several places, which are used by the ui tests to instruct the app to set up some specific state or perform some action required for the ui tests to run.
  • this works pretty well, but (imho) is not exactly ideal as it means that the ui existing in the app when it is being tested is not the same as the ui existing in the app when it is deployed and used irl

---

the ui tests currently treat the firebase backend as a black box, having no insight whatsoever into what is going on at any given point in time.
this means that, for example, the tests cannot really verify if data uploaded by the app actually has reached the backend.
it also means that the tests cannot make any modifications/writes of their own to the backend, in order to test how the app would respond to certain events.

Solution

ideas:

• we should add more accessibility identifiers/labels/values throughout the app, in order to give our ui tests more insight into the app's state at any point during the test execution
• we should instead run each test in total isolation, in the context of a temporary user created solely for the purposes of that single test run.
  • the app in fact already contains code for this, and two test already use it; we should expand this to all other tests
• we also should somehow give the tests access to the firebase emulator
  • this would enable us to:
    • be able to directly reach into the backend and validate specific data created by the app
    • be able to mutate data in the user account and see how the app responds to it
      • eg: if we wanted to test the consent renewal flow, we ideally would like to overwrite the user's lastSignedConsentVersion with an outdated value, in order to have the next launch trigger the renewal flow, and for the tests to be able to test and validate that
  • approaches here would be:
    • we cannot simply use the firebase-ios-sdk APIs in the UITests target, as they require an iOS keychain be present (which isn't the case for the targets)
    • we could use the emulator's rest endpoints to interact with it by sending raw queries
      • this would be the least-invasive approach, but also the most limiting (as it would only cover the firebase backend)
    • we could introduce some communication tunnel between the app and the tests
      • would only be present if the app is in a debug build that is being ui-tested
      • would allow the tests to send commands to the app
      • would allow us to get rid of the testing-only debug menus we currently have in some places
      • if this were a bidirectional tunnel, it would also allow the app to inform the tests about specific events (making it easier for the tests to validate that specific things have happened; currently we can only use the "does thing X appear on screen" assertions, which will fail if something only appears on screen for a short amount of time, and also only work for events that actually have some kind of accompanying on-screen presence)
      • multiple possible approaches:
        • a shared file that is being monitored by both the app and the tests, which both can read/write
        • using shared darwin notification center, we should be able to send notifications between the tests and the app, which should be able to contain payloads
        • we could have the tests establish a simple long-lived (for the duration of the tests) tcp connection with the app, which is then used to send back and forth messages for events/commands/etc
        • we could add a mhc:// url scheme to the app, and use XCUIDevice.shared.system.open(_:) to send commands to the app via that. would be unidirectional (tests → app, and we'd neither be able to get responses nor be able to receive events from the app)

Alternatives considered

we could simply keep the current approach, and keep adding launch options and secret debug menus

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct and Contributing Guidelines