Skip to content

[No QA] feat: add native span benchmark tooling - #98144

Open
chrispader wants to merge 20 commits into
Expensify:mainfrom
margelo:codex/benchmark-sentry-spans
Open

[No QA] feat: add native span benchmark tooling#98144
chrispader wants to merge 20 commits into
Expensify:mainfrom
margelo:codex/benchmark-sentry-spans

Conversation

@chrispader

@chrispader chrispader commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Release builds do not provide a local, structured way to inspect selected Sentry spans, so startup measurements have relied on one-off logging and platform-specific commands. That makes results hard to reproduce and compare across builds. This PR adds opt-in span output, repeatable startup benchmarks, and shared Android and iOS device tooling for local release builds.

@roryabraham @mountiny @tgolen

Explanation of Change

  • Developers can list exact span names in the git-ignored root .env with EXPO_PUBLIC_BENCHMARK_SENTRY_SPANS. Successful, non-canceled spans emit a machine-readable [EXPENSIFY_BENCHMARK] JSON event. iOS also writes per-span marker files to the app container because release device logs are not reliable enough for automated collection.
  • scripts/benchmarkAppStartup.ts runs one warm-up followed by repeated process or true-cold launches. It records every configured span by default, supports selecting one span, and can stop when a chosen span ends or after a fixed wait time.
  • Each benchmark prints an Average, P50, P75, P90, P95, P99, Min, and Max table. It always exports both the raw samples and the same summarized results as CSV files, with options to override their paths.
  • The results command reads one or more existing raw sample files, combines their samples by span, prints the recalculated table, and writes a new results.csv file.
  • scripts/lib/benchmarkStatistics.ts owns the shared calculations, sample-file parsing, table formatting, and CSV writing. Live benchmarks and the results command use the same functions.
  • Alternating comparison mode launches two separately installed binaries in A/B order for every cycle. This reduces bias from device temperature and other conditions that change during a benchmark run.
  • scripts/bootstrapForDevice.ts creates side-by-side Android and iOS builds with separate application identifiers. It updates local signing and entitlements, and adds the chosen suffix to the launcher name on both platforms.
  • Shared adb and xcrun devicectl behavior lives in scripts/lib/nativeAppBenchmark.ts, whose workflows are exported for reuse by other local tooling.
  • Span durations use one monotonic clock after translating explicit Sentry start timestamps. This prevents the negative durations caused by subtracting timestamps from different clock domains.
  • contributingGuides/BENCHMARKING.md documents the local release-build and benchmark workflow.

Fixed Issues

$ #98280
PROPOSAL:

Tests

  1. Add at least two exact span names to the git-ignored root .env:
    EXPO_PUBLIC_BENCHMARK_SENTRY_SPANS=ManualAppStartup,ManualAppStartupNetworkRequest
  2. Run nr bootstrap-device android --suffix candidate or nr bootstrap-device ios --suffix candidate, then build and install the release app from Mobile-Expensify.
  3. Verify the installed launcher name contains the suffix, for example Expensify (candidate), and that it can be installed beside the app without that suffix.
  4. Run the benchmark on the connected device:
    nr benchmark-app-startup ios 20 --device "Developer's iPhone" --wait-time 30 --wait-until-span ManualAppStartup
  5. Verify the script prints the resolved platform, device, application identifier, spans, mode, run count, wait settings, raw sample path, and results path before launching the app.
  6. Verify one unmeasured warm-up is followed by 20 measured launches. Confirm the final console table contains one row per configured span, the raw sample CSV contains each run, and the matching *-results.csv file contains the displayed aggregate values.
  7. Recreate a results table from the raw samples:
    nr benchmark-app-startup results --input-files .benchmarks/sample-a.csv,.benchmarks/sample-b.csv --results-output .benchmarks/results.csv
    Verify the command combines samples by span, prints their statistics, and writes the same values to .benchmarks/results.csv. Repeat without --results-output and verify it writes .benchmarks/results.csv by default.
  8. Run the results command with an invalid sample header and verify it exits with a clear error. Repeat a live benchmark with a --span value absent from the allowlist and verify it exits without launching the app.
  9. Install two bootstrapped builds with different identifiers. Run alternating mode with --app-id-a and --app-id-b, then verify the terminal alternates A and B for every cycle and produces separate raw and results CSV files for both binaries.
  10. Verify no unexpected errors appear in the JS or device console apart from the expected [EXPENSIFY_BENCHMARK] warning output.
  • Verify that no unexpected errors appear in the JS/device console apart from the expected [EXPENSIFY_BENCHMARK] warning output.

Offline tests

Offline testing was not performed. The change adds local release-build logging and device automation without changing network behavior. Running a benchmark offline would measure the existing offline behavior of the selected span, not behavior introduced by this PR.

QA Steps

This PR cannot be tested in a standard staging or production build. The span allowlist, app identifiers, and signing changes are supplied while producing a local release build, and QA does not have access to that build workflow or the required command-line tooling.

  • Verify that no errors appear in the JS console

PR Author Checklist

  • I linked the correct issue in the ### Fixed Issues section above
  • I wrote clear testing steps that cover the changes made in this PR
    • I added steps for local testing in the Tests section
    • I added steps for the expected offline behavior in the Offline steps section
    • I added steps for Staging and/or Production testing in the QA steps section
    • I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
    • I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
    • I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).
  • I included screenshots or videos for tests on all platforms
  • I ran the tests on all platforms & verified they passed on:
    • Android: Native
    • Android: mWeb Chrome
    • iOS: Native
    • iOS: mWeb Safari
    • MacOS: Chrome / Safari
  • I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
  • I followed proper code patterns (see Reviewing the code)
    • I verified that comments were added to code that is not self explanatory
    • I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
    • I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
  • If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
  • I followed the guidelines as stated in the Review Guidelines
  • I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
  • If a new CSS style is added I verified that:
    • A similar style doesn't already exist
    • The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))
  • If new assets were added or existing ones were modified, I verified that:
    • The assets are optimized and compressed (for SVG files, run npm run compress-svg)
    • The assets load correctly across all supported platforms.
  • If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
  • If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
  • If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
  • If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
  • If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
    • I verified that all the inputs inside a form are aligned with each other.
    • I added Design label and/or tagged @Expensify/design so the design team can review the changes.
  • I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
  • If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

No screenshots or videos are included because the benchmark has no in-app UI. The iOS launcher-name suffix was verified locally as part of the developer setup flow.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ This PR is possibly changing native code and/or updating libraries, it may cause problems with HybridApp. Please check if any patch updates are required in the HybridApp repo and run an AdHoc build to verify that HybridApp will not break. Ask Contributor Plus for help if you are not sure how to handle this. ⚠️

@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ Changes either increased or maintained existing code coverage, great job!

Files with missing lines Coverage Δ
src/libs/telemetry/activeSpans.ts 98.38% <100.00%> (+1.77%) ⬆️
src/libs/telemetry/logBenchmarkSpanEnd.ts 88.88% <88.88%> (ø)
src/libs/telemetry/writeBenchmarkLog/index.ios.ts 77.77% <77.77%> (ø)
src/libs/telemetry/writeBenchmarkLog/index.ts 0.00% <0.00%> (ø)
... and 197 files with indirect coverage changes

@chrispader
chrispader force-pushed the codex/benchmark-sentry-spans branch from 36666c0 to 24d3bc5 Compare August 12, 2026 15:11
@quinthar quinthar removed the #quality label Aug 13, 2026
@chrispader chrispader changed the title feat: add native span benchmark tooling [No QA] feat: add native span benchmark tooling Aug 24, 2026
@chrispader
chrispader marked this pull request as ready for review August 24, 2026 19:41
@chrispader
chrispader requested a review from a team as a code owner August 24, 2026 19:41
@chrispader
chrispader requested a review from a team as a code owner August 24, 2026 19:41
@melvin-bot
melvin-bot Bot requested review from sobitneupane and removed request for a team August 24, 2026 19:41
@melvin-bot

melvin-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown

@sobitneupane Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

@melvin-bot
melvin-bot Bot requested review from heyjennahay and removed request for a team August 24, 2026 19:41
Comment thread scripts/benchmarkAppStartup.ts
Comment thread scripts/bootstrapForDevice.ts
@chrispader

Copy link
Copy Markdown
Contributor Author

@sobitneupane this PR is now ready for review! 🙌🏼

@sobitneupane

Copy link
Copy Markdown
Contributor

@chrispader Could you please reassign the issue. I won't be able to review the PR before the weekend.

@chrispader

Copy link
Copy Markdown
Contributor Author

@heyjennahay could we please re-assign to another C+ to work on this issue? 🙌🏼

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants