You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Release builds do not provide a local, structured way to inspect selected Sentry spans, so startup measurements have relied on one-off logging and platform-specific commands. That makes results hard to reproduce and compare across builds. This PR adds opt-in span output, repeatable startup benchmarks, and shared Android and iOS device tooling for local release builds.
Developers can list exact span names in the git-ignored root .env with EXPO_PUBLIC_BENCHMARK_SENTRY_SPANS. Successful, non-canceled spans emit a machine-readable [EXPENSIFY_BENCHMARK] JSON event. iOS also writes per-span marker files to the app container because release device logs are not reliable enough for automated collection.
scripts/benchmarkAppStartup.ts runs one warm-up followed by repeated process or true-cold launches. It records every configured span by default, supports selecting one span, and can stop when a chosen span ends or after a fixed wait time.
Each benchmark prints an Average, P50, P75, P90, P95, P99, Min, and Max table. It always exports both the raw samples and the same summarized results as CSV files, with options to override their paths.
The results command reads one or more existing raw sample files, combines their samples by span, prints the recalculated table, and writes a new results.csv file.
scripts/lib/benchmarkStatistics.ts owns the shared calculations, sample-file parsing, table formatting, and CSV writing. Live benchmarks and the results command use the same functions.
Alternating comparison mode launches two separately installed binaries in A/B order for every cycle. This reduces bias from device temperature and other conditions that change during a benchmark run.
scripts/bootstrapForDevice.ts creates side-by-side Android and iOS builds with separate application identifiers. It updates local signing and entitlements, and adds the chosen suffix to the launcher name on both platforms.
Shared adb and xcrun devicectl behavior lives in scripts/lib/nativeAppBenchmark.ts, whose workflows are exported for reuse by other local tooling.
Span durations use one monotonic clock after translating explicit Sentry start timestamps. This prevents the negative durations caused by subtracting timestamps from different clock domains.
contributingGuides/BENCHMARKING.md documents the local release-build and benchmark workflow.
Run nr bootstrap-device android --suffix candidate or nr bootstrap-device ios --suffix candidate, then build and install the release app from Mobile-Expensify.
Verify the installed launcher name contains the suffix, for example Expensify (candidate), and that it can be installed beside the app without that suffix.
Verify the script prints the resolved platform, device, application identifier, spans, mode, run count, wait settings, raw sample path, and results path before launching the app.
Verify one unmeasured warm-up is followed by 20 measured launches. Confirm the final console table contains one row per configured span, the raw sample CSV contains each run, and the matching *-results.csv file contains the displayed aggregate values.
Recreate a results table from the raw samples:
nr benchmark-app-startup results --input-files .benchmarks/sample-a.csv,.benchmarks/sample-b.csv --results-output .benchmarks/results.csv
Verify the command combines samples by span, prints their statistics, and writes the same values to .benchmarks/results.csv. Repeat without --results-output and verify it writes .benchmarks/results.csv by default.
Run the results command with an invalid sample header and verify it exits with a clear error. Repeat a live benchmark with a --span value absent from the allowlist and verify it exits without launching the app.
Install two bootstrapped builds with different identifiers. Run alternating mode with --app-id-a and --app-id-b, then verify the terminal alternates A and B for every cycle and produces separate raw and results CSV files for both binaries.
Verify no unexpected errors appear in the JS or device console apart from the expected [EXPENSIFY_BENCHMARK] warning output.
Verify that no unexpected errors appear in the JS/device console apart from the expected [EXPENSIFY_BENCHMARK] warning output.
Offline tests
Offline testing was not performed. The change adds local release-build logging and device automation without changing network behavior. Running a benchmark offline would measure the existing offline behavior of the selected span, not behavior introduced by this PR.
QA Steps
This PR cannot be tested in a standard staging or production build. The span allowlist, app identifiers, and signing changes are supplied while producing a local release build, and QA does not have access to that build workflow or the required command-line tooling.
Verify that no errors appear in the JS console
PR Author Checklist
I linked the correct issue in the ### Fixed Issues section above
I wrote clear testing steps that cover the changes made in this PR
I added steps for local testing in the Tests section
I added steps for the expected offline behavior in the Offline steps section
I added steps for Staging and/or Production testing in the QA steps section
I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).
I included screenshots or videos for tests on all platforms
I ran the tests on all platforms & verified they passed on:
Android: Native
Android: mWeb Chrome
iOS: Native
iOS: mWeb Safari
MacOS: Chrome / Safari
I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
I verified that comments were added to code that is not self explanatory
I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
If a new CSS style is added I verified that:
A similar style doesn't already exist
The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))
If new assets were added or existing ones were modified, I verified that:
The assets are optimized and compressed (for SVG files, run npm run compress-svg)
The assets load correctly across all supported platforms.
If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
I verified that all the inputs inside a form are aligned with each other.
I added Design label and/or tagged @Expensify/design so the design team can review the changes.
I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
Screenshots/Videos
No screenshots or videos are included because the benchmark has no in-app UI. The iOS launcher-name suffix was verified locally as part of the developer setup flow.
⚠️ This PR is possibly changing native code and/or updating libraries, it may cause problems with HybridApp. Please check if any patch updates are required in the HybridApp repo and run an AdHoc build to verify that HybridApp will not break. Ask Contributor Plus for help if you are not sure how to handle this. ⚠️
@sobitneupane Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]
@heyjennahay could we please re-assign to another C+ to work on this issue? 🙌🏼
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release builds do not provide a local, structured way to inspect selected Sentry spans, so startup measurements have relied on one-off logging and platform-specific commands. That makes results hard to reproduce and compare across builds. This PR adds opt-in span output, repeatable startup benchmarks, and shared Android and iOS device tooling for local release builds.
@roryabraham @mountiny @tgolen
Explanation of Change
.envwithEXPO_PUBLIC_BENCHMARK_SENTRY_SPANS. Successful, non-canceled spans emit a machine-readable[EXPENSIFY_BENCHMARK]JSON event. iOS also writes per-span marker files to the app container because release device logs are not reliable enough for automated collection.scripts/benchmarkAppStartup.tsruns one warm-up followed by repeated process or true-cold launches. It records every configured span by default, supports selecting one span, and can stop when a chosen span ends or after a fixed wait time.resultscommand reads one or more existing raw sample files, combines their samples by span, prints the recalculated table, and writes a newresults.csvfile.scripts/lib/benchmarkStatistics.tsowns the shared calculations, sample-file parsing, table formatting, and CSV writing. Live benchmarks and theresultscommand use the same functions.scripts/bootstrapForDevice.tscreates side-by-side Android and iOS builds with separate application identifiers. It updates local signing and entitlements, and adds the chosen suffix to the launcher name on both platforms.adbandxcrun devicectlbehavior lives inscripts/lib/nativeAppBenchmark.ts, whose workflows are exported for reuse by other local tooling.contributingGuides/BENCHMARKING.mddocuments the local release-build and benchmark workflow.Fixed Issues
$ #98280
PROPOSAL:
Tests
.env:nr bootstrap-device android --suffix candidateornr bootstrap-device ios --suffix candidate, then build and install the release app fromMobile-Expensify.Expensify (candidate), and that it can be installed beside the app without that suffix.nr benchmark-app-startup ios 20 --device "Developer's iPhone" --wait-time 30 --wait-until-span ManualAppStartup*-results.csvfile contains the displayed aggregate values..benchmarks/results.csv. Repeat without--results-outputand verify it writes.benchmarks/results.csvby default.resultscommand with an invalid sample header and verify it exits with a clear error. Repeat a live benchmark with a--spanvalue absent from the allowlist and verify it exits without launching the app.--app-id-aand--app-id-b, then verify the terminal alternates A and B for every cycle and produces separate raw and results CSV files for both binaries.[EXPENSIFY_BENCHMARK]warning output.[EXPENSIFY_BENCHMARK]warning output.Offline tests
Offline testing was not performed. The change adds local release-build logging and device automation without changing network behavior. Running a benchmark offline would measure the existing offline behavior of the selected span, not behavior introduced by this PR.
QA Steps
This PR cannot be tested in a standard staging or production build. The span allowlist, app identifiers, and signing changes are supplied while producing a local release build, and QA does not have access to that build workflow or the required command-line tooling.
PR Author Checklist
### Fixed Issuessection aboveTestssectionOffline stepssectionQA stepssectionAvatar, I verified the components usingAvatarare working as expected)StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))npm run compress-svg)Avataris modified, I verified thatAvataris working as expected in all cases)Designlabel and/or tagged@Expensify/designso the design team can review the changes.mainbranch was merged into this PR after a review, I tested again and verified the outcome was still expected according to theTeststeps.Screenshots/Videos
No screenshots or videos are included because the benchmark has no in-app UI. The iOS launcher-name suffix was verified locally as part of the developer setup flow.