Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

MadLittleMods · 2025-08-29T00:34:40Z

Implements MSC3871: Gappy timelines

Add the gaps field to the /messages endpoint

Complement tests: matrix-org/complement#801

Testing strategy

Clone synapse
Checkout the correct branch: git checkout madlittlemods/msc3871-gappy-timeline (this PR)
Install Synapse's dependencies: poetry install
Clone complement next to synapse as a sibling
Checkout the correct branch: git checkout madlittlemods/msc3871-gappy-timeline (Tests for MSC3871: Gappy timelines matrix-org/complement#801)

In order to manually test with a real Matrix client:

In complement, uncomment the sleep call in TestRoomMessagesGaps
Start the Complement tests: COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestRoomMessagesGaps
Find which port hs3 is running on: docker ps -f name=complement_ (ex. 0.0.0.0:33413->8008/tcp)
Configure your Matrix client with that homeserver URL: http://localhost:33413
Register a new user
Join the room from the test (in the log output)
Back paginate in the room: /messages?dir=b&backfill=false and notice the gaps in the room
To fill in gaps: /messages?dir=b&backfill=true&from=<prev_pagination_token>

Reference: Complement docs on how to hook up a Matrix client

Dev notes

Document how to hook up Element to the resultant homeservers from Complement: matrix-org/complement#164

matrix-org/complement -> ONBOARDING.md -> How do I hook up a Matrix client like Element to the homeservers spun up by Complement after a test runs?

Todo

Put the behavior behind a experimental feature flag
Add test_gaps_going_forwards
Add some Complement tests
Separate out the /messages?backfill=true/false changes (simplified version of MSC4282)

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

See [MSC: Gappy timelines](matrix-org/matrix-spec-proposals#3871)

To try out the flow: - **Default to fast responses with gaps**: As a default, we can always respond quickly and indicate gaps ([MSC3871] (matrix-org/matrix-spec-proposals#3871)) for clients to paginate at their leisure. - **Fast back-pagination**: Clients back-paginate with `/messages?dir=b&backfill=false`, and Synapse skips backfilling entirely, returning only local history with gaps as necessary. - **Explicit gap filling**: To fill in gaps, clients use `/messages?dir=b&backfill=true` which works just like today to do a best effort backfill. This allows the client to back-paginate the history we already have without delay. And can fill in the gaps as they see fit. This is basically a simplified version of [MSC4282] (matrix-org/matrix-spec-proposals#4282).

MadLittleMods · 2025-08-29T01:11:14Z

synapse/types/__init__.py

-    def get_stream_pos_for_instance(self, instance_name: str) -> int:
-        """Get the stream position that the given writer was at at this token.
+    def is_before_or_eq(self, other_token: Self) -> bool:
+        is_before_or_eq_stream_ordering = super().is_before_or_eq(other_token)
+        if not is_before_or_eq_stream_ordering:
+            return False

-        This only makes sense for "live" tokens that may have a vector clock
-        component, and so asserts that this is a "live" token.
-        """
-        assert self.topological is None


Removed get_stream_pos_for_instance because there is no need to have this specialized version that only allows stream-based tokens. We will fallback to the super() version which is the same but without the topological restriction.

While only "live" tokens may have a vector clock component, historical topological tokens still include a stream position (t426-2633508) and it makes sense for this to still work.

synapse/synapse/types/__init__.py

Lines 697 to 703 in 68068de

Historic tokens start with a "t" followed by the `depth`

(`topological_ordering` in the event graph) of the event that comes before

the position of the token, followed by "-", followed by the

`stream_ordering` of the event that comes before the position of the token.

An example token is:

t426-2633508

…mate comparisons

MadLittleMods · 2025-09-05T15:25:43Z

synapse/storage/databases/main/event_federation.py

+        It's best to pad the `current_depth` by the number of messages you plan to
+        backfill from these points.


This is a good useful change that we could ship outside of this PR.

Noting in case this PR goes stale

MadLittleMods · 2025-09-05T15:26:08Z

synapse/storage/databases/main/events_worker.py

+# from synapse.storage.databases.main.stream import (
+#     generate_next_token,
+# )


Suggested change

# from synapse.storage.databases.main.stream import (

# generate_next_token,

# )

MadLittleMods · 2025-09-05T15:27:19Z

synapse/storage/databases/main/events_worker.py

+            ignore_gap_after_latest: Whether the gap after the latest events (forward
+                extremeties) in the room should be considered as an actual gap.


Per matrix-org/matrix-spec-proposals#3871 (comment), we should revert the ignore_gap_after_latest change.

The default of "omit the gap after the latest messages in the room" is the correct choice

MadLittleMods added 2 commits August 28, 2025 16:58

First stab at gaps in /messages

a14808e

See [MSC: Gappy timelines](matrix-org/matrix-spec-proposals#3871)

Add test_gaps_going_backwards

50d8337

MadLittleMods added the A-Messages-Endpoint label Aug 29, 2025

MadLittleMods added 2 commits August 28, 2025 19:38

Add changelog

d353cfc

MadLittleMods commented Aug 29, 2025

View reviewed changes

Picking backfill points: current_depth should be padded for approxi…

e835134

…mate comparisons

MadLittleMods mentioned this pull request Sep 2, 2025

Tests for MSC3871: Gappy timelines matrix-org/complement#801

Draft

2 tasks

MadLittleMods commented Sep 5, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

Uh oh!

MadLittleMods commented Aug 29, 2025 •

edited

Loading

Uh oh!

MadLittleMods Aug 29, 2025 •

edited

Loading

Uh oh!

MadLittleMods Sep 5, 2025

Uh oh!

MadLittleMods Sep 5, 2025

Uh oh!

MadLittleMods Sep 5, 2025

Uh oh!

Uh oh!

	Historic tokens start with a "t" followed by the `depth`
	(`topological_ordering` in the event graph) of the event that comes before
	the position of the token, followed by "-", followed by the
	`stream_ordering` of the event that comes before the position of the token.
	An example token is:

	t426-2633508

		It's best to pad the `current_depth` by the number of messages you plan to
		backfill from these points.

	# from synapse.storage.databases.main.stream import (
	# generate_next_token,
	# )

		ignore_gap_after_latest: Whether the gap after the latest events (forward
		extremeties) in the room should be considered as an actual gap.

Implement MSC3871: Gappy timelines - indicate gaps in /messages #18873

Are you sure you want to change the base?

Implement MSC3871: Gappy timelines - indicate gaps in /messages #18873

Uh oh!

Conversation

MadLittleMods commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing strategy

Dev notes

Todo

Pull Request Checklist

Uh oh!

MadLittleMods Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

Implement MSC3871: Gappy timelines - indicate `gaps` in `/messages` #18873

MadLittleMods commented Aug 29, 2025 •

edited

Loading

MadLittleMods Aug 29, 2025 •

edited

Loading