Keep track of the known topics in store #1951

sm-sayedi · 2025-10-28T21:24:22Z

#1508 but rebased on top of main with the review of @gnprice addressed.

(Thanks @PIG208 for all of your previous work in #1508)

Fixes: #1499

chrisbobbe

Thanks! Glad to be maintaining a data structure for this. 🙂

Comments below, from reading the implementation in the first commit (I haven't yet read the tests):

d6b2242 channel: Keep track of channel topics, and keep up-to-date with events

chrisbobbe · 2025-11-04T03:57:05Z

lib/model/channel.dart

 import 'store.dart';
 import 'user.dart';

+final _apiGetChannelTopics = getStreamTopics;  // similar to _apiSendMessage in lib/model/message.dart


nit: too-long line; put comment on line above

chrisbobbe · 2025-11-04T04:08:11Z

lib/model/channel.dart

+  /// can be retrieved with [getChannelTopics].
+  Future<void> fetchTopics(int channelId);
+
+  /// Pairs of the known topics and its latest message ID, in the given channel.


Suggested change

/// Pairs of the known topics and its latest message ID, in the given channel.

/// The topics in the given channel, along with their latest message ID.

chrisbobbe · 2025-11-04T04:09:01Z

lib/model/channel.dart

+
+  /// Pairs of the known topics and its latest message ID, in the given channel.
+  ///
+  /// Returns null if the data has never been fetched yet.


nit:

Suggested change

/// Returns null if the data has never been fetched yet.

/// Returns null if the data has not been fetched yet.

chrisbobbe · 2025-11-04T04:09:45Z

lib/model/channel.dart

+  /// The result is guaranteed to be sorted by [GetStreamTopicsEntry.maxId]
+  /// descending, and the topics are guaranteed to be distinct.


nit, omit needless words:

Suggested change

/// The result is guaranteed to be sorted by [GetStreamTopicsEntry.maxId]

/// descending, and the topics are guaranteed to be distinct.

/// The result is sorted by [GetStreamTopicsEntry.maxId] descending,

/// and the topics are distinct.

chrisbobbe · 2025-11-04T04:34:54Z

lib/model/channel.dart

+  /// In some cases, the same maxId affected by message moves can be present in
+  /// multiple [GetStreamTopicsEntry] entries. For this reason, the caller
+  /// should not rely on [getChannelTopics] to determine which topic the message
+  /// is in. Instead, refer to [PerAccountStore.messages].
+  /// See [handleUpdateMessageEvent] on how this could happen.


It seems worth highlighting in general that maxId might be…incorrect, basically. (Not just that two topics might have the same maxId.) Maybe something like:

(Also, isn't message deletion another reason maxId could be wrong?)

Suggested change

/// In some cases, the same maxId affected by message moves can be present in

/// multiple [GetStreamTopicsEntry] entries. For this reason, the caller

/// should not rely on [getChannelTopics] to determine which topic the message

/// is in. Instead, refer to [PerAccountStore.messages].

/// See [handleUpdateMessageEvent] on how this could happen.

/// Occasionally, [GetStreamTopicsEntry.maxId] will refer to a message

/// that doesn't exist or is no longer in the topic.

/// This happens when a topic's latest message is moved or deleted

/// and we don't have enough information

/// to replace [GetStreamTopicsEntry.maxId] accurately.

/// (We don't keep a snapshot of all messages.)

/// Use [PerAccountStore.messages] to check a message's topic accurately.

chrisbobbe · 2025-11-04T05:14:42Z

lib/model/channel.dart

+  /// Handle a [MessageEvent], returning whether listeners should be notified.
+  bool handleMessageEvent(MessageEvent event) {
+    if (event.message is! StreamMessage) return false;
+    final StreamMessage(:streamId, :topic) = event.message as StreamMessage;


nit:

Suggested change

final StreamMessage(:streamId, :topic) = event.message as StreamMessage;

final StreamMessage(streamId: channelId, :topic) = event.message as StreamMessage;

chrisbobbe · 2025-11-04T05:18:28Z

lib/model/channel.dart

+    // If this message is already the latest message in the topic because it was
+    // received through fetch in fetch/event race, or it is a message sent even
+    // before the latest message of the fetch, we don't do the update.


I'm getting confused trying to parse this sentence. Instead, how about, inside the if just below this, say something like:

// The event raced with a message fetch.

chrisbobbe · 2025-11-04T05:25:50Z

lib/model/channel.dart

+    // If we don't already know about the list of topics of the channel this
+    // message belongs to, we don't want to proceed and put one entry about the
+    // topic of this message, otherwise [fetchTopics] and the callers of
+    // [getChannelTopics] would assume that the channel only has this one topic
+    // and would never fetch the complete list of topics for that matter.
+    if (latestMessageIdsByTopic == null) return false;


How about:

Suggested change

// If we don't already know about the list of topics of the channel this

// message belongs to, we don't want to proceed and put one entry about the

// topic of this message, otherwise [fetchTopics] and the callers of

// [getChannelTopics] would assume that the channel only has this one topic

// and would never fetch the complete list of topics for that matter.

if (latestMessageIdsByTopic == null) return false;

if (latestMessageIdsByTopic == null) {

// We're not tracking this channel's topics yet.

// We start doing that when [fetchTopics] is called,

// and we fill in all the topics at that time.

return false;

}

chrisbobbe · 2025-11-04T05:39:53Z

lib/model/channel.dart

+    final origLatestMessageIdsByTopics = _latestMessageIdsByChannelTopic[origStreamId];
+    // We only handle the case where all the messages of [origTopic] are
+    // moved to [newTopic]; in that case we can remove [origTopic] safely.
+    // But if only one messsage is moved (`PropagateMode.changeOne`) or a few
+    // messages are moved (`PropagateMode.changeLater`), we cannot do anything
+    // about [origTopic] here as we cannot determine the new `maxId` for it.
+    // (This is the case where there could be multiple channel-topic keys with
+    //  the same `maxId`)
+    if (propagateMode == PropagateMode.changeAll
+        && origLatestMessageIdsByTopics != null) {
+      shouldNotify = origLatestMessageIdsByTopics.remove(origTopic) != null;
+    }


With a switch/case on propagateMode, I think we can be less verbose in the comment, e.g.:

Suggested change

final origLatestMessageIdsByTopics = _latestMessageIdsByChannelTopic[origStreamId];

// We only handle the case where all the messages of [origTopic] are

// moved to [newTopic]; in that case we can remove [origTopic] safely.

// But if only one messsage is moved (`PropagateMode.changeOne`) or a few

// messages are moved (`PropagateMode.changeLater`), we cannot do anything

// about [origTopic] here as we cannot determine the new `maxId` for it.

// (This is the case where there could be multiple channel-topic keys with

// the same `maxId`)

if (propagateMode == PropagateMode.changeAll

&& origLatestMessageIdsByTopics != null) {

shouldNotify = origLatestMessageIdsByTopics.remove(origTopic) != null;

}

final origLatestMessageIdsByTopics = _latestMessageIdsByChannelTopic[origStreamId];

switch (propagateMode) {

case PropagateMode.changeOne:

case PropagateMode.changeLater:

// We can't know the new `maxId` for the original topic.

// Shrug; leave it unchanged. (See dartdoc of [getChannelTopics],

// where we call out this possibility that `maxId` is incorrect.

break;

case PropagateMode.changeAll:

if (origLatestMessageIdsByTopics != null) {

origLatestMessageIdsByTopics.remove(origTopic);

shouldNotify = true;

}

}

chrisbobbe · 2025-11-04T05:54:30Z

lib/model/channel.dart

+    final newLatestMessageIdsByTopics = _latestMessageIdsByChannelTopic[newStreamId];
+    if (newLatestMessageIdsByTopics != null) {
+      final movedMaxId = event.messageIds.max;
+      if (!newLatestMessageIdsByTopics.containsKey(newTopic)
+          || newLatestMessageIdsByTopics[newTopic]! < movedMaxId) {
+        newLatestMessageIdsByTopics[newTopic] = movedMaxId;
+        shouldNotify = true;
+      }
+    }


A bit easier to read, I think:

Suggested change

final newLatestMessageIdsByTopics = _latestMessageIdsByChannelTopic[newStreamId];

if (newLatestMessageIdsByTopics != null) {

final movedMaxId = event.messageIds.max;

if (!newLatestMessageIdsByTopics.containsKey(newTopic)

|| newLatestMessageIdsByTopics[newTopic]! < movedMaxId) {

newLatestMessageIdsByTopics[newTopic] = movedMaxId;

shouldNotify = true;

}

}

final newLatestMessageIdsByTopics = _latestMessageIdsByChannelTopic[newStreamId];

if (newLatestMessageIdsByTopics != null) {

final movedMaxId = event.messageIds.max;

final currentMaxId = newLatestMessageIdsByTopics[newTopic];

if (currentMaxId == null || currentMaxId < movedMaxId) {

newLatestMessageIdsByTopics[newTopic] = movedMaxId;

shouldNotify = true;

}

}

sm-sayedi · 2025-11-04T19:22:05Z

Thanks @chrisbobbe for the review. Pushed new revision.

Co-authored-by: Sayed Mahmood Sayedi <[email protected]>

Fixes: zulip#1499 Co-authored-by: Sayed Mahmood Sayedi <[email protected]>

chrisbobbe

Thanks, and sorry for the delay! Comments below, this time from a full review.

chrisbobbe · 2025-11-08T01:57:46Z

test/widgets/topic_list_test.dart

  });

-  testWidgets('fetch again when navigating away and back', (tester) async {
+  testWidgets("dont't fetch again when navigating away and back", (tester) async {


nit: spelling of "don't"

chrisbobbe · 2025-11-19T00:37:29Z

test/api/route/route_checks.dart

  Subject<Uri> get realmUrl => has((e) => e.realmUrl, 'realmUrl');
 }

+extension GetStreamTopicEntryChecks on Subject<GetStreamTopicsEntry> {


"GetStreamTopicsEntryChecks"

chrisbobbe · 2025-11-19T00:44:45Z

test/model/store_checks.dart

  Subject<Map<int, ZulipStream>> get streams => has((x) => x.streams, 'streams');
  Subject<Map<String, ZulipStream>> get streamsByName => has((x) => x.streamsByName, 'streamsByName');
  Subject<Map<int, Subscription>> get subscriptions => has((x) => x.subscriptions, 'subscriptions');
+  Subject<List<GetStreamTopicsEntry>?> getStreamTopics(int streamId) => has((x) => x.getChannelTopics(streamId), 'getStreamTopics');


getChannelTopics and 'getChannelTopics', to match PerAccountStore.getChannelTopics

chrisbobbe · 2025-11-19T00:50:01Z

lib/model/channel.dart

 import 'user.dart';

+// similar to _apiSendMessage in lib/model/message.dart
+final _apiGetChannelTopics = getStreamTopics;


Could you add a prep commit that renames the existing binding getStreamTopics to getChannelTopics, then consistently use "channel topics" naming instead of "stream topics" naming?

chrisbobbe · 2025-11-19T00:50:55Z

lib/model/channel.dart

+  /// can be retrieved with [getChannelTopics].
+  Future<void> fetchTopics(int channelId);
+
+  /// The topics in the given channel, along with their latest message ID.


Suggested change

/// The topics in the given channel, along with their latest message ID.

/// The topics the user can access, along with their latest message ID,

/// reflecting updates from events that arrived since the data was fetched.

chrisbobbe · 2025-11-19T04:10:18Z

test/widgets/topic_list_test.dart

    connection.prepare(json: GetStreamTopicsResult(
      topics: [eg.getStreamTopicsEntry(name: 'topic B')]).toJson());


If we don't expect to fetch again, we shouldn't prepare a response, right?

chrisbobbe · 2025-11-19T04:13:44Z

test/widgets/topic_list_test.dart

-    ]);
+  testWidgets('show topic action sheet before and after moves', (tester) async {
+    final channel = eg.stream();
+    final message = eg.streamMessage(id: 123, stream: channel, topic: 'foo');


Is 123 chosen because it equals eg.defaultStreamMessageStreamId? We should use eg.defaultStreamMessageStreamId instead, if so, or else pass 123 as streamId to the eg.stream() call.

chrisbobbe · 2025-11-19T04:14:25Z

test/widgets/topic_list_test.dart

+    // After the move, the message with maxId moved away from "foo". The topic
+    // actions that require `someMessageIdInTopic` is no longer available.


nit: "are no longer available"

chrisbobbe · 2025-11-19T04:14:59Z

test/widgets/topic_list_test.dart

+    await tester.tap(find.text('Cancel'));
+    await tester.pump();
+
+    // Topic actions that require `someMessageIdInTopic` is available


nit: "are available"

chrisbobbe · 2025-11-19T04:16:53Z

test/widgets/topic_list_test.dart

+    check(find.descendant(of: topicItemFinder,
+      matching: find.text('foo'))).findsOne();


How about:

check(findInTopicItemAt(0, find.text('foo'))).findsOne();

?

sm-sayedi added the maintainer review PR ready for review by Zulip maintainers label Oct 28, 2025

sm-sayedi requested a review from chrisbobbe October 28, 2025 21:25

sm-sayedi assigned chrisbobbe Oct 28, 2025

sm-sayedi force-pushed the 1499-track-topics branch from 1019746 to 2e0fcfd Compare November 3, 2025 17:06

chrisbobbe reviewed Nov 4, 2025

View reviewed changes

sm-sayedi force-pushed the 1499-track-topics branch from 2e0fcfd to c37de07 Compare November 4, 2025 19:18

sm-sayedi requested a review from chrisbobbe November 4, 2025 19:22

sm-sayedi force-pushed the 1499-track-topics branch from c37de07 to 02ed253 Compare November 7, 2025 16:41

PIG208 and others added 5 commits November 14, 2025 18:24

channel: Keep track of channel topics, and keep up-to-date with events

8cb43e6

Co-authored-by: Sayed Mahmood Sayedi <[email protected]>

autocomplete test [nfc]: s/streams/topics in a topic test name

c604a9f

autocomplete test [nfc]: Add and use isTopic condition

7cda4cf

autocomplete: Use topic data from store in topic autocomplete

e0baf65

topics: Keep topic-list page updated, by using topic data from store

7fca9bb

Fixes: zulip#1499 Co-authored-by: Sayed Mahmood Sayedi <[email protected]>

sm-sayedi force-pushed the 1499-track-topics branch from 02ed253 to 7fca9bb Compare November 14, 2025 13:54

chrisbobbe reviewed Nov 19, 2025

View reviewed changes

	/// Pairs of the known topics and its latest message ID, in the given channel.
	/// The topics in the given channel, along with their latest message ID.

	/// Returns null if the data has never been fetched yet.
	/// Returns null if the data has not been fetched yet.

		/// The result is guaranteed to be sorted by [GetStreamTopicsEntry.maxId]
		/// descending, and the topics are guaranteed to be distinct.

-  /// In some cases, the same maxId affected by message moves can be present in
-  /// multiple [GetStreamTopicsEntry] entries. For this reason, the caller
-  /// should not rely on [getChannelTopics] to determine which topic the message
-  /// is in. Instead, refer to [PerAccountStore.messages].
-  /// See [handleUpdateMessageEvent] on how this could happen.
+  /// Occasionally, [GetStreamTopicsEntry.maxId] will refer to a message
+  /// that doesn't exist or is no longer in the topic.
+  /// This happens when a topic's latest message is moved or deleted
+  /// and we don't have enough information
+  /// to replace [GetStreamTopicsEntry.maxId] accurately.
+  /// (We don't keep a snapshot of all messages.)
+  /// Use [PerAccountStore.messages] to check a message's topic accurately.

	final StreamMessage(:streamId, :topic) = event.message as StreamMessage;
	final StreamMessage(streamId: channelId, :topic) = event.message as StreamMessage;

	/// The topics in the given channel, along with their latest message ID.
	/// The topics the user can access, along with their latest message ID,
	/// reflecting updates from events that arrived since the data was fetched.

		connection.prepare(json: GetStreamTopicsResult(
		topics: [eg.getStreamTopicsEntry(name: 'topic B')]).toJson());

		// After the move, the message with maxId moved away from "foo". The topic
		// actions that require `someMessageIdInTopic` is no longer available.

		check(find.descendant(of: topicItemFinder,
		matching: find.text('foo'))).findsOne();

Keep track of the known topics in store #1951

Are you sure you want to change the base?

Keep track of the known topics in store #1951

Uh oh!

Conversation

sm-sayedi commented Oct 28, 2025

Uh oh!

chrisbobbe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sm-sayedi commented Nov 4, 2025

Uh oh!

chrisbobbe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants