Add "Messaging API" spec

[weboko]

This PR attempts to define Messaging API as per all previous conversations in various threads or offline.

This doc describes following parts of said umbrella API:

• initial configuration - just configuration;
• Send API - send for message with retry and ack checks;
• Subscribe API - initiate, track and manage ongoing source of messages;
• Messaging Storage API - entry point for access for messages;
• History API - allows to fetch past messages;
• Event Source API - provides reactive way for app devs;

These sections cover programmatic API as well as REST API.

For code sections typescript was used AND I am not sure it is the best option for our case so I propose either to:

• add definitions in other languages alongside;
• change completely to pseudocode for definitions;

Unfortunately the document turned out rather big, with some degree of repetitions in explanations and references between sections. Please, let me know right after first read if it is too much and I will rewrite it for further review.

[weboko]

Thumb up if you want me to walk you through the document on a short call. :)

[weboko]

General question to anyone from me:

• I don't believe we actually need this property AS Messaging API can be smart enough to determine and prioritize underlying protocols.

I am in favor of removing it altogether giving a seamless way for mounting Messaging API on any node.

[vpavlin]

Automatic resolution of mode would be the best, but at the same time, I would kinda expect to have 3 modes - auto (default), edge, service - ideally I never have to switch manually, but then I also know my shitty internet connection at home and being able to explicitly switch (as long as an app exposes that option, obviously) to edge mode would make sense to me.

[fryorcraken]

On a first RAW spec I would not include an auto mode. We can plan and add this later. Probably around the time we build a full stack JS.
I'd suggest

• edge
• relay: relay + service for filter, light push and peer exchange - like "relay mode" in status desktop
• service (or store): relay + provide service for store.

[weboko]

Then I keep edge and introduce relay.
As for the service, correct me if I am wrong, but I think it is out of the scope of Messaging API.

[weboko]

b6e6e62

[fryorcraken]

Are we suggesting that for now, if one wants to run a store node or full blow service node, they would use a binary such a wakunode2 and not the messaging API?

Sounds fair (for now).

[vpavlin]

Requiring store does not feel right - Store nodes have different expectations than Lightpush and Filter nodes - those can come and go and still be useful (unless the fluctuation is in order of seconds or minutes), but Store (even with Store sync) should be more stable and available + enabling the protocol puts requirement on hosting the DB.

I'd personally see store managed separately

[vpavlin]

To add to it, I might be fine if my phone is a service node when on Wifi and plugged in the charger, but I would not want it to store GBs of random and "useless" data:)

[fryorcraken]

I agree here. see my response in https://github.com/waku-org/specs/pull/19/files#r1998346439. Modes should be edge, relay, store. with relay providing services for light push, filter and px

[vpavlin]

Should there also be isPending(requestId: RequestID) method, or is iterating over getPendingRequests() result enough?

[Ivansete-status]

I think I prefer this idea of getting info from a particular RequestID
Maybe, another option could be:

public getRequestState(requestId: RequestID): ReqState

Where ReqState can be:

• Delivered
• Pending
• NotDelivered
• Cancelled

[fryorcraken]

Ok very interesting here.

Due to the nature of Waku's domain (async network request), it seems much more appropriate to have an event based approach.

See https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/36/bindings-api.md#waku_set_event_callback

having getRequestState implies someone is going to loop on that - not a great design.

I'd remove in favor of event listening/callbacks

[chaitanyaprem]

I agree with @fryorcraken here. It would be better to have an event based approach to indicate success/failure/timeout etc.

[vpavlin]

Can we avoid using pubsubTopic and say shard? Since that is what is configured on node's creation?

[vpavlin]

One thing that I was never really sure about is why do we splic Decoder and SubsciptionListener - given that Decoder has all the info about what to listen for (and how to prep the message e.g. with decryption) and then SubsciptionListener has info about what to do with the message - why not to make SubscriptionListener a property of decoder? And then Subscribe would just receive the decoder. I am curious what am I mising:D

[fryorcraken]

I would remove decoder.

[vpavlin]

As I mentioned above - would it make sense to move away from pubsubTopic in dev/user facing parts and only refer to shard?

[vpavlin]

I understood confirmStored as targetting sent messages (i.e. to confirm a message I published reached the network).

What if I don't necessarily need confirmStore for sending (e..g I am just publishing an "I am online" heartbeat, so I don't really care if they are missed sometime), but I do care for regular store queries as I'd really like to make sure I get all the messages?

Basically, are we conflating 2 separate features under confirmStored?

[chaitanyaprem]

I agree and think that we should have 2 configs as it is currently implemented in status. We have separate flag to indicate sendConfirmations vs fetchMissingMessages

[vpavlin]

There is no clusterId in /subscribe - this message would utterly confuse me as a developer:) Probably related to my question(s) about replacing pubsubTopic with shard (and then also adding clusterid to /subscribe for completeness)

[vpavlin]

Should there also be 404 or if the subscription does not exist, or would it return 200 since it is ok, kinda

[vpavlin]

Should this be URL encoded?

[Ivansete-status]

Thanks so much for it! 🙌
Some comments so far

[Ivansete-status]

I would start without those and always ensure any message is delivered by default, i.e., for now I wouldn't add additional constraints until we get the implementations more mature

[chaitanyaprem]

It might be better to have this config e.g in case of status there might be many content-topics which are used for broadcasting repetetive status's and other info even if missed once would get updated in the next broadcast interval. Such topics might not require store confirmations.
From an implementation pov this doesn't seem complicated.

[Ivansete-status]

I think is interesting to mention store-client, filter-client, and lightpush-client. The "no-client" component is used by service nodes.

[weboko]

@Ivansete-status I am not sure I understand this comment, can you elaborate, please?

[Ivansete-status]

@Ivansete-status I am not sure I understand this comment, can you elaborate, please?

Sure! The edge nodes must act as both lightpush-client and filter-client. In both cases, the edge node is not directly mounting LIGHTPUSH nor FILTER protocols. Instead, the edge node just dial peers that mount those protocols. Therefore, I think is more precise to mention that.

Another option would be something like:

MUST use [LIGHTPUSH](../standards/core/lightpush.md) as a client and connect to peers that mount LIGHTPUSH

and

[FILTER](https://github.com/vacp2p/rfc-index/blob/7b443c1aab627894e3f22f5adfbb93f4c4eac4f6/waku/standards/core/12/filter.md) as a client and connect to peers that mount FILTER

[Ivansete-status]

I wouldn't define that Encoder type

[Ivansete-status]

As suggested in the send section , I think we need to bring more feedback to the caller.

[Ivansete-status]

I think the "subscription" term doesn't reflect completely what it does

Suggested change

	##### `SubscriptionListener`
	##### `OnMsgReceived`

[Ivansete-status]

I think this is not needed.
The onMsgReceived(contentTopic: string, payload: array[byte]) , or similar , should suffice.

[vpavlin]

This depends on how smart the Messaging API should be - e.g. in Waku Dispatcher my "onMsgReceived` function signature looks like this: https://github.com/vpavlin/waku-dispatcher/blob/main/src/dispatcher.ts#L135 - I kind of like the flexibility of using Encoder/Decoder gives us and potentially gives all the devs (I could implement my own Decoder which would do some smart validation or processing)

Another thing is you cannot just pass payload and contentTopic - the dev also needs the rest of the fields - like meta, timestamp or rln proof:)

[Ivansete-status]

I kind of like the flexibility of using Encoder/Decoder gives us and potentially gives all the devs

ah okay I see. I have the impression that the Encoder/Decoder concept fits very well from typescript PoV but I wonder how it would fit from a C API PoV

---

Another thing is you cannot just pass payload and contentTopic - the dev also needs the rest of the fields - like meta, timestamp or rln proof:)

Yes good point! I agree that meta and timestamp is needed but maybe rln proof not needed

[chaitanyaprem]

Another thing is you cannot just pass payload and contentTopic - the dev also needs the rest of the fields - like meta, timestamp or rln proof:)

Why not pass WakuMessage itself which has all these fields?

[fryorcraken]

I would move away from the Decoder in the context of the spec.

But js-waku is welcome to have an abstraction specific to JS/TS over the messaging API is decoder/encoder is a better dev ex.

[Ivansete-status]

I think the users shouldn't be aware to any Storage. We need to handle the store queries internally and the users shouldn't worry about any kind of history API. We already have the "expert" API for that purpose :)

[vpavlin]

I think we need some Message Storage API:-) The naming might need to be tweaked, but as a dev, I want to be able to sift through the messages I have sent and received, to filter them, or verify things - and it would be bad if everyone had to implement this on their own - Messaging API is a great place for it:) (speaking again based on what I needed in Waku Dispatcher - https://github.com/vpavlin/waku-dispatcher/blob/main/src/dispatcher.ts#L482)

[chaitanyaprem]

How about we have this API only to do time-range based retrieval queries where a node has come back from offline state and wants history.

The hash based queries can be done behind the scenes as part of the Messaging API for reliability checks.

[fryorcraken]

I think we are talking about local storage. Hence, i would rename to Local Message Storage API.

I would NOT include this at this point in time, and would instead aim for the scope to be narrower and spec leaner.

However, I believe that the opposite is missing. A Waku implementation needs access to a local DB or cache to function properly, and remember message ids. etc.
This should be defined here. See how go-waku interfaces with status-go DB to mark messages as sent etc.

So when you start a node, there is a couple of components you should pass to your node, and we need to define the APIs those component must implement.
Another one is a time source, and DNS source (or maybe just DNS Servers ips or DoH addresses).

[fryorcraken]

I have reviewed half. I think it's enough comment for now.

Let me know if you prefer a full review.

Hopefully, lot of opportunity to thin it out.

[fryorcraken]

Not convinced encoder concept works here. Let's see how it looks on C API.

[chaitanyaprem]

Why do we need to have an encoder? Shouldn't encoder be more like a utility from waku/sdk app devs can use to encode their payload and then supply WakuMessage to be sent? This gives flexibility for devs to decide whether to use Waku encoder or some other way to encode their messages.

This looks like better separate of concerns in case apps want to use their own way to encode different message types e.g: in status we have private messages which are encrypted/encoded differently and public messages which are done differently. Ultimately status backend handsover only WakuMessage to the send API which seems more cleaner to me.

[fryorcraken]

Do we need/use that? sounds too low level

[chaitanyaprem]

Agreed it is too low level, rather a more simpler approach would be for the user to indicate a timeout for the send API. The user can also pass a context(this might be too Golang specific but is a very good concept for an API in general) which can be cancelled or timeout.

[weboko]

I thought of having it as a backup for consumers to have granular control
but after all discussions with you I think maybe some start and stop methods will be enought

[fryorcraken]

let's do simple first. https://www.notion.so/Waku-Mission-and-Methodology-1a58f96fb65c80b9b789c1bbb9e99915?pvs=4#1d78f96fb65c802c928af2576c55bc20

So yes, start/stop sounds fair

[fryorcraken]

Ok very interesting here.

Due to the nature of Waku's domain (async network request), it seems much more appropriate to have an event based approach.

See https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/36/bindings-api.md#waku_set_event_callback

having getRequestState implies someone is going to loop on that - not a great design.

I'd remove in favor of event listening/callbacks

[fryorcraken]

No mention of pubsubtopic in the API, we only deal with shards and clusters.

[fryorcraken]

I would not define things twice. Only once in C and provide a recommendation on how this should translate to a REST API

[fryorcraken]

@weboko thoughts on this?

IMO

define a C API:

char* waku_send(char* message, int shard, int timeoutMs, WakuCallBack onOkCb, WakuCallBack onErrorCb){}

and define a way to deduce the REST endpoint:

1. name function because route, _ becomes /
2. data arguments are properties of the JSON payload
3. bytes payload are encoded in base64
4. callbacks are properties on the response JSON/websocket

Which would be:

``
POST /waku/send

{
message: WakuMessageJSON,
shard: number,
}

etc

[fryorcraken]

Discussion around store abstraction: https://forum.vac.dev/t/no-store-in-messaging-api/455/4

[fryorcraken]

I would also reserve a space for RLN. It's not in scope but I overlook to specify it https://www.notion.so/Waku-FURPS-1498f96fb65c803faedef2a591c22c00?d=1c48f96fb65c80369e8f001ca5c7e353&pvs=4#1578f96fb65c80778215ef012c6f6136

[chaitanyaprem]

Reviewed till Send API, will continue filter review little later.

[chaitanyaprem]

It might be better to have this config e.g in case of status there might be many content-topics which are used for broadcasting repetetive status's and other info even if missed once would get updated in the next broadcast interval. Such topics might not require store confirmations.
From an implementation pov this doesn't seem complicated.

[chaitanyaprem]

Just to confirm that all the shards that are supported within the cluster are to be specified even if a node is only interested in a few of them?
wondering how to address this scenario though or is it something for later stage?

[weboko]

By this description I implied that this is a list of shards that node is interested in, i.e MUST operate at
is present description misleading, @chaitanyaprem ?

[chaitanyaprem]

for autosharding logic to work properly there must be a config somewhere to be specified how many shards are there in the cluster.
wanted to know which config is this one. if it is only the shards the node is interested in, then we may need to specify another config to indicate total shards in the network.
e.g cluster-id 1 has 8 shards, but node is only interested in shards 0,1 then this config will list [0,1].
But somewhere shardCount 8 also has to be specified so that autosharding logic can use that to determine shard based on contentTopic.

[weboko]

@chaitanyaprem , actually you are right, I missed it
for some reason I thought all auto shard clusters expected to have 8 shards
adding an option for it with default value being 8 shards

let me know if you agree with that - cb446bd

on a side note - it seems numbers of shards is only needed for deriving shard from content topic
in that sense, shouldn't it be included into content topic format itself?

[chaitanyaprem]

on a side note - it seems numbers of shards is only needed for deriving shard from content topic
in that sense, shouldn't it be included into content topic format itself?

well, the idea was to make users agnostic to shards altogether and only use content-topic. but as each cluster can define its own number of shards, it is requiring to be configured locally. Note that the ideal way to get this info would be through some network configuration made maybe inside RLN contract rather than local configuration. As we don't have network configuration specified anywhere as of now, it is required to be configured locally.

The idea of encoding shard-count in content-topic would create a dependency of content-topic on shard count. In a cluster ideally the number of shards can change over time based on the network's requirements e.g today cluster 1 has 8 shards based on some napkin math of network requirements and bandwidth usage, but this may get increased in future. If we make content-topic format dependent on shard-count this would create problem to all existing content-topics when shard count changes in the network and would require migration etc.

[chaitanyaprem]

let me know if you agree with that - cb446bd

maybe the order and naming of config could be something like this as clusterId and shardsInCluster are network level config which are not specific to node being initialized whereas shardsOfInterest is something specific to the local node.

clusterId:
shardsInCluster:
shardsOfInterest:

[fryorcraken]

I started to look into this, but I haven;t yet done a PR on nwaku side to propose a solution. I somewhat agree with the above but wanted to summarize a bit more.

1) Autosharding:

The number of shards in cluster is needed shardsInCluster is a good name. I would not use "under".

2) Static sharding

In this instance, the shards to subscribe to need to be specified when one wants to receive messages from it.
shards is not qualifying enough, so I'd be more explicit with shardsToSubscribe

Note that those configuration are mutually exclusive and there's a question on whether we want to represent that in the config.

type shardingMode = "auto" | "static"
autoShardingConf: {
	shardsInCluster: number;
	},
staticShardingConf: {
	shardsToSubscribe: number[]
	}
}

Now there is the question on whether shards subscriptions should be part of the init config, or done when sending/receiving messages.

I think it is fine to aim for simple first, and make it part of the init config, which is restrictive. Which leads to:

type shardingMode = "auto" | "static"

autoShardingConf: {
	shardsInCluster: number;
	subscribeTo: string[]
	},

staticShardingConf: {
	shardsToSubscribe: number[]
	}
}

Knowing that subscribeTo would only be used in relay mode. The name can be better. But contentTopics is not really appropriate as we don't use the full content topic, just part of it. More thoughts for food.
How do we call the part of the content topic we use for autosharding?

[chaitanyaprem]

Agreed it is too low level, rather a more simpler approach would be for the user to indicate a timeout for the send API. The user can also pass a context(this might be too Golang specific but is a very good concept for an API in general) which can be cancelled or timeout.

[chaitanyaprem]

I agree with @fryorcraken here. It would be better to have an event based approach to indicate success/failure/timeout etc.

[chaitanyaprem]

Why do we need to have an encoder? Shouldn't encoder be more like a utility from waku/sdk app devs can use to encode their payload and then supply WakuMessage to be sent? This gives flexibility for devs to decide whether to use Waku encoder or some other way to encode their messages.

This looks like better separate of concerns in case apps want to use their own way to encode different message types e.g: in status we have private messages which are encrypted/encoded differently and public messages which are done differently. Ultimately status backend handsover only WakuMessage to the send API which seems more cleaner to me.

[chaitanyaprem]

Why not use the term messageHash as it is used in Store API and in all other existing specs?

[weboko]

@chaitanyaprem

since message hash takes into account timestamp - due to retries it can change over time

hence we need a better way to identify requests TO Messaging API FOR sending a message
then we have a way to match RequestID with end message hash

This was my reasoning here, let me know what you think!

[chaitanyaprem]

ok, i see what you mean. I am wondering this may pose challenge in quering message from store as user won't have messageHash to work with or use SDS (as i think it relies on messageHashes).

But then again if the idea is to abstract away both of these from user, then it is fine i guess. But this is more of a transactionID between user and local waku messaging API and nothing beyond that right.

[fryorcraken]

I think it's fine for the request id to be purely implementation specific, and mainly used to track events.

[chaitanyaprem]

In case of relay contentTopic would be optional, whereas in case of Filter it is mandatory. Would be good to document his in the API so that users are aware based on the nodeType how the params are to be passed.

[fryorcraken]

In case of relay contentTopic would be optional, whereas in case of Filter it is mandatory.

Well, ideally one uses autosharding, making it the same interface for both relay and filter :)

[chaitanyaprem]

I agree and think that we should have 2 configs as it is currently implemented in status. We have separate flag to indicate sendConfirmations vs fetchMissingMessages

[chaitanyaprem]

Maybe better to add a note that users would need to have a way to deduplicate messages (returned by 2 filter subscriptions or store nodes etc). Or is that implemented underneath this API? If so, better to mention that as well for implementers.

[fryorcraken]

yes I agree here, we may need to define a deduplication logic on payload within a timeframe.

[chaitanyaprem]

wouldn't it be better to return an error indicating the same?

[fryorcraken]

I think ignore is fine, because you may to unsubscribe for a clean shutdown but if the remote node already unsubscribe you because you were disconnected or something, you dont' want to be collecting errors.

[chaitanyaprem]

Another thing is you cannot just pass payload and contentTopic - the dev also needs the rest of the fields - like meta, timestamp or rln proof:)

Why not pass WakuMessage itself which has all these fields?

[chaitanyaprem]

Generic Comments:

1. Might be good to have a section listing all API's and referring to subsections for each of them. otherwise for a dev/user it would be hard to navigate the spec to read so much detail.
2. Also it might be a good idea to provide an API for the user to get localNodeInfo i.e (MultiAddr node is listening on, ENR of the node if applicable). This would be useful in relay node+service node scenarios. So maybe this API is applicable only if node is initialized with relay/serviceNode.

[weboko]

Added table of content f1d100f

As for (2) - I think it is out of scope of Messaging API as this is something that, I think, must be provided by any Waku client.

We can thought use an opportunity here to specify it. Let me know if you are in favor of it and I will push that change, @chaitanyaprem .

[chaitanyaprem]

As for (2) - I think it is out of scope of Messaging API as this is something that, I think, must be provided by any Waku client.

ok, since we are defining the API i thought we can provide this as well as it might be useful for users to know their nodeinfo. We can add it as it is a small API.

[weboko]

sure, it's only couple of lines
which part of API do you think it fits best, @chaitanyaprem ?

[chaitanyaprem]

I am guessing where we have health indicator.

[chaitanyaprem]

Would be good to add a note that the callback is expected to not do any long processing so that underlying threads don't get blocked and drops messages.

We have noticed this issue in status once where message's were getting dropped as the app thread was blocked due to some sync logic which was causing reliability issues.

[fryorcraken]

good job. You did not address the question of language. I suggest to try C with a programmatic way of deriving the REST API indicators.

I have commented and there are a number of things that can be left to implementation detail. The spec is being too detailed right now.

I also expressed some opinion around the (local) message storage. I think it should be the opposite where we should define an API here of what a storage solution MUST implement (ie, provided by application developer), for the waku node to use.

[fryorcraken]

If anyone has a better name than "relay" to express the "relay mode" in Status Desktop (service for PX+LP+Filter, usage of discv5+px+rdv for discovery, relay for sending and receiving.), I am all ears.

[fryorcraken]

I started to look into this, but I haven;t yet done a PR on nwaku side to propose a solution. I somewhat agree with the above but wanted to summarize a bit more.

1) Autosharding:

The number of shards in cluster is needed shardsInCluster is a good name. I would not use "under".

2) Static sharding

In this instance, the shards to subscribe to need to be specified when one wants to receive messages from it.
shards is not qualifying enough, so I'd be more explicit with shardsToSubscribe

Note that those configuration are mutually exclusive and there's a question on whether we want to represent that in the config.

type shardingMode = "auto" | "static"
autoShardingConf: {
	shardsInCluster: number;
	},
staticShardingConf: {
	shardsToSubscribe: number[]
	}
}

Now there is the question on whether shards subscriptions should be part of the init config, or done when sending/receiving messages.

I think it is fine to aim for simple first, and make it part of the init config, which is restrictive. Which leads to:

type shardingMode = "auto" | "static"

autoShardingConf: {
	shardsInCluster: number;
	subscribeTo: string[]
	},

staticShardingConf: {
	shardsToSubscribe: number[]
	}
}

Knowing that subscribeTo would only be used in relay mode. The name can be better. But contentTopics is not really appropriate as we don't use the full content topic, just part of it. More thoughts for food.
How do we call the part of the content topic we use for autosharding?

[fryorcraken]

Suggested change

	##### `shardsUnderCluster`
	##### `shardsInCluster`

[fryorcraken]

in a C interface you can specify must be uint

[fryorcraken]

Drop any default in this document. Let's reduce the scope for now to avoid nitpicking and leave it to implementation detail.
I will create a set of "cluster presets" as well so we can solve some of the interop issues we discussed.

[fryorcraken]

as discussed earlier, merged those to a confirmed status.

[fryorcraken]

Too many APIs, especially with getPendingRequests already there.

I would say pick ONE way to check on message status at first, and we can then extend as we learn from dogfooding.

[fryorcraken]

delete that and just referred to the verbose definition in eponymous sections below