Allow purging or filtering specific discord channels or servers

[DCsunset]

Hi,

Thanks for this great project! One issue I encounter is that the archived traffic keeps growing quickly and uses a lot of space because I've joined many servers. But at the same time, I don't wanna archive all channels or all servers cuz I'm mostly interested in archiving DMs.

So is there a way to filter or purge traffic from discord servers or channels? It would be very helpful if this feature could be supported.

Thanks

[recursivetree]

Currently, there is no way to achieve reduce storage space by filtering per-server or DM. I don't see why it couldn't be done, but someone has to implement it.

One reason discordless doesn't filter traffic is so that in case discord changes their protocol, we keep the recording so an updated exporter can deal with them later on (kind of like harvest now, decrypt later attacks on cryptography).

If disk space is a real issue, you could consider exporting occasionally and discard the originals afterwards.

The size of the exported data is usually less of an issue than the raw recordings, but at least the htmeml exporter can filter by servers using the --limit-guilds option.

[DCsunset]

Thanks for the quick response. II can try helping implement it but I guess i need some time to understand the protocol and storage layout first.

As for the reason you mentioned (keep the recording so an updated exporter can deal with them later), I'm still a bit confused. If we only filter out traffic for some servers, it shouldn't matter since we still keep the original format of other traffic right?

[recursivetree]

Thanks for the quick response. II can try helping implement it but I guess i need some time to understand the protocol and storage layout first.

Take your time and open a PR once you are ready. I'll take a look.

As for the reason you mentioned (keep the recording so an updated exporter can deal with them later), I'm still a bit confused. If we only filter out traffic for some servers, it shouldn't matter since we still keep the original format of other traffic right?

Discord generates a lot of traffic to many endpoints. Any effective filtering at recording time would have to filter out the relevant data on a whitelist basis. If now e.g. the URL of the messages endpoint changes, the whitelist wouldn't catch it.

As long as it is optional, I don't really mind adding something like this though.

Besides that, for filtering after servers, witm for example doesn't know which channel belongs to which server (this data is really well hidden in gateway messages). This approach would significantly complicated witm and has a few problems we have to solve first.

A perhaps better alternative would be a "compressor" that converts discordless recordings to discordless recordings, but trims them of unnecessary data.

[DCsunset]

Yeah I think the compressor idea sounds neat and practical. So we can filter unnecessary data, archive, and then delete the original traffic. Future traffic can be processed in the same way and merged into the same archive