Partition requests in traffic archive

[Sanqui]

After two months of running Discordless, my traffic_archive/requests directory has reached 69 thousand files. That's nice, but many programs and operating systems struggle with directories containing tens of thousands of files. This is solved by bucketing or partitioning the files into subdirectories. I suggest implementing a feature in Discordless that will create new subdirectories for requests either by date or when a reasonable number of files like 5000 files is reached. I'm happy to go ahead and make a PR for this feature if it's desired and there isn't something I'm missing.

[Roachbones]

I wouldn't be opposed to it. Do you usually open the directory because you want to inspect recent files? Or is there some other workflow that gets broken by the directory size?

I think a daily subdirectory makes the most sense.

[Sanqui]

The most immediate motivation was that I wanted to demonstrate to somebody how the last images I viewed on Discord get automatically stored, but Nautilus choked on the directory. It's just a little impractical and I'm concerned about the boundless growth here.

I think daily makes intuitive sense, and it's convenient for browsing, but I'm concerned about two things. The first is time zones, DST, and drift - it would need to be UTC at least. The second concern is, if the user runs a selfbot for archiving entire channels or servers - like DiscordChatExporter (proxied through discordless) or a fictional discard3 - perhaps there could easily be more than 10k files stored in a day (for example with a reasonable rate of 10 requests a minute). I don't want to bikeshed this too much, though, so I'm happy to move ahead with YYYY-MM-DD subdirectories.

[recursivetree]

I agree that it is an issue. I currently have around 200k files and growing. Anything but ls | grep doesn't work anymore.

Your approach of using subdirectories sounds reasonable.

However: Some people, like me, have recordings going for like 1+ years that I'd like to keep in a tool-processable state. How do we keep compatibility with old recordings when using the converter tools? I think we'll have to answer that question first.

[Sanqui]

I definitely think no existing traffic_archive directories should break as a result of this change. Considering that request_index contains filenames relative to the requests directory, I can think of two ways to handle this and neither are ideal:

1. When discordless is run with an existing traffic_archive, a one-time migration is performed which creates subdirectories in traffic_archive/requests, relocates old files, and rewrites the request_index file. New requests will be stored in subdirectories in traffic_archive/requests/ as per the new scheme.
2. If it is desired that existing files stay in their current position (so that any references to them continue working, and the request_index isn't mutated), a second directory could be used (requests2 or requests_by_date). The index would either have to point to subdirectories (../requests2/YYYY-MM-DD/... - ugly!) or a new index would have to be made.

With the first strategy, a migration has to be performed, but tools that work with the traffic_archive should continue working without change as long as they're not broken by the fact that the filenames in request_index now start with the subdirectory. With the second strategy, no migration needs to be performed, but tools need to be adjusted to understand that there are now two directories and possibly two indexes.

I find the first solution more clean, I think.

I suppose we should also think whether the gateways directory should be similarly future-proofed, so that we have a unified solution to this problem.

[recursivetree]

While I agree that strategy 1 would give a cleaner traffic_archive, I'm not sure beauty of the layout is the most important factor. For me, it comes with other disadvantages: Since we don't want to break any existing traffic archive, the migration would have to run in the exporters, meaning the exporter will have to rewrite the traffic_archive. This means the traffic_archive has to be writable for the exporter. At least in my automated setup, the traffic_archive is read-only for the exporter to avoid corrupting the traffic_archive if something goes wrong. I'd prefer to keep the traffic_archive read-only for exporters.

I actually quite like the second option. Instead of requests2, I'd perhaps call it requests_by_date. Yes, having two request directories is slightly ugly. But I'd suggest that new install won't even create the old directory, so new install will have a clean layout. Only some old, existing archives will have this quirk.

A third option I was thinking about was rotating through whole traffic_archive directories. Then the structure of a traffic_archive doesn't have to be modified.

And yes, I'd implement the same thing for gateways too.