Hey I made a distributed reddit zst to parquet converter to make it much easier to query this dataset locally. It got a bit overcomplicated but it worked! All the parquet files also have a single schema to make queries more efficient and merging months not much of a hassle for queries. Your arctic_shift project helped with all the schema info and being a central place of where you can get more current reddit archives. Take a look: https://github.com/theontho/reddit-zst-to-parquet
Hey I made a distributed reddit zst to parquet converter to make it much easier to query this dataset locally. It got a bit overcomplicated but it worked! All the parquet files also have a single schema to make queries more efficient and merging months not much of a hassle for queries. Your arctic_shift project helped with all the schema info and being a central place of where you can get more current reddit archives. Take a look: https://github.com/theontho/reddit-zst-to-parquet