Python can't read `colexifications.csv` (with default settings)

[johenglisch]

I updated the meta database the other day and the clics4 dataset triggered an error. It seems that python's built-in csv parser rejects the file due to a cell being too large:

import csv
import io
import zipfile
with zipfile.ZipFile('cldf/colexifications.csv.zip') as zf:
    with zf.open('colexifications.csv') as raw_f:
        unicode_f = io.TextIOWrapper(raw_f, encoding='utf-8')
        rdr = csv.reader(unicode_f)
        rows = list(rdr)

…throws:

_csv.Error: field larger than field limit (131072)

Consuming programs can avoid the problem by extending python's field size limit, e.g.:

csv.field_size_limit(256 * 1024)

I did that on my end but to be completely honest, the fact that an individual data point can't fit into 128 KiB is kind of a ‘data smell’. Maybe those Forms, Varieties, and Languages columns should be put into separate tables rather than being in-lined arrays within a table cell.

(As a side note: this also means that you currently can't run cldf validate on the data. At least not from the command-line – the dataset can still be validated from within Python after expanding the field size limit.)

[xrotwang]

I don't think large cell values in a CSV file are a data smell per se. To the contrary, the fact that one can put things like JSON or data URLs reliably (and transparently) into CSV tables seems like a big advantage to me. I didn't want to increase Python's cell limit by default, because often large cell values can be a symptom of other problems.

But of course, increasing the cell limit should be an option available to all CLDF tools. See cldf/pycldf#195

[johenglisch]

I don't think large cell values in a CSV file are a data smell per se. To the
contrary, the fact that one can put things like JSON or data URLs reliably (and
transparently) into CSV tables seems like a big advantage to me.

Yes, 100%.

I just think there comes a point where large becomes too large. Where that threshold lies depends on context, naturally. Obviously an array of five elements is not too large; obviously a 5GB data point is. 128 KB is ultimately just a number a C programmer came up to have a nice fixed buffer size. But after like 700 releases this is the first dataset to actually hit this number (at least in the tables I load into the meta database), so I thought it might be worth bringing up whether there's something odd going on here or whether this is still within reasonable margins for clics. (At the end of the day a ‘smell’ is just that – a vague feeling that something might be off, not an immediate alarm signal)

I didn't want to increase Python's cell limit by default, because often large
cell values can be a symptom of other problems.

That makes perfect sense.

But of course, increasing the cell limit should be an option available to all
CLDF tools. See cldf/pycldf#195

Yeah, that would be good, too, thanks. (^^)

[xrotwang]

I turns out cldf validate does accept a -z option already - just as option of the main cldf command, not the subcommand. So cldf -z <big number> validate ... should work.