Problem of Worfklow

[LinguList]

The merging routine in CLDF will make a new CLDF datasets. We can only use this NEW dataset in order to compute colexifications. If we want to make a routine in which the colexifications are then also rendered IN CLDF, this is quite difficult. An intermediate solution would be (that would also work):

1. aggregate data in a first stage, store CLDF data for this stage in a special folder, use some lexibank_clics4_preprocessing.py script for this purpose.
2. compute colexifications and re-write data to CLDF in the main directory, use lexibank_clics4.py

One could also use two directories: clics4-data and clics4.

But for now, it seems the easiest way, since it would be too difficult to hack into CLToolkit and create a CLDF on the fly to feed to the get_colexification function from pyclics (in colexifications branch).

[LinguList]

In fact, this seems too call for a combined workflow, in which we first load the CLDF Wordlist, and then make a structural dataset. Since the writing process is independent, we can combine this within one script!!

[LinguList]

I meant: since the forms.csv etc. are written after leaving the with statement in the script, we can indeed first write the data once, and then infer the colexifications from the data.

[xrotwang]

If we are writing two datasets (rather than just a Wordlist with a ValueTable thrown in), this is indeed possible, see
https://github.com/cldf-datasets/liljegrenhindukush/blob/8f2e22a2d2629a6158e2d3959165504166a8b61e/lexibank_liljegrenhindukush.py#L207

In the second writer context, it should also be possible to access the data previously written via self.cldf_reader().

[LinguList]

Ah, cool. I gave it some thoughts and figured: in fact, we write one dataset, where we first store all forms in a wordlist and then colexifications are handled as a structure dataset, where colexifications are the parameters, and we list exhaustively values for each language if it has them or not. While this will be large, many people always wanted this, and I hope that zipping will keep files considerably small here, and I think access via sqlite should then be easy.

[LinguList]

Nice thing is: if this works, we can do the same for partial colexifications and use it as a template for all future colexification businesses.

[xrotwang]

in fact, we write one dataset, where we first store all forms in a wordlist and then colexifications are handled as a structure dataset,

I'm confused. So do you want to write one CLDF dataset or two?

[LinguList]

I want the same as allenbai. We have two datasets and share the languagetable. All in one folder.

[LinguList]

Generated by one script.

[LinguList]

I think, lexibank/allenbai is similar to liliegrenhindukush in this regard

[LinguList]

Just could not check the code before as I was checking from mobile. But I do not know if these are two or one dataset. But important is the way the lexibank script works with two writers.

[xrotwang]

Yes, ok, that's what I meant, too. Here, they might even share the ParameterTable, right?

[LinguList]

Hmmm. For me, parameters are colexifications in the structure dataset.

[xrotwang]

Here's the two datasets written in the liljegrenhindukush case:
https://github.com/cldf-datasets/liljegrenhindukush/blob/8f2e22a2d2629a6158e2d3959165504166a8b61e/lexibank_liljegrenhindukush.py#L73-L82

[LinguList]

They are built from the concepts, but not the same, right?

[xrotwang]

Hmmm. For me, parameters are colexifications in the structure dataset.

Ah, ok. Yes that makes more sense.