Add structure from existing alignment

[FredericBlum]

I am currently running into trouble while trying to create a CV structure based on a pre-existing alignments with manually trimmed data. Let's use for example the following data:

ID	DOCULECT	CONCEPT	VALUE	FORM	TOKEN	COGID	ALIGNMENT
1	Marubo	type of bank	kɨnã	kɨnã	k ɨ n ã	60	['k', 'ɨ', 'n', 'ã', '-', '(', '-', '-', ')']
2	Chakobo	type of bank	kɨˈnanɨ	kɨˈnanɨ	k ɨ n a n ɨ	60	['k', 'ɨ', 'n', 'a', '-', '(', 'n', 'ɨ', ')']

Is there any pre-defined way of doing this, or would I need to create a new tokens-column based on the alignments, removing all content within brackets in a dynamic way? @LinguList Maybe you can help me on this.

[LinguList]

I think you need to make the immediate strings that are all trimmed.

[LinguList]

But in order to make them, you can iterate over the alignment:

for cogid, msa in alms.msa["cogid"].items():
    print(msa["alignment"])
    print(msa)

The msa-object that is generated for you here stores the alignments, I think it even automatically only retains those that are not inside brackets!

[LinguList]

But I'd ask you to test for this.

[LinguList]

Please check also this function, that could otherwise be applied to the alignment object: https://lingpy.org/reference/lingpy.align.html#lingpy.align.sca.Alignments.reduce_alignments

[LinguList]

So, please check alms.reduce_alignments and how it behaves, as I do not remember if it returns a list of reduced alignments or if it reduces alignments in the alms.msa["cogid"] attribute, etc.

[FredericBlum]

Not much happens. It applies directly to alms and adds a _alignment entry in the Dictionary, but nothing is reduced. Even after looking at the code, I do not understand where this would proceed.

[LinguList]

Please use a minimal example and share it here (you can zip it). WE only need 3 cogids with 3 alignments that are "to be reduced".

[LinguList]

And please check the import statement in the script on align/sca.py in lingpy:

from lingpy.read.qlc import read_msa, normalize_alignment, reduce_alignment

So you have the function in read.qlc!

[LinguList]

def reduce_alignment(alignment):
    """
    Function reduces a given alignment.
    
    Notes
    -----
    Reduction here means that the output alignment consists only of those parts
    which have not been marked to be ignored by the user (parts in brackets).
    It requires that all data is properly coded. If reduction fails, this will
    throw a warning, and all brackets are simply removed in the output
    alignment.
    """

    # check for bracket indices in all columns
    cols = misc.transpose(alignment)

    ignore_indices = []
    ignore = False
    for i, col in enumerate(cols):
        reduced_col = sorted(set(col))

        if '(' in reduced_col:
            if len(reduced_col) == 1:
                ignore_indices += [i]
                ignore = True
            else:
                ignore = False
        elif ')' in reduced_col:
            if len(reduced_col) == 1:
                ignore_indices += [i]
                ignore = False
            else:
                ignore_indices = []
        elif ignore:
            ignore_indices += [i]

    if ignore_indices:
        new_cols = []
        for i, col in enumerate(cols):
            if i not in ignore_indices:
                new_cols += [col]
    else:
        new_cols = cols

    new_alm = misc.transpose(new_cols)

    for i, alm in enumerate(new_alm):
        for j, char in enumerate(alm):
            if char in '()':
                new_alm[i][j] = '-'

    return new_alm

[LinguList]

The alignment here is a simple array, so it is like the msa["alignment"] what you can pass it. This should reduce the alignment, and you can from the reduced alignment then store tokesn [x for x in alm if x != "-"] for each reduced alignment as well.

It is probably easier to integrate all of this later into the Sites class in lingrex, but it is not that difficult to make a small preprocessing with pure lingpy here.

[LinguList]

from lingpy.read.qlc import reduce_alignment
from lingpy import basictypes

dct = {}
for idx, msa in alms.msa["cogid"].items():
    reduced = reduce_alignment(msa["alignment"])
    for i, row in enumerate(reduced):
        dct[msa["ID"][i] = row

alms.add_entries("old_tokens", "tokens", lambda x: x)
alms.add_entries("tokens", dct, lambda x: basictypes.lists([y for y in x if y != "-"]), override=True)

# same for alignemnts

[FredericBlum]

minimum_example.zip

Here comes the minimum example based on your code, showcasing the problem. There is no reduction apparently.

[LinguList]

You DO realize that you have python lists typed into the alignment column, not space-segmented strings?

[LinguList]

['m', 'ã', 'n', '(', '-', ')', 'ã', '-', '(', '-', '-', '-', '-', ')']

[LinguList]

That is the first alignment ;-)