build fails when building with PERL_NO_COW

[bram-perl]

Noticed when attempting to build with -Accflags='-DPERL_NO_COW'

    $ ./Configure -des -Dusedevel -Accflags='-DPERL_NO_COW'
    $ make -j8 miniperl
    ...
    ./miniperl -Ilib -f write_buildcustomize.pl
    makefile:363: recipe for target 'lib/buildcustomize.pl' failed
    make: *** [lib/buildcustomize.pl] Segmentation fault (core dumped)

Rebuilding with -DDEBUGGING shows an assertion failure:

    $ ./Configure -des -Dusedevel -Accflags='-DPERL_NO_COW' -DDEBUGGING
    $ make -j miniperl
    ...
    ./miniperl -Ilib -f write_buildcustomize.pl
    miniperl: hv.c:2972: S_unshare_hek_or_pvn: Assertion `he->shared_he_he.hent_hek == hek' failed.
    makefile:363: recipe for target 'lib/buildcustomize.pl' failed

Reducing to a minimal test case:

    $ ./miniperl -wle 'my $x = (0 == 1);'
    miniperl: hv.c:2972: S_unshare_hek_or_pvn: Assertion `he->shared_he_he.hent_hek == hek' failed.
    Aborted (core dumped)

Bisecting points to commit 914bb57: (@leonerd)

    commit 914bb57489325d34ddbb7c0557c53df7baa84d86
    Author: Paul "LeoNerd" Evans <leonerd@leonerd.org.uk>
    Date:   Sat Aug 7 14:46:48 2021 +0100

        Define a third kind of COW state; STATIC

        Previously, when IsCOW flag was set there were two cases:
          SvLEN()==0:
              PV is really a shared HEK

          SvLEN()!=0:
              PV is a COW structure with 1..256 refcount stored in its extra final byte

        This change adds a third state:

          SvLEN()==0 && SvFLAGS() & SVppv_STATIC:
              PV is a shared static const pointer and must not be modified

        sv_setsv_flags() and sv_setsv_cow() will preserve this state

        sv_uncow() will copy it out to a regular string buffer

        sv_dup() will preserve the static pointer into cloned threads

(Note: I have no use case for -DPERL_NO_COW, I only tried it while looking at another issue)

Steps to reproduce

1. ./Configure -des -Dusedevel -Accflags='-DPERL_NO_COW' -DDEBUGGING
2. make -j8 miniperl
3. ./miniperl -wle 'my $x = (0 == 1);'

Actual result

Segmentation fault/Assertion failure:

S_unshare_hek_or_pvn: Assertion `he->shared_he_he.hent_hek == hek' failed.

Expected result

No crash.

[jkeenan]

Noticed when attempting to build with -Accflags='-DPERL_NO_COW'

    $ ./Configure -des -Dusedevel -Accflags='-DPERL_NO_COW'
    $ make -j8 miniperl
    ...
    ./miniperl -Ilib -f write_buildcustomize.pl
    makefile:363: recipe for target 'lib/buildcustomize.pl' failed
    make: *** [lib/buildcustomize.pl] Segmentation fault (core dumped)

Observed on FreeBSD as well.

[iabyn]

On Sat, Jul 23, 2022 at 01:16:13PM -0700, Bram wrote: Noticed when attempting to build with `-Accflags='-DPERL_NO_COW'` $ ./Configure -des -Dusedevel -Accflags='-DPERL_NO_COW' $ make -j8 miniperl ... ./miniperl -Ilib -f write_buildcustomize.pl makefile:363: recipe for target 'lib/buildcustomize.pl' failed make: *** [lib/buildcustomize.pl] Segmentation fault (core dumped)

Given that it's been broken for a year and you're the first to notice, I wonder whether the time has come to remove the facility to build a non-COW perl? It's been on by default for 8 years now.

…

-- All wight. I will give you one more chance. This time, I want to hear no Wubens. No Weginalds. No Wudolf the wed-nosed weindeers. -- Life of Brian

[bram-perl]

It's broken in a dev release since v5.35.4 (2021-Sep-20)
In a non-dev release it's been broken since v5.36.0 (2022-05-27)

Anyway, briefly discussing this with @leonerd :

• Boolean need copy on write
• firstclass bool values are cow-static strings

The question is if someone still has a use-case for building without Copy on Write.. (I do not have one)
If someone does have then it should show up when they attempt to build perl v5.36..
If nothing shows up then "disabling CoW" could be ripped out..

[nielsl]

I have delayed upgrading to newer perl because they fail when PERL_NO_COW is on. But that too is now failing, because older perls have other compilation errors on the latest Linux systems. COW does not seem to extend to C functions: when passing a string reference to a C function that updates it (as a byte array) then the cloning does not happen it seems. If this feature disappears I will have to do explicit copying in many places. I think this compilation problem should be fixed (same as above, except on Linux Mint 21.1) rather than shipping for a long time something that doesn't work. Whether the feature is useful or not, thats a separate discussion. I think it is. So my big software package is stuck between one compilation error or another. I will even pay reasonably to have it fixed. (Btw, the switch to Github instead of email means shifting convenience from users to developers, maybe not a good idea, at least I always try to do the opposite with my code).

[demerphq]

@nielsl I suspect you just need to change your code a bit so that SvPV() are changed to be one of the following:

            char*  SvPV_force              (SV* sv, STRLEN len)
            char*  SvPV_force_nolen        (SV* sv)
            char*  SvPVx_force             (SV* sv, STRLEN len)
            char*  SvPV_force_nomg         (SV* sv, STRLEN len)
            char*  SvPV_force_nomg_nolen   (SV * sv)
            char*  SvPV_force_mutable      (SV * sv, STRLEN len)
            char*  SvPV_force_flags        (SV * sv, STRLEN len, U32 flags)
            char*  SvPV_force_flags_nolen  (SV * sv, U32 flags)
            char*  SvPV_force_flags_mutable(SV * sv, STRLEN len, U32 flags)
            char*  SvPVbyte_force          (SV* sv, STRLEN len)
            char*  SvPVbytex_force         (SV* sv, STRLEN len)
            char*  SvPVutf8_force          (SV* sv, STRLEN len)
            char*  SvPVutf8x_force         (SV* sv, STRLEN len)

You can also use the function sv_force_normal() as well. See perldoc perlapi for details.

COW does not refer to the Unix COW facility, it is purely an internal feature.

I would be happy to discuss more privately, you can find my email on the list.

Having said that, I do think that we should make PERL_NO_COW work. Nothing we do should depend on COW. That would be/is a very sad state of affairs, as it would mean we have a hard dependency on some particular implementation of COW, and IMO we should be able to change how COW works without causing havok. The whole scheme of putting the COW bytes at the end of the PV buffer, and various other aspects of its implementation definitely could be improved, as it leaves cases that can't be COW'ed and causes other problems (there is some kind of hairy interaction with file operations and regexes that we never really fixed afair), and if we have somehow hard coded a dependency on COW then IMO we should fix it.

[demerphq]

@bram-perl said:

Anyway, briefly discussing this with @leonerd :

• Boolean need copy on write
• firstclass bool values are cow-static strings

That doesn't make sense to me, it feels like an unforced issue that we can resolve if we choose to. I will add this to my todo list.

[leonerd]

That doesn't make sense to me, it feels like an unforced issue that we can resolve if we choose to. I will add this to my todo list.

Yah; firstclass-bools don't actually rely on COW, they just happen to exploit the fact that if they set the COW flag, then the PV buffer is shared by plain pointer address, and so we can use that pointer to distinguish if it is the special boolean true/false value. You could arrange for that pointer to be statically-shared by some other mechanism if COW was not enabled.

[nielsl]

@demerphq Thank you for giving me a way out at the C level. I am not very good with Perl/C, but managed to create "pidgin-C" functions in all the places where there is a memory or speed bottleneck, and where the code doesn't change.

[nielsl]

@bram-perl said:

Anyway, briefly discussing this with @leonerd :

• Boolean need copy on write
• firstclass bool values are cow-static strings

That doesn't make sense to me, it feels like an unforced issue that we can resolve if we choose to. I will add this to my todo list.

Oh that would be wonderful, thanks. I can then update to the latest Perl and to all the non-standard Perl and GNU packages that I depend on. Then I can install my system on the latest Linux and Unix systems again. And if the fix is not immediate, I can now make temporary workarounds knowing that there will be a fix. Bravo.

[nielsl]

@bram-perl said:

Anyway, briefly discussing this with @leonerd :

• Boolean need copy on write
• firstclass bool values are cow-static strings

That doesn't make sense to me, it feels like an unforced issue that we can resolve if we choose to. I will add this to my todo list.

Is this done by now?

[nielsl]

@bram-perl said:

Anyway, briefly discussing this with @leonerd :

• Boolean need copy on write
• firstclass bool values are cow-static strings

That doesn't make sense to me, it feels like an unforced issue that we can resolve if we choose to. I will add this to my todo list.

I can answer my own question: in 5.38.2, compiling with -DPERL_NO_COW still crashes on the most commonly used Linux systems. Attached are the configure and make outputs from 5.38.2 and 5.26.1, and the hardware configuration. Below the gcc version and system software.

I am using 5.26.1 with a large amount of perl modules that I wrote for a DNA analysis package. Customers use it and I cannot build it. I could upgrade to the latest stable, and I will, but it means updating CPAN modules and testing them in combination for breakage (no, I cannot just always take the latest, something will break). As was suggested above, I could change C-function calls throughout, but doing and testing that would take me 1-2 weeks of studying and repair probably (well I have no idea really). I hoped it would be fixed by now. I don't mind paying in advance if this will be fixed quickly, since customers will have to postpone otherwise. Do you see it as an easy fix, or is it difficult? can the fix be back-ported to 5.26.1? is there a fix already in development versions? Contact info below.

5.38.2-error.txt
5.26.1-error.txt
hw-short.txt

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=21.3
DISTRIB_CODENAME=virginia
DISTRIB_DESCRIPTION="Linux Mint 21.3 Virginia"

Niels Larsen, PhD, CEO
Danish Genome Institute

E-mail: niels@genomics.dk
Skype: niels_larsen_denmark
Mobile: +45-3091-5426 (GMT+1)