mmap() dumps core upon resizing the underlying file

[VladimirUshakov]

BPO	16212
Nosy	@jcea, @pitrou, @tiran

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-10-12.14:04:23.961>
labels = ['3.10', '3.8', 'expert-IO', 'type-crash', '3.9']
title = 'mmap() dumps core upon resizing the underlying file'
updated_at = <Date 2020-11-06.16:38:53.124>
user = 'https://bugs.python.org/VladimirUshakov'

bugs.python.org fields:

activity = <Date 2020-11-06.16:38:53.124>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IO']
creation = <Date 2012-10-12.14:04:23.961>
creator = 'Vladimir.Ushakov'
dependencies = []
files = []
hgrepos = []
issue_num = 16212
keywords = []
message_count = 21.0
messages = ['172745', '172746', '172759', '172760', '172761', '172765', '172782', '172783', '172785', '172826', '172835', '172837', '172842', '172857', '172859', '172860', '172862', '172864', '172866', '172889', '172892']
nosy_count = 5.0
nosy_names = ['jcea', 'pitrou', 'christian.heimes', 'neologix', 'Vladimir.Ushakov']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue16212'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[VladimirUshakov]

The following code crashes the interpreter on Linux:

#!/usr/bin/python3

import mmap

with open('test', 'wb') as f:
    f.write(bytes(1))

with open('test', 'r+b') as f:
    m = mmap.mmap(f.fileno(), 0)
    f.truncate()
    a = m[:]

---

It's not specific to the zero size truncation, it's enough if the file size decreases beyond a page border.

[tiran]

I'm able to reproduce the bug. Here is a stack trace:

#0 0x00000000005a650d in PyBytes_FromStringAndSize (str=0x7f44c127a000 <Address 0x7f44c127a000 out of bounds>, size=1)
at Objects/bytesobject.c:82
82 (op = characters[*str & UCHAR_MAX]) != NULL)
(gdb) bt
#0 0x00000000005a650d in PyBytes_FromStringAndSize (str=0x7f44c127a000 <Address 0x7f44c127a000 out of bounds>, size=1)
at Objects/bytesobject.c:82
#1 0x00007f44bf7c11b2 in mmap_subscript (self=0x7f44bf9ca350, item=<slice at remote 0x7f44c1168618>)
at /home/heimes/dev/python/cpython/Modules/mmapmodule.c:799
#2 0x0000000000590863 in PyObject_GetItem (o=<mmap.mmap at remote 0x7f44bf9ca350>, key=<slice at remote 0x7f44c1168618>)
at Objects/abstract.c:143

[neologix]

That's normal.
You're truncating the file after having it mapped, so touching the pages corresponding to the truncated part of the file will result in a segfault.
See mmap's man page:
"""
Use of a mapped region can result in these signals:

   SIGBUS Attempted access to a portion of the buffer that does not correspond to
          the file (for example, beyond the end of the file, including the case
          where another process has truncated the file).

"""

The only way to guard against it would be to call fstat() in every buffer-protocol method in case of file-backed mapping to check against the current file size, but that would be awfully slow in case of sequential access (just imagine performing an md5sum of an mmap-ed file).

[tiran]

The fstat() check isn't bullet proof, too. It has a race condition as another process could truncate the file between the fstat() check and the code lines that access the mmapped file.

[neologix]

No, it's not.
That's why I think there's nothing that can be done.

[VladimirUshakov]

I think, handling the signal would do.

[neologix]

I think, handling the signal would do.

You can't.
Handling a signal like SIGSEGV or SIGBUS in any other way that exiting
the process will result in an infinite loop (as soon as the signal
handler returns the faulty instruction will be re-executed). Well,
there are some higly non-portable ways to try to escape this (see
http://stackoverflow.com/questions/2663456/write-a-signal-handler-to-catch-sigsegv),
but it won't fly in our case.
You may want to have a look at the faulthandler module for "graceful"
shutdown, but nothing can't be done.

[vstinner]

"Well, there are some higly non-portable ways to try to escape this (see
http://stackoverflow.com/questions/2663456/write-a-signal-handler-to-catch-sigsegv), but it won't fly in our case."

There is also the libsigsegv library, but it's hard and unsafe to handle such low level signals in Python.
http://www.gnu.org/software/libsigsegv/

"It has a race condition as another process could truncate the file between the fstat() check and the code lines that access the mmapped file."

You can use file locks to be protected against such race condition. See for example:
http://pypi.python.org/pypi/lockfile

[neologix]

You can use file locks to be protected against such race condition.
See for example:

Those are advisory locks, not mandatory locks.
It's possible to do some mandatory locking on some systems, but on every file, and it's not portable.

[VladimirUshakov]

I know how to avoid the problem in my case, the bug does not really affect me. I posted it because I thought that the possibility to crash the interpreter is something to be avoided at all costs.

I've found a few examples of handling non-restartable signals with longjmps, but not that familiar with the codebase to estimate how reliably it can be done on all supported platforms. I don't really know how this code would behave, say, on Windows.

I'm gonna do some research on the topic when I have a little more time.

[neologix]

I know how to avoid the problem in my case, the bug does not really affect me. I posted it because I thought that the possibility to crash the interpreter is something to be avoided at all costs.

I fully agree with you.
However, that's a problem inherent to mmap(), and I don't think
there's a way to avoid this, but I could be wrong.

I've found a few examples of handling non-restartable signals with longjmps, but not that familiar with the codebase to estimate how reliably it can be done on all supported platforms. I don't really know how this code would behave, say, on Windows.

You can't use longjmp from signal handlers. Well, you can, but 99% of
the code that does it is broken, because you can only call async-safe
functions from within a signal handler, and certainly can't run the
intepreter.

[pitrou]

A pity Posix isn't smart enough to refuse truncate()ing when there's a mmap open on the affected pages. Python 3's buffer API is superior in that respect :-)

[VladimirUshakov]

You can't use longjmp from signal handlers. Well, you can, but 99% of the code that does it is broken, because you can only call async-safe functions from within a signal handler, and certainly can't run the intepreter.

I don't see the reason to run the interpreter. But a quick look at the source shows that the current implementation already uses longjmps to handle SIGFPE (see pyfpe.h and fpectlmodule.c). It seems to be in use on many platforms.

The only problem I see so far is the possibility that we have to protect too much. However, as I could guess from quickly browsing through the mmap() implementation, the address of the mmap()ed region never leaves the module, all accesses are done using exported methods. If it is really the case, we can wrap them into something similar to PyFPE_START_PROTECT() and PyFPE_END_PROTECT() (see comments in pyfpe.h).