[doc] clarify that except does not match virtual subclasses of the specified exception type

[acooke]

BPO	12029
Nosy	@gvanrossum, @warsaw, @ncoghlan, @pitrou, @benjaminp, @jwilk, @merwok, @alex, @bitdancer, @Trundle, @durban, @ericsnowcurrently, @berkerpeksag, @vadmium, @JimJJewett, @serhiy-storchaka, @charettes, @MojoVampire, @mbmccoy, @iritkatriel
PRs	bpo-12029: Exception handling should match subclasses #6461 bpo-12029: [doc] clarify that except does not match virtual subclasses of the specified exception type #32027 [3.10] bpo-12029: [doc] clarify that except does not match virtual subclasses of the specified exception type (GH-32027) #32034 [3.9] bpo-12029: [doc] clarify that except does not match virtual subclasses of the specified exception type (GH-32027) #32035
Dependencies	bpo-22540: speed up isinstance and issubclass for the usual cases
Files	issue12029.patch exception_proper_subclass_matching_v2.patch exception_proper_subclass_matching_v3.patch pyobject_issubclass_isinstance_speedup.patch exception_proper_subclass_matching_v4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-21.21:28:15.751>
created_at = <Date 2011-05-08.08:53:43.241>
labels = ['interpreter-core', '3.9', '3.10', '3.11', 'type-feature', 'docs']
title = '[doc] clarify that except does not match virtual subclasses of the specified exception type'
updated_at = <Date 2022-03-21.21:28:16.619>
user = 'https://bugs.python.org/acooke'

bugs.python.org fields:

activity = <Date 2022-03-21.21:28:16.619>
actor = 'iritkatriel'
assignee = 'docs@python'
closed = True
closed_date = <Date 2022-03-21.21:28:15.751>
closer = 'iritkatriel'
components = ['Documentation', 'Interpreter Core']
creation = <Date 2011-05-08.08:53:43.241>
creator = 'acooke'
dependencies = ['22540']
files = ['25548', '36777', '36778', '36779', '36780']
hgrepos = []
issue_num = 12029
keywords = ['patch']
message_count = 51.0
messages = ['135521', '136712', '136717', '160399', '160418', '160426', '160427', '160428', '160430', '160432', '160435', '160436', '160461', '160468', '161473', '200418', '200420', '200421', '200829', '201329', '202019', '205830', '205880', '228158', '228159', '228171', '228195', '228196', '228198', '228209', '228213', '228214', '228215', '228216', '228219', '234434', '253270', '253272', '253288', '253958', '273436', '315257', '315267', '315268', '412778', '412798', '412799', '412801', '415700', '415705', '415707']
nosy_count = 28.0
nosy_names = ['gvanrossum', 'barry', 'jamesh', 'ncoghlan', 'pitrou', 'fijal', 'benjamin.peterson', 'jwilk', 'eric.araujo', 'alex', 'r.david.murray', 'Trundle', 'cvrebert', 'daniel.urban', 'yorik.sar', 'docs@python', 'Yury.Selivanov', 'eric.snow', 'berker.peksag', 'martin.panter', 'Jim.Jewett', 'serhiy.storchaka', 'gcbirzan', 'charettes', 'josh.r', 'V\xc3\xa1clav Dvo\xc5\x99\xc3\xa1k', 'Michael McCoy', 'iritkatriel']
pr_nums = ['6461', '32027', '32034', '32035']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12029'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[acooke]

Hi,

In general, registering a class with an ABC is equivalent to making it a subclass (isinstance and issubclass are patched through ABCMeta). However, this does not work for exceptions (see example below, where exception is not caught).

This doesn't seem terribly surprising to me - I imagine that checking would slow down exception handling - but I couldn't find any documentation (and posting on c.l.p didn't turn up anything either).

So I thought I would raise it here - perhaps there is a possible fix (my obscure use case is that I have a backtracking search; backtracking occurs when a certain exception is encountered; making that exception an ABC and allowing existing exceptions to be registered with it allows the search to work with existing code without a wrapper that catches and translates exceptions that should trigger a backtrack). Or perhaps the docs could be extended. Or perhaps I've misunderstood something...

Cheers,
Andrew

Python 3.2 (r32:88445, Feb 27 2011, 13:00:05) 
[GCC 4.5.0 20100604 [gcc-4_5-branch revision 160292]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from abc import ABCMeta
>>> class RootException(Exception,metaclass=ABCMeta): pass
... 
>>> class MyException(Exception): pass
... 
>>> RootException.register(MyException)
>>> try:
...     raise MyException
... except RootException:
...     print('caught')
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
__main__.MyException

[cvrebert]

Scouting around the CPython codebase a bit, I speculate that the cause of this behavior is that PyErr_GivenExceptionMatches() in errors.c uses PyType_IsSubtype() [which simply walks a class's __mro__ checking for pointer equality] rather than PyObject_IsSubclass()/PyObject_IsInstance() [which are smart enough to consult __subclasscheck__()/instancecheck() if they exist].

Of course, the more important issue here is whether this behavior is intended or not. I surmise python-dev needs to have a discussion about it?

[cvrebert]

Surveying the docs, the current behavior *is* /technically/ correct (in a suspiciously precise way) according to the Language Reference:
http://docs.python.org/dev/reference/compound_stmts.html#grammar-token-try_stmt :
"For an except clause with an expression [...] the clause matches the exception if the resulting object is 'compatible' with the exception. An object is compatible with an exception if it is the class or a base class of the exception object" (which exactly describes what PyType_IsSubtype() checks for)

The Tutorial is by contrast much more vague:
http://docs.python.org/dev/tutorial/errors.html#handling-exceptions :
"if [the raised exception's] type matches the exception named after the except keyword, the except clause is executed, and then execution continues after the try statement."
No definition of what it means for the types to "match" seems to be given.

[jamesh]

The documentation for ABCMeta.register() says that it makes the other class a "virtual subclass". That would make the ABC a "virtual base class".

So whether the current behaviour is correct depends on whether you consider a "virtual base" to count as a base. From the reasoning behind the introduction of ABCs, it certainly sounds like it should count.

Also, this is a feature that works correctly in Pyton 2.7, so could trip people up who are trying to move to Python 3.

[gvanrossum]

I agree it's a bug and should be fixed. It's too confusing that there would be two slightly different interpretations of isinstance/issubclass where the isinstance() and issubclass() would be using the extended interpretation but the except clause would use the narrow interpretation.

The exception matching done by the except clause ought to be explainable in terms of issubclass/isinstance.

[benjaminp]

I think being able to catch exception with ABCs is esssentially useless. The originally stated "usecase" can be simply solved by putting classes into a tuple and putting that in the except clause.

In general, the whole abc machinary causes lots of code which expects instance and subclass checks to be side-effect free to be able to execute arbitrary code, which creates messes.

[pitrou]

Perhaps ABCMeta could raise a UserWarning when creating an Exception subclass?

[acooke]

perhaps it could just work in a simple, consistent way?

in my original report i wondered whether there was a significant performance hit. but so far the objections against fixing this seem to be (1) a lawyer could be convinced the current behaviour is consistent with the docs (2) python 3 should remain compatible with python 2 (3) abcmeta is the sucksorz.

those don't seem like great arguments against making it just work right, to me.

[pitrou]

perhaps it could just work in a simple, consistent way?

That would be best obviously. But as Benjamin explained it's quite delicate to make it work while avoiding pitfalls where code involved in exception checking may itself fail with arbitrary errors - say, enter an infinite recursion. It's also why I think it would be a bad idea to fix it in 3.2 (the bugfix branch). In 3.3 we can take riskier decisions.

[benjaminp]

Basically, someone needs to produce a patch and we can go from there.