Split compiler into code-gen, optimizer and assembler.

[markshannon]

BPO	42926
Nosy	@gvanrossum, @markshannon, @corona10, @brandtbucher, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-01-13.15:03:23.987>
labels = []
title = 'Split compiler into code-gen, optimizer and assembler.'
updated_at = <Date 2022-02-03.22:11:26.525>
user = 'https://github.com/markshannon'

bugs.python.org fields:

activity = <Date 2022-02-03.22:11:26.525>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-01-13.15:03:23.987>
creator = 'Mark.Shannon'
dependencies = []
files = []
hgrepos = []
issue_num = 42926
keywords = ['patch']
message_count = 2.0
messages = ['385033', '385043']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'Mark.Shannon', 'corona10', 'brandtbucher', 'iritkatriel']
pr_nums = ['31116']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue42926'
versions = []

• PR: gh-87092: do not allocate PyFutureFeatures dynamically #98913

• PR: gh-87092: remove unused SET_LOC/UNSET_LOC macros #98914

• PR: gh-87092: expose the compiler's codegen to python for unit tests #99111

• PR: gh-87092: move all localsplus preparation into separate function called from assembler stage #99869

• PR: gh-87092: Make jump target label equal to the offset of the target in the instructions sequence #102093

• PR: gh-87092: compiler's CFG construction moved to after codegen stage #102320

• PR: gh-87092: refactor assemble() to a number of separate functions, which do not need the compiler struct #102562

• PR: gh-87092: create new opcode_utils.h file and move macros there, so they can be shared #102926

• PR: gh-87092: move CFG related code from compile.c to flowgraph.c #103021

• PR: gh-87092: move assembler related code from compile.c to assemble.c #103277

• PR: gh-87092: fix refleak in peepholer test harness #103448

• PR: gh-87092: fix a few cases of incorrect error handling in compiler #103456

• PR: gh-87092: change assembler to use instruction sequence instead of CFG #103933

• PR: gh-87092: update CODEOWNERS for split of compile.c to 3 files #103941

• PR: gh-87092: Expose assembler to unit tests #103988

• PR: gh-87092: avoid gcc warning on uninitialized struct field in assemble.c #104460

• PR: gh-87092: avoid gcc warning on uninitialized struct field in assemble… #105243

• PR: gh-87092: remove unused macros #105247

[markshannon]

Currently the compiler operates in three main passes:

Code-gen
Optimize
Assemble

The problem is that these passes use the same basic-block based CFG, leading to unnecessary coupling and inefficiencies.
A basic block CFG is awkward and error-prone for the code-gen, but not very efficient for the optimizer and assembler.

A better design would be for the code-gen to create a single linear sequence of instructions. The optimizer would take this and produce a list of extended-blocks for the assembler to consume.

code-gen -> (list of instructions) -> optimizer
optimizer -> (list of extended blocks) -> assembler

(Extended blocks have a single entry and multiple exits, unlike basic blocks which have a single entry and single exit)

This would:

1. Reduce memory use considerably (the size of instruction and block data structures would be about 60% of current)
2. Be faster (Less CFG management).
3. Produce better code (extended blocks are a better unit for optimization that basic blocks).
4. Be easier to maintain:
   a) Code-gen wouldn't have to worry about creating a correct CFG.
   b) The optimizer wouldn't need to handle empty blocks and track which basic blocks form an extended block.

Apart from the changes to the compiler, it would help if we made all branch instructions absolute (or have a backward dual) to accommodate free reordering of blocks in the optimizer.

[gvanrossum]

SGTM. But I’m not the one who has to work with it.