Memoize atom so that a STRING+ node only gets parsed once

[lysnikolaou]

I think it makes sense to memoize atom, but if we don't want to do so, we can extract STRING+ into a new rule and memoize that.

Closes #64.

[gvanrossum]

We should really have a script that automatically reproduces the benchmarks I posted to the PEP, so we can compare the effect of a proposed change.

[lysnikolaou]

We should really have a script that automatically reproduces the benchmarks I posted to the PEP, so we can compare the effect of a proposed change.

I could quickly write up a script that does that, but the workflow you used to get those stats is not clear to me.

[gvanrossum]

I could quickly write up a script that does that, but the workflow you used to get those stats is not clear to me.

I timed things in the old "pegen" repo.

• The numbers for the canonical test file were done using the "make time_compile" and "make time_stdlib_compile" targets, running each three times and taking the middle outcome.

• The numbers for compiling the stdlib were done using the "make bench" command. This runs "make simpy_cpython" three times, I just used the middle outcome. Note that it will print memory stats if you pip install psutil. In order to get numbers for compilation to bytecode I changed the test_parse_directory.py script to set mode=2. I then did another run with the parse.parse_file() call replaced by compile(open(file).read(), file, "exec") (IIRC) to get numbers for the old compiler with the same input set.

I would rerun things in the Tools/peg_generator subtree here, but currently the "make build" target fails for me there.

[gvanrossum]

Yeah!