Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: Comptime functions #727

Open
wants to merge 70 commits into
base: main
Choose a base branch
from
Open

feat!: Comptime functions #727

wants to merge 70 commits into from

Conversation

mark-koch
Copy link
Collaborator

@mark-koch mark-koch commented Dec 17, 2024

Adds a new guppy.comptime decorator for functions that construct Hugr via tracing of the Python interpreter.

The most important bits are in the new guppylang.tracing module:

  • state.py defines the shared state that is maintained during tracing
  • function.py contains the main entry point for tracing of a function: trace_function. It also contains the logic for handling function calls during tracing: trace_call
  • object.py defines the new GuppyObject class. This is the abstract representation of values that are not statically known (e.g. function inputs, outputs of called functions, etc.). It also contains the GuppyDefinition class which is the new object that is returned to users when they decorate a function.
  • unpacking.py defines the logic for turning GuppyObjects representing structured types into their Python equivalent and vice versa. For example, Guppy tuples are turned into Python tuples, Guppy arrays are turned into Python lists.

Some more esoteric things:

  • builtins_mock.py contains some metaclass hacks to mock builtin functions int, float, and len to accept GuppyObjects
  • frozenlist.py is an implementation of immutable lists. They are needed to prevent users from mutating function inputs to preserve Python semantics.
  • util.py defines the logic for dealing with exceptions and modifying tracebacks to hide internal compiler frames from the user

Other changes:

  • Updated the execute_llvm Rust library to support passing function inputs. This is to make testing of arithmetic functions easier.
  • I found a nicer way to handle error tests with pytest, so the the compiler no longer needs to check whether pytest is running. Now, we're just calling sys.excepthook on the caught exception in the test (see tests.error.util.run_error_test)
  • Tracebacks in Python 3.11+ look different to Python 3.10. To avoid duplicating the golden test files, I'm doing some regex matching in tests.error.util.run_error_test to ignore those differences

Follow-ups:

BREAKING CHANGE: Guppy decorators now return instances of GuppyDefinition

Closes #751

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch
@mark-koch mark-koch mentioned this pull request Dec 17, 2024

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch
@codecov-commenter
Copy link

codecov-commenter commented Jan 6, 2025

Codecov Report

Attention: Patch coverage is 93.36609% with 54 lines in your changes missing coverage. Please review.

Project coverage is 93.12%. Comparing base (d9fc9fd) to head (512b9ee).

Files with missing lines Patch % Lines
guppylang/tracing/object.py 92.52% 24 Missing ⚠️
guppylang/tracing/frozenlist.py 70.96% 9 Missing ⚠️
guppylang/tracing/unpacking.py 92.85% 8 Missing ⚠️
guppylang/definition/traced.py 92.53% 5 Missing ⚠️
guppylang/error.py 20.00% 4 Missing ⚠️
guppylang/tracing/util.py 94.23% 3 Missing ⚠️
guppylang/std/_internal/compiler/array.py 93.75% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main     #727    +/-   ##
========================================
  Coverage   93.11%   93.12%            
========================================
  Files          74       82     +8     
  Lines        8632     9393   +761     
========================================
+ Hits         8038     8747   +709     
- Misses        594      646    +52     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch
@qartik qartik linked an issue Feb 4, 2025 that may be closed by this pull request
Comment on lines 13 to 14
Previous use occurred in
linear_copy6.py:15
Copy link
Collaborator Author

@mark-koch mark-koch Feb 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line-break is a bug in diagnostics rendering. See #818

@mark-koch mark-koch changed the title feat: Tracing prototype feat: Comptime functions Feb 25, 2025
@mark-koch mark-koch changed the title feat: Comptime functions feat!: Comptime functions Feb 25, 2025
@mark-koch mark-koch marked this pull request as ready for review February 27, 2025 10:36
@mark-koch mark-koch requested a review from a team as a code owner February 27, 2025 10:36
@mark-koch mark-koch requested a review from croyzor February 27, 2025 10:36
@ss2165 ss2165 self-requested a review March 6, 2025 11:23
Copy link
Member

@ss2165 ss2165 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review part 1, excluding tests

I suggest you expand the compiler docs to include discussion of comptime as part of this PR, while it is still fresh

# Type error in the return statement. For example, this happens if users
# try to return a struct with invalid field values
raise GuppyError(TracingReturnTypeError(node, str(err))) from None
except ValueError as err:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused as to why these are raising standard Python errors rather than guppy internal ones

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Errors from guppy_object_from_py can also surface to the user so they are not internal. We just catch them here since they ocurr while processing the function return, so we don't have a useful traceback

If you prefer, we could create a new user-facing GuppyComptimeError instead of using Python's builtin ValueError and TypeError?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes would prefer that

using `guppy_object_from_py`, updates the wires of any `GuppyObjects` contained in
`v` to the new wires specified by `obj`.

Also resets the used flag on any of those updated wires.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This corresponds to making a value available again after a borrow expires. I'll add this to the docstring

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this done? I couldn't find it

Comment on lines 86 to 90
class array(Generic[_T, _n]):
class array(list[_T], Generic[_T, _n]):
"""Class to import in order to use arrays."""

def __init__(self, *args: Any):
pass
list.__init__(self, args)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

include this as a comment in the code

from guppylang.module import GuppyModule

import guppylang.decorator as decorator


# Regular expression to match the `~~~~~^^^~~~` highlights that are printed in
# tracebacks from Python 3.11 onwards. We strip those out so we can use the same golden
# files for Python 3.10
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this feels super flaky

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have another preferred solution? We could store multiple .err files for each Python version, but then each we make will need to run pytest --snapshot-update for all versions which would be annoying

Copy link
Member

@ss2165 ss2165 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review part 2, tests

10 | xs.pop()
| ^^^^^^^^^^^^

Borrowed argument `xs` cannot be returned back to the caller. Expected it to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest rephrasing something like "xs is borrowed so must be returned to the caller. ..."

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should also mention that this return is happening implicitly. Otherwise, useres might get confused and try to return it themselves?

| ^^^^^^^^^^^^^^

Borrowed argument `xs` cannot be returned back to the caller. Cannot infer the
type of empty list
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
type of empty list
type of empty list.

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch
@mark-koch mark-koch requested a review from ss2165 March 17, 2025 10:20

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch

Verified

This commit was signed with the committer’s verified signature.
mark-koch Mark Koch
@@ -3,4 +3,4 @@ Traceback (most recent call last):
module.compile()
File "$FILE", line 13, in test
foo(1, 2, 3, 4)
guppylang.error.GuppyComptimeError: Too many arguments: Expected 2, got 4
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happened to the expected line

using `guppy_object_from_py`, updates the wires of any `GuppyObjects` contained in
`v` to the new wires specified by `obj`.

Also resets the used flag on any of those updated wires.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this done? I couldn't find it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tracing MVP
3 participants