Add Generator Expr support #232

Zehen-249 · 2025-04-01T06:16:36Z

Pull Request description

Add Generator Expression Supports

Solve #199

Pull Request checklists

Note:

Share updated images of the graph representation of the new ASTx node
proposed in this PR, in both image and ASCII formats. For more
information, check this Google Colab notebook:
https://colab.research.google.com/drive/1xXwHmOMkJKFSmhRvn4WYfSAsdDzMnawf?usp=sharing

This PR is a:

bug-fix
new feature
maintenance

About this PR:

it includes tests.
the tests are executed on CI.
the tests generate log file(s) (path).
pre-commit hooks were executed locally.
this PR requires a project documentation update.

Author's checklist:

I have reviewed the changes and it contains no misspelling.
The code is well commented, especially in the parts that contain more
complexity.
New and old tests passed locally.

Additional information

Generator Expr Object Image representation

String Representation

GeneratorExpr[element=BinaryOp[+](Variable[x],Variable[x]), target=Variable[x], iterable=Identifier, conditions=['BinaryOp[<](Variable[x],LiteralInt32(8))', 'BinaryOp[%](Variable[x],LiteralInt32(2))']

Reviewer's checklist

Copy and paste this template for your review's note:

## Reviewer's Checklist

- [ ] I managed to reproduce the problem locally from the `main` branch
- [ ] I managed to test the new changes locally
- [ ] I confirm that the issues mentioned were fixed/resolved .

review-notebook-app · 2025-04-01T06:16:42Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Zehen-249 · 2025-04-01T06:23:39Z

Hey @xmnlab
I am encountering this failure during the pre-commit checks

mypy.....................................................................Failed
- hook id: mypy
- exit code: 1

libs\astx\src\astx\flows.py:742: error: Unsupported target for indexed assignment ("Union[list[DataTypesStruct], DictDataTypesStruct, Undefined]")  [index]
libs\astx\src\astx\flows.py:742: error: No overload variant of "__setitem__" of "list" matches argument types "str", "Union[list[DataTypesStruct], DictDataTypesStruct, Undefined]"  [call-overload]
libs\astx\src\astx\flows.py:742: note: Possible overload variants:
libs\astx\src\astx\flows.py:742: note:     def __setitem__(self, SupportsIndex, DataTypesStruct, /) -> None
libs\astx\src\astx\flows.py:742: note:     def __setitem__(self, slice[Any, Any, Any], Iterable[DataTypesStruct], /) -> None
Found 2 errors in 1 file (checked 65 source files)

The source is the data type of the conditions key i am using in the get_structs method

If I convert the data type to List the pre-commit tests will pass but the nodes for conditions get lost in the image representation.

I was hoping if you could provide some insight on this

Zehen-249 · 2025-04-02T17:25:26Z

I figured it out, is there anything left in this PR before merge into main

xmnlab · 2025-04-02T21:34:08Z

libs/astx-transpilers/tests/test_python.py

+        ],
+    )
+    generated_code = transpiler.visit(gen_expr)
+    expected_code = "(x for (x + x) in range(10) if (x > 3) if (x < 7))"


maybe I didn't understand it .. but it seems it has an incorrect python syntax:

>>> (x for (x + x) in range(10) if (x > 3) if (x < 7)) File "<stdin>", line 1 (x for (x + x) in range(10) if (x > 3) if (x < 7)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SyntaxError: expected 'else' after 'if' expression

instead of using transpiler.visit, please use translate (e.g. https://github.com/arxlang/astx/pull/232/files#diff-213fa69c2a6fc65240961486159b29cc98cffaab2968b96afbae37a47abc4624R52)

it uses the translate.visit + a function to check if the syntax is correct

xmnlab · 2025-04-02T21:34:52Z

libs/astx-transpilers/tests/test_python.py

+    )
+
+    generated_code = transpiler.visit(gen_expr)
+    expected_code = "(x for (x + x) in range(10))"


it is not a valid python syntax:

>>> (x for (x + x) in range(10)) File "<stdin>", line 1 (x for (x + x) in range(10)) ^^^^^ SyntaxError: cannot assign to expression

instead of using transpiler.visit, please use translate (e.g. https://github.com/arxlang/astx/pull/232/files#diff-213fa69c2a6fc65240961486159b29cc98cffaab2968b96afbae37a47abc4624R52)

it uses the translate.visit + a function to check if the syntax is correct

yeah I fixed the bug.

xmnlab · 2025-04-03T21:16:07Z

@Zehen-249 this is the current graph representation in image format

xmnlab · 2025-04-03T21:21:30Z

as you can see in the image above, the "conditions" is not properly implemented.

instead of list of expr it should be a astnodes of ifexpr
I will share a similar code here:

https://github.com/arxlang/astx/blob/main/libs/astx/src/astx/classes.py#L39

https://github.com/arxlang/astx/blob/main/libs/astx/src/astx/classes.py#L52

https://github.com/arxlang/astx/blob/main/libs/astx/src/astx/classes.py#L74

so basically in the class you will set the type as ASTNodes .. in the init you should accept both ASTNodes and Iterable .. and if it is an iterable you need to convert it to astnodes

in this way, in the get_struct, the conditions can be easily prepared such as any other AST node.

let me know if you have any question

xmnlab · 2025-04-03T21:27:38Z

please, after you fix the issue share in a new comment the graph representation in image and ascii format .. so it is easier to give you feedback

Zehen-249 · 2025-04-04T05:36:44Z

https://github.com/arxlang/astx/blob/main/libs/astx/src/astx/classes.py#L52

https://github.com/arxlang/astx/blob/main/libs/astx/src/astx/classes.py#L74

so basically in the class you will set the type as ASTNodes .. in the init you should accept both ASTNodes and Iterable .. and if it is an iterable you need to convert it to astnodes

in this way, in the get_struct, the conditions can be easily prepared such as any other AST node.

let me know if you have any question

if I use ASTNodes[IfExpr] the the transpiler for IfExpr generates 'pass if (x > 2) else None' something like this (here block node is empty if I append Variable x in it then it becomes 'x if (x > 2) else None'
But this is not the syntax to use in generator expr
I must avoid the then block and else block only the condition is used in generators ( 'if (x > 2)' )

This can be avoid if conditions are just ASTNodes[Expr] type.
should do this ??

Zehen-249 · 2025-04-04T05:57:29Z

Changed the type for conditions to ASTNodes[Expr]

Object

conditions = astx.ASTNodes[astx.Expr]()
conditions.append(
    astx.BinaryOp(
        op_code="<", lhs=astx.Variable("x"), rhs=astx.LiteralInt32(8)
    )
)
conditions.append(
    astx.BinaryOp(
        op_code="%", lhs=astx.Variable("x"), rhs=astx.LiteralInt32(2)
    )
)
gen_expr = astx.GeneratorExpr(
    element=astx.BinaryOp(
        op_code="+", lhs=astx.Variable("x"), rhs=astx.Variable("x")
    ),
    target=astx.Variable("x"),
    iterable=astx.Identifier("range(10)"),
    conditions=conditions,
)

ASCII Representation

Image Representation

transpiller output

((x + x) for x in range(10) if (x < 8) if (x % 2))

String Representation

GeneratorExpr[element=BinaryOp[+](Variable[x],Variable[x]), target=Variable[x], iterable=Identifier, conditions=['BinaryOp[<](Variable[x],LiteralInt32(8))', 'BinaryOp[%](Variable[x],LiteralInt32(2))']

xmnlab

LGTM!! thanks for working on that @Zehen-249 !

github-actions · 2025-04-14T00:08:22Z

🎉 This PR is included in version 0.21.0 🎉

The release is available on:

0.21.0
GitHub release

Your semantic-release bot 📦🚀

Zehen-249 force-pushed the generator_exp branch 5 times, most recently from 3836911 to 7c3f8b8 Compare April 2, 2025 17:23

xmnlab reviewed Apr 2, 2025

View reviewed changes

Zehen-249 force-pushed the generator_exp branch 3 times, most recently from 230055c to 3ed3171 Compare April 3, 2025 11:45

feat: Add generator expr support

9f1c839

Zehen-249 force-pushed the generator_exp branch from 3ed3171 to 98d68be Compare April 3, 2025 11:50

fix:generator expression tests

890148a

Zehen-249 force-pushed the generator_exp branch from e6cc012 to 890148a Compare April 3, 2025 12:50

fix: update conditions paramenter type

8562803

Zehen-249 force-pushed the generator_exp branch from aeda3a1 to 8562803 Compare April 5, 2025 15:26

xmnlab approved these changes Apr 6, 2025

View reviewed changes

xmnlab merged commit c163c4c into arxlang:main Apr 6, 2025
19 checks passed

github-actions bot added the released label Apr 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Generator Expr support #232

Add Generator Expr support #232

Zehen-249 commented Apr 1, 2025 •

edited

Loading

review-notebook-app bot commented Apr 1, 2025

Zehen-249 commented Apr 1, 2025 •

edited

Loading

Zehen-249 commented Apr 2, 2025

xmnlab Apr 2, 2025

xmnlab Apr 2, 2025

Zehen-249 Apr 3, 2025

xmnlab Apr 2, 2025

xmnlab Apr 2, 2025

Zehen-249 Apr 3, 2025

xmnlab commented Apr 3, 2025

xmnlab commented Apr 3, 2025

xmnlab commented Apr 3, 2025

Zehen-249 commented Apr 4, 2025

Zehen-249 commented Apr 4, 2025 •

edited

Loading

xmnlab left a comment

github-actions bot commented Apr 14, 2025

Add Generator Expr support #232

Add Generator Expr support #232

Conversation

Zehen-249 commented Apr 1, 2025 • edited Loading

Pull Request description

Solve #199

Pull Request checklists

Additional information

Generator Expr Object Image representation

String Representation

Reviewer's checklist

review-notebook-app bot commented Apr 1, 2025

Zehen-249 commented Apr 1, 2025 • edited Loading

The source is the data type of the conditions key i am using in the get_structs method

If I convert the data type to List the pre-commit tests will pass but the nodes for conditions get lost in the image representation.

Zehen-249 commented Apr 2, 2025

xmnlab Apr 2, 2025

Choose a reason for hiding this comment

xmnlab Apr 2, 2025

Choose a reason for hiding this comment

Zehen-249 Apr 3, 2025

Choose a reason for hiding this comment

xmnlab Apr 2, 2025

Choose a reason for hiding this comment

xmnlab Apr 2, 2025

Choose a reason for hiding this comment

Zehen-249 Apr 3, 2025

Choose a reason for hiding this comment

xmnlab commented Apr 3, 2025

xmnlab commented Apr 3, 2025

xmnlab commented Apr 3, 2025

Zehen-249 commented Apr 4, 2025

Zehen-249 commented Apr 4, 2025 • edited Loading

Object

ASCII Representation

Image Representation

transpiller output

String Representation

xmnlab left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 14, 2025

Zehen-249 commented Apr 1, 2025 •

edited

Loading

Zehen-249 commented Apr 1, 2025 •

edited

Loading

Zehen-249 commented Apr 4, 2025 •

edited

Loading