Fix issue with CALL/YIELD for user defined and qualified functions. by jrgemignani · Pull Request #2217 · apache/age

jrgemignani · 2025-09-09T19:14:21Z

Fixed 2 issues with CALL/YIELD -

If a user defined function was in search_path, the transform_FuncCall
logic would only find it, if it were part of an extension.
If a function were qualified, the transform_cypher_call_subquery
logic would mistakenly extract the schema name instead of the
function name.

NOTE: transform_FuncCall should be reviewed for possible refactor.

Added regression tests.

modified:   src/backend/parser/cypher_clause.c
modified:   src/backend/parser/cypher_expr.c
modified:   regress/expected/cypher_call.out
modified:   regress/sql/cypher_call.sql

Fixed 2 issues with CALL/YIELD - 1) If a user defined function was in search_path, the transform_FuncCall logic would only find it, if it were part of an extension. 2) If a function were qualified, the transform_cypher_call_subquery logic would mistakenly extract the schema name instead of the function name. NOTE: transform_FuncCall should be reviewed for possible refactor. Added regression tests. modified: src/backend/parser/cypher_clause.c modified: src/backend/parser/cypher_expr.c modified: regress/expected/cypher_call.out modified: regress/sql/cypher_call.sql

…pache#2217) Fixed 2 issues with CALL/YIELD - 1) If a user defined function was in search_path, the transform_FuncCall logic would only find it, if it were part of an extension. 2) If a function were qualified, the transform_cypher_call_subquery logic would mistakenly extract the schema name instead of the function name. NOTE: transform_FuncCall should be reviewed for possible refactor. Added regression tests. modified: src/backend/parser/cypher_clause.c modified: src/backend/parser/cypher_expr.c modified: regress/expected/cypher_call.out modified: regress/sql/cypher_call.sql

…2217) Fixed 2 issues with CALL/YIELD - 1) If a user defined function was in search_path, the transform_FuncCall logic would only find it, if it were part of an extension. 2) If a function were qualified, the transform_cypher_call_subquery logic would mistakenly extract the schema name instead of the function name. NOTE: transform_FuncCall should be reviewed for possible refactor. Added regression tests. modified: src/backend/parser/cypher_clause.c modified: src/backend/parser/cypher_expr.c modified: regress/expected/cypher_call.out modified: regress/sql/cypher_call.sql

…pache#2217) Fixed 2 issues with CALL/YIELD - 1) If a user defined function was in search_path, the transform_FuncCall logic would only find it, if it were part of an extension. 2) If a function were qualified, the transform_cypher_call_subquery logic would mistakenly extract the schema name instead of the function name. NOTE: transform_FuncCall should be reviewed for possible refactor. Added regression tests. modified: src/backend/parser/cypher_clause.c modified: src/backend/parser/cypher_expr.c modified: regress/expected/cypher_call.out modified: regress/sql/cypher_call.sql

…2217) Fixed 2 issues with CALL/YIELD - 1) If a user defined function was in search_path, the transform_FuncCall logic would only find it, if it were part of an extension. 2) If a function were qualified, the transform_cypher_call_subquery logic would mistakenly extract the schema name instead of the function name. NOTE: transform_FuncCall should be reviewed for possible refactor. Added regression tests. modified: src/backend/parser/cypher_clause.c modified: src/backend/parser/cypher_expr.c modified: regress/expected/cypher_call.out modified: regress/sql/cypher_call.sql

* Implement Returnless Unions in Subqueries (#1803) Added logic to support returnless unions in subqueries. Added regression tests to support this addition. Modified regression tests to capture more cases. (cherry picked from commit ffc9869232a6c8d9a824277971bf4dcc4956bedf) * Make CALL YIELD grammar more precise (#1852) Replaces `CALL expr . expr` to `CALL expr_var . expr_func_norm`. There was a if-block before that checked if the first `expr` is a ColumnRef and the second `expr` is a FuncCall node. That check is not need now since expr_var and expr_func_norm are reduced to ColumnRef and FuncCall node respectively. This change also prevents shift\reduce ambiguity with CALL subquery syntax. (cherry picked from commit ed08b870bf419ed5d82619e800f2645c00fc317e) * Backport Datum and node compatibility fixes for PostgreSQL 15 Partial backport: retain the Datum return-width corrections, nullable node strings and SET property-name guard. Preserve PG15 APIs. PG15 adaptations: explicitly encode non-null empty strings because PG15's outToken otherwise conflates them with NULL. Pass the actual Query to flatten_join_alias_vars, correcting the legacy PG15 PlannerInfo cast. The PostgreSQL 16 platform port is excluded. (cherry picked from commit 81ecd5608e94314b94f2e3e6c97c69d2d948aab0) * Fix issue with CALL/YIELD for user defined and qualified functions. (#2217) Fixed 2 issues with CALL/YIELD - 1) If a user defined function was in search_path, the transform_FuncCall logic would only find it, if it were part of an extension. 2) If a function were qualified, the transform_cypher_call_subquery logic would mistakenly extract the schema name instead of the function name. NOTE: transform_FuncCall should be reviewed for possible refactor. Added regression tests. modified: src/backend/parser/cypher_clause.c modified: src/backend/parser/cypher_expr.c modified: regress/expected/cypher_call.out modified: regress/sql/cypher_call.sql (cherry picked from commit 94770a69751c8ecdbf21887ee83a98ed6ae979c8) * Bump gopkg.in/yaml.v3 from 3.0.0 to 3.0.1 in /drivers/golang (#2212) Bumps gopkg.in/yaml.v3 from 3.0.0 to 3.0.1. --- updated-dependencies: - dependency-name: gopkg.in/yaml.v3 dependency-version: 3.0.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> (cherry picked from commit 9c370f421ff0d138c14080a069524aff14682e19) * Add fast functions for checking edge uniqueness (#2227) Added fast functions for checking edge uniqueness. This will help improve performance for MATCH queries with paths longer than 3 but less than 11. The normal edge uniqueness function will deal with any path 11 and over. modified: age--1.6.0--y.y.y.sql modified: sql/agtype_graphid.sql modified: src/backend/parser/cypher_clause.c modified: src/backend/utils/adt/age_vle.c (cherry picked from commit ec457345e25601c947aff73bfcc7354d97e238a6) * Fix issue 2243 - Regression in string concatenation (#2244) Fixed issue 2243 - Regression in string concatenation using the + operator. The issue was in versions 1.5.0 and 1.6.0, at least. It was due to using Int8GetDatum instead of Int64GetDatum for the agtype integer field in the following functions - get_numeric_datum_from_agtype_value get_string_from_agtype_value This impacted more than what the original issue covered, but those additional cases were resolved too. Added regression tests. modified: regress/expected/agtype.out modified: regress/sql/agtype.sql modified: src/backend/utils/adt/agtype_ops.c (cherry picked from commit 190354c415851b9eba893b314fea55623518b3a6) * Fix issue 2245 - Creating more than 41 vlabels causes crash in drop_graph (#2248) Fixed issue 2245 - Creating more than 41 vlabels causes drop_grapth to fail with "label (relation) cache corrupted" and crashing out on the following command. This was due to corruption of the label_relation_cache during the HASH_DELETE process. As the issue was with a cache flush routine, it was necessary to fix them all. Here is the list of the flush functions that were fixed - static void flush_graph_name_cache(void) static void flush_graph_namespace_cache(void) static void flush_label_name_graph_cache(void) static void flush_label_graph_oid_cache(void) static void flush_label_relation_cache(void) static void flush_label_seq_name_graph_cache(void) Added regression tests. modified: regress/expected/catalog.out modified: regress/sql/catalog.sql modified: src/backend/utils/cache/ag_cache.c (cherry picked from commit 571c1982437fc3a9a0bebca293418c22ba64e270) * Add index on id columns (#2117) - Whenever a label will be created, indices on id columns will be created by default. In case of vertex, a unique index on id column will be created, which will also serve as a unique constraint. In case of edge, a non-unique index on start_id and end_id columns will be created. - This change is expected to improve the performance of queries that involve joins. From some performance tests, it was observed that the performance of queries improved alot. - Loader was updated to insert tuples in indices as well. This has caused to slow the loader down a bit, but it was necessary. - A bug related to command ids in cypher_delete executor was also fixed. PG15 adaptation: retain the native ExecInsertIndexTuples signature while replacing the loader's obsolete temporary-table uniqueness handling. (cherry picked from commit 5aed9ecc5b492a47dc0e449421cf344de58b5edd) * Fix Issue 2256: segmentation fault when calling coalesce function (#2259) Fixed issue 2256: A segmentation fault occurs when calling the coalesce function in PostgreSQL version 17. This likely predates 17 and includes other similar types of "functions". See issues 1124 (PR 1125) and 1303 (PR 1317) for more details. This issue is due to coalesce() being processed differently from other functions. Additionally, greatest() was found to exhibit the same behavior. They were added to the list of types to ignore during the cypher analyze phase. A few others were added: CaseExpr, XmlExpr, ArrayExpr, & RowExpr. Although, I wasn't able to find cases where these caused crashes. Added regression tests. modified: regress/expected/cypher.out modified: regress/sql/cypher.sql modified: src/backend/parser/cypher_analyze.c PG15 adaptation: omit JsonConstructorExpr, which does not exist in PG15. (cherry picked from commit 26f748c42b01b7d6ff5c5ec7c4eb84d97066103a) * Adjust 'could not find rte for' ERROR message (#2266) Adjusted the following type of error message. It was mentioned in issue 2263 as being incorrect, which it isn't. However, it did need some clarification added - ERROR: could not find rte for <column name> Added a HINT for additional clarity - HINT: variable <column name> does not exist within scope of usage For example: CREATE p0=(n0), (n1{k:EXISTS{WITH p0}}) RETURN 1 ERROR: could not find rte for p0 LINE 3: CREATE p0=(n0), (n1{k:EXISTS{WITH p0}}) ^ HINT: variable p0 does not exist within scope of usage Additionally, added pstate->p_expr_kind == EXPR_KIND_INSERT_TARGET to transform_cypher_clause_as_subquery. Updated existing regression tests. Added regression tests from issue. modified: regress/expected/cypher_call.out modified: regress/expected/cypher_subquery.out modified: regress/expected/cypher_union.out modified: regress/expected/cypher_with.out modified: regress/expected/expr.out modified: regress/expected/list_comprehension.out modified: regress/expected/scan.out modified: src/backend/parser/cypher_clause.c modified: src/backend/parser/cypher_expr.c (cherry picked from commit fa9973aab32adb0b611b09ee26ae127b92059af2) * Fix possible memory and file descriptors leaks (#2258) - Used postgres memory allocation functions instead of standard ones. - Wrapped main loop of csv loader in PG_TRY block for better error handling. (cherry picked from commit 0ea94644f25bdc08c803ff6521bfd412a6690401) * Fix ORDER BY alias resolution with AS in Cypher queries (#2269) NOTE: This PR was partially created with AI tools and reviewed by a human. ORDER BY clauses failed when referencing column aliases from RETURN: MATCH (p:Person) RETURN p.age AS age ORDER BY age DESC ERROR: could not find rte for age Added SQL-99 compliant alias matching to find_target_list_entry() that checks if ORDER BY identifier matches a target list alias before attempting expression transformation. This enables standard SQL behavior for sorting by aliased columns with DESC/DESCENDING/ASC/ASCENDING. Updated regression tests. Added regression tests. modified: regress/expected/cypher_match.out modified: regress/expected/expr.out modified: regress/sql/expr.sql modified: src/backend/parser/cypher_clause.c (cherry picked from commit 898481a8c79381cc9f49ca71a9756a6216b61353) * Update grammar file for maintainability (#2270) Consolidated duplicate code, added helper functions, and reviewed the grammar file for issues. NOTE: I used an AI tool to review and cleanup the grammar file. I have reviewed all of the work it did. Improvements: 1. Added KEYWORD_STRDUP macro to eliminate hardcoded string lengths 2. Consolidated EXPLAIN statement handling into make_explain_stmt helper 3. Extracted WITH clause validation into validate_return_item_aliases helper 4. Created make_default_return_node helper for subquery return-less logic Benefits: - Reduced code duplication by ~150 lines - Improved maintainability with helper functions - Eliminated manual string length calculations (error-prone) All 29 existing regression tests pass modified: src/backend/parser/cypher_gram.y (cherry picked from commit 91de779aee0ee241bf8022b4822a9745c8bc9fa3) * Convert string to raw string to remove invalid escape sequence warning (#2267) - Changed '\s' to r'\s' (cherry picked from commit 1bb95bf616abe733dd6d5c02aa32c7f419827646) * Migrate python driver configuration to pyproject.toml (#2272) - Add pyproject.toml with package configuration - Simplify setup.py to minimal backward-compatible wrapper. - Updated CI workflow and .gitignore. - Resolves warning about using setup.py directly. (cherry picked from commit 838926cc35ae64d3514c656959b749719e904b09) * Restrict age_load commands (#2274) This PR applies restrictions to the following age_load commands - load_labels_from_file() load_edges_from_file() They are now tied to a specific root directory and are required to have a specific file extension to eliminate any attempts to force them to access any other files. Nothing else has changed with the actual command formats or parameters, only that they work out of the /tmp/age directory and only access files with an extension of .csv. Added regression tests and updated the location of the csv files for those regression tests. modified: regress/expected/age_load.out modified: regress/sql/age_load.sql modified: src/backend/utils/load/age_load.c (cherry picked from commit 48fca83d90f29cf68fd9748db3b2d628f4978749) * Makefile: fix race condition on cypher_gram_def.h (#2273) The file cypher_gram.c generates cypher_gram_def.h, which is directly necessary for cypher_parser.o and cypher_keywords.o and their respective .bc files. But that direct dependency is not reflected in the Makefile, which only had the indirect dependency of .o on .c. So on high parallel builds, the .h may not have been generated by bison yet. Additionally, the .bc files should have the same dependencies as the .o files, but those are lacking. Here is an example output where the .bc file fails to build, as it was running concurrently with the bison instance that was about to finalize cypher_gram_def.h: In file included from src/backend/parser/cypher_parser.c:24: clang-17 -Wno-ignored-attributes -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -O2 -I.//src/include -I.//src/include/parser -I. -I./ -I/usr/pgsql-17/include/server -I/usr/pgsql-17/include/internal -D_GNU_SOURCE -I/usr/include -I/usr/include/libxml2 -flto=thin -emit-llvm -c -o src/backend/parser/cypher_parser.bc src/backend/parser/cypher_parser.c .//src/include/parser/cypher_gram.h:65:10: fatal error: 'parser/cypher_gram_def.h' file not found 65 | #include "parser/cypher_gram_def.h" | ^~~~~~~~~~~~~~~~~~~~~~~~~~ 1 error generated. make: *** [/usr/pgsql-17/lib/pgxs/src/makefiles/../../src/Makefile.global:1085: src/backend/parser/cypher_parser.bc] Error 1 make: *** Waiting for unfinished jobs.... gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wshadow=compatible-local -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -O2 -g -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -fPIC -fvisibility=hidden -I.//src/include -I.//src/include/parser -I. -I./ -I/usr/pgsql-17/include/server -I/usr/pgsql-17/include/internal -D_GNU_SOURCE -I/usr/include -I/usr/include/libxml2 -c -o src/backend/catalog/ag_label.o src/backend/catalog/ag_label.c /usr/bin/bison -Wno-deprecated --defines=src/include/parser/cypher_gram_def.h -o src/backend/parser/cypher_gram.c src/backend/parser/cypher_gram.y Previously, cypher_parser.o was missing the dependency, so it could start before cypher_gram_def.h was available: Considering target file 'src/backend/parser/cypher_parser.o'. File 'src/backend/parser/cypher_parser.o' does not exist. Considering target file 'src/backend/parser/cypher_parser.c'. File 'src/backend/parser/cypher_parser.c' was considered already. Considering target file 'src/backend/parser/cypher_gram.c'. File 'src/backend/parser/cypher_gram.c' was considered already. Finished prerequisites of target file 'src/backend/parser/cypher_parser.o'. Must remake target 'src/backend/parser/cypher_parser.o'. As well as cypher_parser.bc, missing the dependency on cypher_gram_def.h: Considering target file 'src/backend/parser/cypher_parser.bc'. File 'src/backend/parser/cypher_parser.bc' does not exist. Considering target file 'src/backend/parser/cypher_parser.c'. File 'src/backend/parser/cypher_parser.c' was considered already. Finished prerequisites of target file 'src/backend/parser/cypher_parser.bc'. Must remake target 'src/backend/parser/cypher_parser.bc'. Now cypher_parser.o correctly depends on cypher_gram_def.h: Considering target file 'src/backend/parser/cypher_parser.o'. File 'src/backend/parser/cypher_parser.o' does not exist. Considering target file 'src/backend/parser/cypher_parser.c'. File 'src/backend/parser/cypher_parser.c' was considered already. Considering target file 'src/backend/parser/cypher_gram.c'. File 'src/backend/parser/cypher_gram.c' was considered already. Considering target file 'src/include/parser/cypher_gram_def.h'. File 'src/include/parser/cypher_gram_def.h' was considered already. Finished prerequisites of target file 'src/backend/parser/cypher_parser.o'. Must remake target 'src/backend/parser/cypher_parser.o'. And cypher_parser.bc correctly depends on cypher_gram_def.h as well: Considering target file 'src/backend/parser/cypher_parser.bc'. File 'src/backend/parser/cypher_parser.bc' does not exist. Considering target file 'src/backend/parser/cypher_parser.c'. File 'src/backend/parser/cypher_parser.c' was considered already. Considering target file 'src/backend/parser/cypher_gram.c'. File 'src/backend/parser/cypher_gram.c' was considered already. Considering target file 'src/include/parser/cypher_gram_def.h'. File 'src/include/parser/cypher_gram_def.h' was considered already. Finished prerequisites of target file 'src/backend/parser/cypher_parser.bc'. Must remake target 'src/backend/parser/cypher_parser.bc'. (cherry picked from commit dd6deb70e34e2ab75836dc6d037fde5a360d7075) * Revise README for Python driver updates (#2298) Updated README to from psycopg2 to psycopg3 (psycopg) (cherry picked from commit 4eeceab256f4795e6e8fb1d802935bdf403e5e93) * Fix Issue 2289: handle empty list in IN expression (#2294) NOTE: This PR was created with AI tools and a human. When evaluating 'x IN []' with an empty list, the transform_AEXPR_IN function would return NULL because no expressions were processed. This caused a 'cache lookup failed for type 0' error downstream. This fix adds an early check for the empty list case: - 'x IN []' returns false (nothing can be in an empty list) Additional NOTE: Cypher does not have 'NOT IN' syntax. To check if a value is NOT in a list, use 'NOT (x IN list)'. The NOT operator will invert the false from an empty list to true as expected. The fix returns a boolean constant directly, avoiding the NULL result that caused the type lookup failure. Added regression tests. modified: regress/expected/expr.out modified: regress/sql/expr.sql modified: src/backend/parser/cypher_expr.c (cherry picked from commit 2e8f7ab992fc5db6cedf88cbdffc15df3a3cf932) * Fix and improve index.sql regression test coverage (#2300) NOTE: This PR was created with AI tools and a human. - Remove unused copy command (leftover from deleted agload_test_graph test) - Replace broken Section 4 that referenced non-existent graph with comprehensive WHERE clause tests covering string, int, bool, and float properties with AND/OR/NOT operators - Add EXPLAIN tests to verify index usage: - Section 3: Validate GIN indices (load_city_gin_idx, load_country_gin_idx) show Bitmap Index Scan for property matching - Section 4: Validate all expression indices (city_country_code_idx, city_id_idx, city_west_coast_idx, country_life_exp_idx) show Index Scan for WHERE clause filtering All indices now have EXPLAIN verification confirming they are used as expected. modified: regress/expected/index.out modified: regress/sql/index.sql (cherry picked from commit 7beb653303529b05391667de4204ffb4da318eeb) * Fix and improve index.sql addendum (#2301) NOTE: This PR was created with the help of AI tools and a human. Added additional requested regression tests - *EXPLAIN for pattern with WHERE clause *EXPLAIN for pattern with filters on both country and city modified: regress/expected/index.out modified: regress/sql/index.sql (cherry picked from commit a1f472d6f9344dc4449ac7343bf7d81a31b66f02) * feat: Add 32-bit platform support for graphid type (#2286) * feat: Add 32-bit platform support for graphid type This enables AGE to work on 32-bit platforms including WebAssembly (WASM). Problem: - graphid is int64 (8 bytes) with PASSEDBYVALUE - On 32-bit systems, Datum is only 4 bytes - PostgreSQL rejects pass-by-value types larger than Datum Solution: - Makefile-only change (no C code modifications) - When SIZEOF_DATUM=4 is passed to make, strip PASSEDBYVALUE from the generated SQL - If not specified, normal 64-bit behavior is preserved (PASSEDBYVALUE kept) This change is backward compatible: - 64-bit systems continue using pass-by-value - 32-bit systems now work with pass-by-reference Motivation: PGlite (PostgreSQL compiled to WebAssembly) uses 32-bit pointers and requires this patch to run AGE. Tested on: - 64-bit Linux (regression tests pass) - 32-bit WebAssembly via PGlite (all operations work) Co-authored-by: abbuehlj <jean-paul.abbuehl@roche.com> (cherry picked from commit c979380e9865a624ad73eef01ec84717bb817f2b) * Optimize vertex/edge field access with direct array indexing (#2302) NOTE: This PR was created using AI tools and a human. Leverage deterministic key ordering from uniqueify_agtype_object() to access vertex/edge fields in O(1) instead of O(log n) binary search. Fields are sorted by key length, giving fixed positions: - Vertex: id(0), label(1), properties(2) - Edge: id(0), label(1), end_id(2), start_id(3), properties(4) Changes: - Add field index constants and accessor macros to agtype.h - Update age_id(), age_start_id(), age_end_id(), age_label(), age_properties() to use direct field access - Add fill_agtype_value_no_copy() for read-only scalar extraction without memory allocation - Add compare_agtype_scalar_containers() fast path for scalar comparison - Update hash_agtype_value(), equals_agtype_scalar_value(), and compare_agtype_scalar_values() to use direct field access macros - Add fast path in get_one_agtype_from_variadic_args() bypassing extract_variadic_args() for single argument case - Add comprehensive regression test (30 tests) Performance impact: Improves ORDER BY, hash joins, aggregations, and Cypher functions (id, start_id, end_id, label, properties) on vertices and edges. All previous regression tests were not impacted. Additional regression test added to enhance coverage. modified: Makefile new file: regress/expected/direct_field_access.out new file: regress/sql/direct_field_access.sql modified: src/backend/utils/adt/agtype.c modified: src/backend/utils/adt/agtype_util.c modified: src/include/utils/agtype.h (cherry picked from commit b9d0982892306abff0013dd8f336e153684b02e9) * Upgrade Jest to v29 for node: protocol compatibility (#2307) Note: This PR was created with AI tools and a human. The pg-connection-string module (dependency of pg) now uses the node: protocol prefix for built-in modules (e.g., require('node:process')). Jest 26 does not support this syntax, causing test failures. Changes: - Upgrade jest from ^26.6.3 to ^29.7.0 - Upgrade ts-jest from ^26.5.1 to ^29.4.6 - Upgrade @types/jest from ^26.0.20 to ^29.5.14 - Update typescript to ^4.9.5 This also resolves 19 npm audit vulnerabilities (17 moderate, 2 high) that existed in the older Jest 26 dependency tree. modified: drivers/nodejs/package.json (cherry picked from commit 8bdeec54e898451771f4dc021a13b00781d6d1a0) * Fix Issue 1884: Ambiguous column reference (#2306) Fix Issue 1884: Ambiguous column reference and invalid AGT header errors. Note: This PR was created with AI tools and a human, or 2. This commit addresses two related bugs that occur when using SET to store graph elements (vertices, edges, paths) as property values: Issue 1884 - "column reference is ambiguous" error: When a Cypher query uses the same variable in both the SET expression RHS and the RETURN clause (e.g., SET n.prop = n RETURN n), PostgreSQL would report "column reference is ambiguous" because the variable appeared in multiple subqueries without proper qualification. Solution: The fix for this issue was already in place through the target entry naming scheme that qualifies column references. "Invalid AGT header value" offset error: When deserializing nested VERTEX, EDGE, or PATH values stored in properties, the system would fail with errors like "Invalid AGT header value: 0x00000041". This occurred because ag_serialize_extended_type() did not include alignment padding (padlen) in the agtentry length calculation for these types, while fill_agtype_value() uses INTALIGN() when reading, causing offset mismatch. Solution: Modified ag_serialize_extended_type() in agtype_ext.c to include padlen in the agtentry length for VERTEX, EDGE, and PATH cases, matching the existing pattern used for INTEGER, FLOAT, and NUMERIC types: *agtentry = AGTENTRY_IS_AGTYPE | (padlen + (AGTENTRY_OFFLENMASK & ...)); This ensures the serialized length accounts for alignment padding, allowing correct deserialization of nested graph elements. Appropriate regression tests were added to verify the fixes. Co-authored by: Zainab Saad <105385638+Zainab-Saad@users.noreply.github.com> modified: regress/expected/cypher_set.out modified: regress/sql/cypher_set.sql modified: src/backend/parser/cypher_clause.c modified: src/backend/utils/adt/agtype_ext.c (cherry picked from commit 56a92d8c1be364e07bac51665a362ca91957194b) * Replace libcsv with pg COPY for csv loading (#2310) - Commit also adds permission checks - Resolves a critical memory spike issue on loading large file - Use pg's COPY infrastructure (BeginCopyFrom, NextCopyFromRawFields) for 64KB buffered CSV parsing instead of libcsv - Add byte based flush threshold (64KB) matching COPY behavior for memory safety - Use heap_multi_insert with BulkInsertState for optimized batch inserts - Add per batch memory context to prevent memory growth during large loads - Remove libcsv dependency (libcsv.c, csv.h) - Improves loading performance by 15-20% - No previous regression tests were impacted - Added regression tests for permissions/rls Assisted-by AI PG15 adaptation: store loader permissions on RangeTblEntry and use the PG15 ExecInitRangeTable signature. Retain PG15's JSON-validation helper. (cherry picked from commit b29ca5e7d2f84cfe2619eea70c4ace2cba41aa0b) * Add RLS support and fix permission checks (#2309) - Previously, age only set ACL_SELECT and ACL_INSERT in RTEPermissionInfo, bypassing pg's privilege checking for DELETE and UPDATE operations. - Additionally, RLS policies were not enforced because AGE uses CMD_SELECT for all Cypher queries, causing the rewriter to skip RLS policy application. Permission fixes: - Add ACL_DELETE permission flag for DELETE clause operations - Add ACL_UPDATE permission flag for SET/REMOVE clause operations - Recursively search RTEs including subqueries for permission info RLS support: - Implemented at executor level because age transforms all cypher queries to CMD_SELECT, so pg's rewriter never adds RLS policies for INSERT/UPDATE/DELETE operations. There isnt an appropriate rewriter hook to modify this behavior, so we do it in executor instead. - Add setup_wcos() to apply WITH CHECK policies at execution time for CREATE, SET, and MERGE operations - Add setup_security_quals() and check_security_quals() to apply USING policies for UPDATE and DELETE operations - USING policies silently filter rows (matching pg behavior) - WITH CHECK policies raise errors on violation - DETACH DELETE raises error if edge RLS blocks deletion to prevent dangling edges - Add permission checks and rls in startnode/endnode functions - Add regression tests Assisted-by AI PG15 adaptation: recursively update RangeTblEntry.requiredPerms instead of the newer RTEPermissionInfo structure. (cherry picked from commit 1702ae075de6897e9555fa527060f622304a1d93) * Backport index ownership and header fixes for PostgreSQL 15 (#2315) Partial backport of the PG18 port: retain the index ownership guard, TupleDescAttr accessors, explicit includes, exported declarations and keyword header fix. Preserve PG15 APIs, build targets and expected order. Co-authored-by: Krishnakumar R (KK) <65895020+kk-src@users.noreply.github.com> (cherry picked from commit b3219fd4228cc261df06070e37512c76b8c07678) * Fix upgrade script for 1.6.0 to 1.7.0 (#2320) - Added index creation for existing labels Assisted-by AI (cherry picked from commit 6287af85f291e10d048c37ffd22b902950779af7) * Advance master branch to Apache AGE version 1.7.0 (#2316) Updated the following files to advance the Apache AGE version to 1.7.0 modified: Makefile modified: README.md modified: RELEASE renamed: age--1.6.0--y.y.y.sql -> age--1.6.0--1.7.0.sql new file: age--1.7.0--y.y.y.sql modified: age.control modified: docker/Dockerfile deleted: age--1.5.0--1.6.0.sql PG15 adaptation: preserve the existing 1.5.0-to-1.6.0 upgrade script and PostgreSQL 15 documentation and Docker targets. (cherry picked from commit 858747c7e2a414651b4f8e76558fdce1bbdc0af8) * Add pg_upgrade support functions for PostgreSQL (#2326) Add pg_upgrade support functions for PostgreSQL for major version upgrades NOTE: This PR was created with AI tools and a human. The ag_graph.namespace column uses the regnamespace type, which pg_upgrade cannot handle in user tables. This commit adds four SQL functions to enable seamless PostgreSQL major version upgrades while preserving all graph data. New functions in ag_catalog: - age_prepare_pg_upgrade(): Converts namespace from regnamespace to oid, creates backup table with graph-to-namespace mappings (stores nspname directly to avoid quoting issues) - age_finish_pg_upgrade(): Remaps stale OIDs after upgrade, restores regnamespace type, invalidates AGE caches while preserving schema ownership - age_revert_pg_upgrade_changes(): Cancels preparation if upgrade is aborted - age_pg_upgrade_status(): Returns current upgrade readiness status Usage: 1. Before pg_upgrade: SELECT age_prepare_pg_upgrade(); 2. Run pg_upgrade as normal 3. After pg_upgrade: SELECT age_finish_pg_upgrade(); Key implementation details: - Uses transaction-level advisory locks (pg_advisory_xact_lock) for safety - Preserves original schema ownership during cache invalidation - Validates all backup rows are mapped before proceeding - Handles zero-graph edge case gracefully - Handles insufficient privileges gracefully with informative notices - Backup table deleted only after all steps succeed Files changed: - sql/age_pg_upgrade.sql: New file with function implementations - sql/sql_files: Added age_pg_upgrade entry - age--1.7.0--y.y.y.sql: Added functions for extension upgrades All regression tests pass. (cherry picked from commit 5fe2121a06f692efc9a8c3d77b57c67b874e1b69) * Fix JDBC driver CI test failures (#2333) Fix JDBC driver CI test failures: encoding, Testcontainers, and Docker compatibility. Note: This PR was created with the help of AI tools and a human. - Set JavaCompile encoding to UTF-8 to fix unicode test failures in CI environments that default to US-ASCII. - Add Testcontainers wait strategy (Wait.forLogMessage) to wait for PostgreSQL to be fully ready before connecting. - Use agensGraphContainer.getHost() instead of hardcoded localhost for Docker-in-Docker compatibility. - Add sslmode=disable to JDBC URL since PostgreSQL driver 42.6.0+ attempts SSL by default. - Remove silent exception swallowing around connection setup to fail fast with meaningful errors instead of NullPointerException. - Upgrade Testcontainers from 1.18.0 to 1.21.4 to fix Docker 29.x detection failure on ubuntu-24.04 runners (docker-java 3.3.x in 1.18.0 cannot negotiate with Docker Engine 29.1.5 API). modified: drivers/jdbc/lib/build.gradle.kts modified: drivers/jdbc/lib/src/test/java/org/apache/age/jdbc/BaseDockerizedTest.java (cherry picked from commit 887564d9ce0f153ad7dc9f30223aa303c3fb60d0) * Update python-driver security and formatting (#2330) Note: This PR was created with AI tools and a human. - Add parameterized query construction using psycopg.sql to prevent SQL injection in all Cypher execution paths (age.py, networkx/lib.py) - Replace all %-format and f-string SQL in networkx/lib.py with sql.Identifier() for schema/table names and sql.Literal() for values - Add validate_graph_name() with AGE-aligned VALID_GRAPH_NAME regex: start with letter/underscore, allow dots and hyphens in middle positions, end with letter/digit/underscore, min 3 chars, max 63 chars - Add validate_identifier() with strict VALID_IDENTIFIER regex for labels, column names, and SQL types (no dots or hyphens) - Add validation calls to all networkx/lib.py entry points: graph names validated on entry, labels validated before SQL construction - Add _validate_column() to sanitize column specifications in buildCypher() - Fix exception constructors (AgeNotSet, GraphNotFound, GraphAlreadyExists) to always call super().__init__() with a meaningful default message so that str(exception) never returns an empty string - Add InvalidGraphName and InvalidIdentifier exception classes with structured name/reason/context fields - Fix builder.py: change erroneous 'return Exception(...)' to 'raise ValueError(...)' for unknown float expressions - Fix copy-paste docstring in create_elabel() ('create_vlabels' -> 'create_elabels') - Remove unused 'from psycopg.adapt import Loader' import in age.py - Add design documentation in source explaining: - VALID_GRAPH_NAME regex uses '*' (not '+') intentionally so that the min-length check fires first with a clear error message - buildCypher uses string concatenation (not sql.Identifier) because column specs are pre-validated 'name type' pairs that don't map to sql.Identifier(); graphName and cypherStmt are NOT embedded - Update test_networkx.py GraphNotFound assertion to use assertIn() instead of assertEqual() to match the improved exception messages - Strip Windows carriage returns (^M) from 7 source files - Fix requirements.txt: convert from UTF-16LE+BOM+CRLF to clean UTF-8+LF, move --no-binary flag from requirements.txt to CI workflow pip command - Upgrade actions/setup-python from v4 (deprecated) to v5 in CI workflow - Add 46 security unit tests in test_security.py covering: - Graph name validation (AGE naming rules, injection, edge cases) - SQL identifier validation (labels, columns, types) - Column spec sanitization - buildCypher injection prevention - Exception constructor correctness (str() never empty) - Add test_security.py to CI pipeline (python-driver.yaml) - pip-audit: 0 known vulnerabilities in all dependencies modified: .github/workflows/python-driver.yaml modified: drivers/python/age/VERSION.py modified: drivers/python/age/__init__.py modified: drivers/python/age/age.py modified: drivers/python/age/builder.py modified: drivers/python/age/exceptions.py modified: drivers/python/age/models.py modified: drivers/python/age/networkx/lib.py modified: drivers/python/requirements.txt modified: drivers/python/setup.py modified: drivers/python/test_agtypes.py modified: drivers/python/test_networkx.py new file: drivers/python/test_security.py (cherry picked from commit 5f5b744a08641225652de83332d73bc7acfc889d) * Fix security vulnerabilities in Node.js driver (#2329) Fix security vulnerabilities in Node.js driver and harden input validation and query construction. Note: This PR was created with AI tools and a human. - Add input validation for graph names, label names, and column names to prevent SQL injection via string interpolation. Graph name rules are based on Apache AGE's naming conventions and Neo4j/openCypher compatibility (hyphens and dots permitted, min 3 chars, max 63 chars). Label names follow AGE's stricter rules (no hyphens or dots). - Add design documentation noting intentional ASCII-only restriction in driver-side regex validation as a security hardening measure (homoglyph/encoding attack surface reduction). AGE's server-side validation (name_validation.h) uses full Unicode ID_Start/ID_Continue and remains the authoritative check for Unicode names. - Add safe query helpers: queryCypher(), createGraph(), dropGraph() with graph name validation and dollar-quoting for Cypher strings - Add runtime typeof check on dropGraph cascade parameter to prevent injection from plain JavaScript consumers (TypeScript types are erased at runtime) - Use BigInt for integer values exceeding Number.MAX_SAFE_INTEGER to prevent silent precision loss with 64-bit AGE graph IDs - Make CREATE EXTENSION opt-in via SetAGETypesOptions.createExtension instead of running DDL automatically without user consent - Wrap LOAD/search_path setup in try/catch with actionable error message mentioning CREATE EXTENSION and { createExtension: true } - Improve agtype-not-found error message with installation guidance - Tighten pg dependency from >=6.0.0 to >=8.0.0 - Add comprehensive test suites covering validation, SQL injection prevention, cascade type safety, hyphenated graph name integration, BigInt parsing, and setAGETypes error handling - Add design note in tests documenting why createExtension: false is the correct default (CI image has AGE pre-installed, auto-creating extensions requires SUPERUSER and conflates concerns) modified: drivers/nodejs/package.json modified: drivers/nodejs/src/antlr4/CustomAgTypeListener.ts modified: drivers/nodejs/src/index.ts modified: drivers/nodejs/test/Agtype.test.ts modified: drivers/nodejs/test/index.test.ts (cherry picked from commit 77a16ece0bbd8f137ad40b4e203c9622c249352d) * Fix null pointer handling in array iteration (#2313) Previously, when iterating through an agtype container, the code would access `elem->val` even when `elem` was null. This adds a null check to set the result type to AGTV_NULL when the element is null, preventing a potential segmentation fault. Fixes: 4274f10 ("Added the toStringList() function (#1084)") Found by PostgresPro. Signed-off-by: Maksim Korotkov <m.korotkov@postgrespro.ru> (cherry picked from commit 0c9a527860cde710175fc9c171b6bbe9aaed45d6) * Fix ISO C90 forbids mixed declarations and code warning (#2334) Fixed the following ISO C90 warning - src/backend/utils/adt/agtype.c:7503:9: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement] 7503 | enum agtype_value_type elem_type = elem ? elem->type : AGTV_NULL; | ^~~~ No regression tests impacted. modified: src/backend/utils/adt/agtype.c (cherry picked from commit fa91350b0a524e596e4fc85c9086565c60c00548) * Remove labeler github action (#2335) - dropped the labeler workflow because `pull_request_target` trigger can be dangerous and the workflow wasn’t particularly useful. (cherry picked from commit 3c4d9cc7456703f37d3f2d02e4cc79731b06542a) * fix incorrect variable assignment (#2336) Fixed an incorrect variable assignment, that was causing a warning message during compilation, on some compilers. The create_index_on_column function assigned InvalidOid, instead of NIL or NULL. - index_col->collation = InvalidOid; + index_col->collation = NIL; No regression tests were impacted. modified: src/backend/commands/label_commands.c (cherry picked from commit 55476ad2875f37f5b65919b6eeab7d81fd65aa7c) * Fix VLE NULL handling for chained OPTIONAL MATCH (#2337) * Fix VLE NULL handling for chained OPTIONAL MATCH (#2092) VLE functions (age_match_vle_terminal_edge, age_match_two_vle_edges, age_match_vle_edge_to_id_qual) threw errors when receiving NULL arguments from OPTIONAL MATCH (LEFT JOIN) contexts. Additionally, build_local_vle_context crashed with a segfault when dereferencing a NULL next_vertex pointer in the cached VLE context path. These functions are used as join quals. In a LEFT JOIN, NULL arguments mean the inner side produced no match. The correct response is FALSE (no match), which lets PostgreSQL emit NULL-extended rows — the expected OPTIONAL MATCH behavior. Errors or crashes are incorrect. Changes: - build_local_vle_context: guard against NULL next_vertex in cached path; return NULL when vertex list is exhausted - age_vle: handle NULL return from build_local_vle_context with SRF_RETURN_DONE - age_match_vle_terminal_edge: return FALSE on NULL arguments instead of ereport(ERROR) - age_match_two_vle_edges: return FALSE on NULL arguments - age_match_vle_edge_to_id_qual: return FALSE on NULL arguments All 32 regression tests pass including new tests for this fix. * Address review feedback: fix error message and add ORDER BY to tests - Fix errmsg in age_match_vle_terminal_edge() to use the correct function name (was age_match_terminal_edge) - Add ORDER BY p.name to regression test queries to avoid nondeterministic row ordering in expected output AI-assisted: Claude (Anthropic) was used in developing this fix. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> (cherry picked from commit 5005c21e5c2aa5daaca909fee7c4f9ed8ccdf984) * Fix VLE queries failing on read-only replicas (#2160) (#2345) The global graph cache used by VLE acquired ShareLock when scanning vertex and edge label tables to populate in-memory hashtables. On read-only replicas (standby servers in recovery), PostgreSQL only allows RowExclusiveLock or less, so VLE queries would fail with: "cannot acquire lock mode ShareLock on database objects while recovery is in progress" Change all three functions in age_global_graph.c to use AccessShareLock instead, which is sufficient for read-only table scans and is consistent with the existing ag_cache.c code that performs identical operations. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> (cherry picked from commit 346f319459e18db89deae30a3af7ea945bba101d) * Fix chained MERGE not seeing sibling MERGE's changes (#1446) (#2344) * Fix chained MERGE not seeing sibling MERGE's changes (#1446) When multiple MERGEs are chained (e.g. MATCH ... MERGE ... MERGE ...), the non-terminal (first) MERGE returned rows one at a time to the parent plan node. The parent MERGE's lateral join would materialize its hash table on the first row, before the child MERGE had finished all its iterations. This caused the second MERGE to not see entities created by the first MERGE, leading to duplicate nodes. Fix by making non-terminal MERGE eager: it processes ALL input rows and buffers the projected results before returning any to the parent. This ensures all entity creations are committed before any parent plan node scans the tables. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix non-terminal MERGE empty-buffer fallthrough and add test When a non-terminal MERGE receives no input rows from its predecessor (e.g., MATCH returns 0 rows), the eager buffer is filled but empty. The condition at line 688 checked `css->eager_tuples != NIL`, which evaluated to false for an empty buffer, causing execution to fall through to the terminal MERGE code path. This could incorrectly create entities when none should be created. Fix by checking `css->eager_buffer_filled` instead, which correctly distinguishes "buffer not yet filled" from "buffer filled but empty". Add regression test for chained MERGE with empty MATCH result. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> (cherry picked from commit 20ada845280c4370909d82b8652f6a082e5aa5df) * Fix MATCH after CREATE returning 0 rows (issue #2308) (#2340) When a MATCH clause follows CREATE + WITH and re-uses bound variables (e.g. CREATE (a)-[e]->(b) WITH a,e,b MATCH p=(a)-[e]->(b)), the MATCH generates filter quals (age_start_id(e) = age_id(a), etc.) that reference only columns from the predecessor subquery. PostgreSQL's optimizer pushes these quals through the transparent subquery layers into the CREATE's child plan, where they evaluate on NULL values before CREATE has executed — always yielding 0 rows. Fix: mark the predecessor subquery RTE as security_barrier when the clause chain contains a data-modifying operation (CREATE, SET, DELETE, or MERGE). This prevents PostgreSQL from pushing filter quals into the subquery, ensuring they evaluate after the DML produces output values. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> (cherry picked from commit 217467a36a1c29df4b918faf6adb6e75aec28817) * Support doubled-quote escaping in Cypher string literals (issue #2222) (#2342) SQL drivers (psycopg2, JDBC, etc.) escape single quotes by doubling them ('isn''t') when substituting parameters into queries. When these substitutions land inside Cypher's $$ block, the Cypher scanner rejects them because it only recognizes backslash escaping (\'). This makes it difficult to pass strings containing apostrophes through SQL drivers. Add '' and "" as escape sequences in the Cypher scanner, following the same pattern already used for backtick-quoted identifiers (``). Flex picks the longer two-character match over the one-character closing quote, so the change is backwards-compatible -- '' was previously a syntax error. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> (cherry picked from commit a21120bf139b156661ef10374630e688c0cc46fb) * Fix entity_exists() CID visibility for CREATE + WITH + MERGE (#2343) * Fix entity_exists() CID visibility for CREATE + WITH + MERGE (#1954) When a Cypher query chains CREATE ... WITH ... MERGE, vertices created by CREATE become invisible to entity_exists() after a threshold number of input rows. This causes MERGE to throw "vertex assigned to variable was deleted". Root cause: CREATE calls CommandCounterIncrement() which advances the global command ID, but does not update es_snapshot->curcid. The Decrement/Increment CID macros used by the executors bring curcid back to the same value on each iteration. After enough rows, newly inserted vertices have a Cmin >= curcid and HeapTupleSatisfiesMVCC rejects them (requires Cmin < curcid). Fix: In entity_exists(), temporarily set es_snapshot->curcid to the current global command ID (via GetCurrentCommandId) for the duration of the scan, then restore it. This makes all entities inserted by preceding clauses in the same query visible to the existence check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use Max() to prevent curcid regression in entity_exists() Address review feedback: es_snapshot->curcid can be ahead of the global CID due to Increment_Estate_CommandId macros. Unconditionally assigning GetCurrentCommandId(false) could decrease curcid, making previously visible tuples invisible. Use Max(saved_curcid, GetCurrentCommandId(false)) to ensure we only ever increase visibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> (cherry picked from commit d0741d8dc90f0c2f636f049b48b9d41eb54516cf) * Fix crash in PREPARE with property parameter when enable_containment is off (#2339) * Fix crash in PREPARE with property parameter when enable_containment is off When age.enable_containment is set to off, executing a PREPARE statement with a property parameter (e.g., MATCH (n $props) RETURN n) causes a segfault. The crash occurs in transform_map_to_ind_recursive because the property_constraints node is a cypher_param, not a cypher_map, but is blindly cast to cypher_map and its keyvals field is dereferenced. Three fixes: - In create_property_constraints, when enable_containment is off and the constraint is a cypher_param, fall back to the containment operator (@>) since map decomposition requires known keys at parse time. - In transform_match_entities, guard the keep_null assignment for both vertex and edge property constraints with is_ag_node checks to avoid writing to the wrong struct layout. Fixes #1964 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix @> vs @>> for =properties form with PREPARE and add tests When MATCH uses the =properties form (e.g., MATCH (n = $props)), the enable_containment=on path correctly uses @>> (top-level containment). The parameter fallback path unconditionally used @> (deep containment), ignoring the use_equals flag. Fix the fallback to mirror the enable_containment path by selecting @>> when use_equals is set. Add regression tests for =properties form with PREPARE for both vertices and edges, with enable_containment on and off. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> (cherry picked from commit 23146a44c36c0ece003b7b91f7942385ae8ac5e7) * Add extension upgrade template regression test (#2364) Note: This PR was created with AI tools and a human. Add a version-agnostic regression test (age_upgrade) that validates the upgrade template (age--<VER>--y.y.y.sql) works correctly by simulating a full extension version upgrade within "make installcheck". Add full upgrade scripts to the install path (DATA) in the Makefile, excluding template upgrade files. This enables the install to copy all version upgrade files into the PG AGE install. This is needed for ALTER EXTENSION Adjusted installcheck.yaml to allow git commit history for this test. Makefile infrastructure: - Build the install SQL (age--<CURR>.sql) from the initial version-bump commit in git history, so CREATE EXTENSION installs "day-one" SQL while the .so comes from current HEAD — implicitly testing backward compat. - Build a synthetic "next" version (age--<NEXT>.sql) from HEAD and stamp the upgrade template to produce age--<CURR>--<NEXT>.sql. - Add an installcheck prerequisite that temporarily installs both synthetic files into the PG extension directory; a generated cleanup script removes them at the end of the test via \! shell escape. EXTRA_CLEAN catches stragglers on "make clean". - Skip the test automatically when: (a) no git history (tarball builds), (b) no upgrade template exists, or (c) a real upgrade script from the current version is already committed (detected via git ls-files).o Regression test (regress/sql/age_upgrade.sql): - Creates 3 graphs (company, network, routes) with 8 vertex labels, 8 edge labels, 23 vertices, 28 edges, and 4 GIN indexes. - Records integrity checksums (agtype sums), vertex/edge counts, and label counts before the upgrade; repeats all checks after ALTER EXTENSION UPDATE to the synthetic next version. - Verifies structural queries: VLE management chains, circular follow chains, flight distances with edge properties. - Verifies all 4 GIN indexes survive the upgrade via pg_indexes. - Uses ORDER BY on all multi-row queries for deterministic output. - Returns agtype natively (no ::numeric casts) for portability. - Avoids version-dependent output (checks boolean IS NOT NULL instead of printing the version string). - Uses JOIN-based label counts to avoid NULL comparison bugs with the internal _ag_catalog graph. - Cleans up all 3 graphs and restores the default AGE version. modified: Makefile new file: regress/expected/age_upgrade.out new file: regress/sql/age_upgrade.sql modified: .github/workflows/installcheck.yaml (cherry picked from commit 90c33eb6b7c36f8280893bdc2b64666debb836de) * Fix nondeterministic age_global_graph regression test (#2365) The age_global_graph test had two issues that could cause intermittent failures: 1. Nondeterministic warning output: The graph_stats() call on a graph with deliberately deleted vertices produces WARNING messages for dangling edges. These warnings are emitted by iterating edge label tables (knows, stalks), and the iteration order is not guaranteed. Since PostgreSQL WARNING messages cannot be caught or counted from SQL (only ERROR and above are catchable via PL/pgSQL exception handling), we suppress them with SET client_min_messages = error. The suppressed warnings are documented verbatim in comments. The graph_stats() result row still validates correct dangling-edge handling. 2. Nondeterministic row ordering: Multiple MATCH...RETURN queries returned multi-row results without ORDER BY, relying on scan order. Added ORDER BY id(u), id(v), id(e), id(n), or id(a) as appropriate to all MATCH...RETURN queries for future-proofing, even those currently returning a single row. Files changed: regress/sql/age_global_graph.sql regress/expected/age_global_graph.out Co-authored-by: GitHub Copilot <noreply@github.com> (cherry picked from commit a29e2810b1ac5a9e4c3e553879b683a6a1902c8b) * Fix MATCH on brand-new label after CREATE returning 0 rows (#2341) * Fix MATCH on brand-new label after CREATE returning 0 rows (issue #2193) When CREATE introduces a new label and a subsequent MATCH references it (e.g., CREATE (:Person) WITH ... MATCH (p:Person)), the query returns 0 rows on first execution but works on the second. Root cause: match_check_valid_label() in transform_cypher_match() runs before transform_prev_cypher_clause() processes the predecessor chain. Since CREATE has not yet executed its transform (which creates the label table as a side effect), the label is not in the cache and the check generates a One-Time Filter: false plan that returns no rows. Fix: Skip the early label validity check when the predecessor clause chain contains a data-modifying operation (CREATE, SET, DELETE, MERGE). After transform_prev_cypher_clause() completes and any new labels exist in the cache, run a deferred label check. If the labels are still invalid at that point, generate an empty result via makeBoolConst(false). This preserves the existing behavior for MATCH without DML predecessors (e.g., MATCH-MATCH chains still get the early check and proper error messages for invalid labels). Depends on: PR #2340 (clause_chain_has_dml helper) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address review feedback: fix variable registration for deferred label check When the deferred label validity check (DML predecessor + non-existent label) found an invalid label, the code skipped transform_match_pattern() entirely, which meant MATCH-introduced variables were never registered in the namespace. This would cause errors if a later clause referenced those variables (e.g., RETURN p). Fix: mirror the early-check strategy by injecting a paradoxical WHERE (true = false) and always calling transform_match_pattern(). Variables get registered normally; zero rows are returned via the impossible qual. Also add ORDER BY to multi-row regression tests for deterministic output, and add a test case for DML predecessor + non-existent label + returning a MATCH-introduced variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address Copilot review: DRY false-where helper, cache has_dml, ORDER BY in tests - Factor duplicated WHERE true=false construction into make_false_where_clause() helper (used in both early and deferred label validation paths) - Compute clause_chain_has_dml() once and reuse, avoiding repeated clause chain traversal - Add ORDER BY to the single-CREATE City regression test for deterministic result ordering * Address Copilot review: volatile false predicate, DML side-effect test 1. Prevent plan elimination of DML predecessor: replace constant (true = false) with volatile (random() IS NULL) in the deferred label check path. PG's planner can constant-fold the former into a One-Time Filter: false, skipping the DML scan entirely. 2. Unify make_false_where_clause(bool volatile_needed): merge the constant and volatile variants into a single parameterized function. Call sites are now self-documenting: - make_false_where_clause(false) for non-DML path - make_false_where_clause(true) for DML predecessor path 3. Document why add_volatile_wrapper() cannot be reused here (it operates post-transform at the Expr level and returns agtype, while the WHERE clause is built at the parse-tree level). 4. Add regression test verifying CREATE side effects persist when MATCH references a non-existent label after a DML predecessor. All regression tests pass (cypher_match: ok). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Replace non-ASCII em dashes with -- in C comments ASCII-only codebase convention; avoids encoding/tooling issues. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> (cherry picked from commit d6f1b7f67087bd86f2e0fe6e1b6c897b7e569926) * Add index scan (#2351) * Add index scan This commit fixes performance degradation during insertion scenarios by replacing SeqScan with IndexScan. Motivation / Problem: As a result of load testing, a significant performance degradation was found in insertion scenarios. The scenarios used were taken from an open-source benchmark and rewritten in pure SQL. Examples of the queries can be found here: https://github.com/ldbc/ldbc_snb_interactive_v1_impls/blob/main/cypher/queries/interactive-update-1.cypher https://github.com/ldbc/ldbc_snb_interactive_v1_impls/blob/main/cypher/queries/interactive-update-6.cypher https://github.com/ldbc/ldbc_snb_interactive_v1_impls/blob/main/cypher/queries/interactive-update-7.cypher Perf analysis showed that the main bottleneck is the entity_exists function. The root cause lies in the use of a Sequential Scan (SeqScan) to check for the existence of an entity prior to insertion. The time complexity of a SeqScan is O(N), meaning the search time grows linearly as the number of rows in the table increases. The larger the graph became, the longer each individual insertion took. This led to a drop in TPS regardless of the concurrency level (the issue was consistently reproduced with both 1 and 30 threads). Co-authored-by: Daria Barsukova <d.barsukova@g.nsu.ru> Co-authored-by: Alexandra Bondar <s6311704@gmail.com> * Address code review commets * Address code review commets * Add brackets --------- Co-authored-by: Alexandra Bondar <a.bondar@postgrespro.ru> Co-authored-by: Daria Barsukova <d.barsukova@g.nsu.ru> PG15 adaptation: use the native five-argument index_beginscan API and include the index/buffer headers explicitly. (cherry picked from commit b3a00eea32c3faa93c43e0d91c0a75671e48ce65) * Add missing include for PR: Add index scan (#2351) (#2379) The CI build did not fail the PR (we need to verify why and correct it) due to the missing include - src/backend/utils/adt/age_global_graph.c:273:25: error: implicit declaration of function ‘namestrcpy’; did you mean ‘strcpy’? [-Wimplicit-function-declaration] 273 | namestrcpy(lval, NameStr(*label_name_ptr)); | ^~~~~~~~~~ | strcpy The build still works as the linker was able to resolve it. This PR will add it in to correct the error going forward. (cherry picked from commit 945a259d86d42204f186f79dec758585afde0eaa) * Improve extension upgrade regression test (addendum to #2364) (#2377) Note: This PR was created with AI tools and a human. This is an addendum to PR #2364 with three improvements. Makefile: - Replace awk-based synthetic version (minor+1) with an _upgrade_test suffix (e.g., 1.7.0 -> 1.7.0_upgrade_test). The awk approach produced numeric versions like 1.8.0 that could collide with real future upgrade scripts, and the ::int[] cast in the SQL version lookup fails on non-numeric version strings. The _upgrade_test suffix avoids both issues and is unambiguously synthetic. - Extend the generated cleanup script to also remove repo-root copies of the synthetic files and to self-delete, preventing stale artifacts from accumulating across repeated test runs. Regression test (regress/sql/age_upgrade.sql): - Simplify version lookup to directly select the _upgrade_test version via LIKE '%_upgrade_test' instead of picking the highest non-default version with string_to_array(version, '.')::int[] DESC. The old approach would fail with a cast error on the _upgrade_test suffix and was unnecessarily indirect — the test knows exactly what synthetic version the Makefile installed. modified: Makefile modified: regress/expected/age_upgrade.out modified: regress/sql/age_upgrade.sql Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit 1847644cf7d2f52486f3a6dba90317ad72f178c7) * Fix upgrade test: build default install SQL from HEAD, not initial commit (#2397) The upgrade test previously built age--<CURR>.sql from the initial version-bump commit, meaning CREATE EXTENSION installed 'day-one' SQL. This caused all 31 non-upgrade regression tests to run WITHOUT SQL functions added after the version bump (e.g., age_invalidate_graph_cache, age_prepare_pg_upgrade, age_vertex_stats, etc.). These functions were never registered in pg_proc, so features depending on them (like VLE cache invalidation triggers) were silently disabled during testing. Fix by inverting the upgrade test direction: Before (installs incomplete SQL, upgrades to complete): age--1.7.0.sql built from version-bump commit (incomplete) age--1.7.0_upgrade_test.sql built from HEAD (complete) age--1.7.0--1.7.0_upgrade_test.sql stamped from template Test: CREATE EXTENSION age -> data -> ALTER EXTENSION UPDATE TO '1.7.0_upgrade_test' After (installs complete SQL, upgrades from synthetic initial): age--1.7.0.sql built from HEAD (complete) age--1.7.0_initial.sql built from version-bump commit (synthetic) age--1.7.0_initial--1.7.0.sql stamped from template Test: CREATE EXTENSION age VERSION '1.7.0_initial' -> data -> ALTER EXTENSION UPDATE TO '1.7.0' This ensures: - All 31 non-upgrade tests run with every SQL function registered - The upgrade template is still validated (initial -> current) - The test ends at the default version (clean state for drop test) - Tarball builds (no git) work identically (cat sql/sql_files) - All synthetic files are cleaned up after the test Makefile changes: - age_sql rule: cat current HEAD sql/sql_files (was: git show from commit) - New age_init_sql rule: git show from version-bump commit - age_upgrade_test_sql: stamps template as INIT->CURR (was CURR->NEXT) - EXTRA_CLEAN, _install_upgrade_test_files: updated filenames - Removed AGE_NEXT_VER, age_next_sql; added AGE_INIT_VER, age_init_sql Test SQL changes: - Step 3: CREATE EXTENSION age VERSION '<init>' (was: CREATE EXTENSION age) - Step 6: ALTER EXTENSION age UPDATE TO default_version (was: LIKE '%_upgrade_test') - Step 7: Check installed = default (was: installed <> default) - Comments updated throughout to reflect inverted direction All 32 regression tests pass. modified: Makefile modified: regress/expected/age_upgrade.out modified: regress/sql/age_upgrade.sql (cherry picked from commit f1a9b1d9b8b171656efd82674dec2e6ce06b8f9a) * Python driver: Add `skip_load` parameter to skip `LOAD 'age'` statement (#2366) * Add skip_load parameter to connect and setUpAge functions to control plugin loading. * Add tests and README docs for skip_load parameter - Add 4 unit tests for setUpAge() skip_load behavior: skip_load=True skips LOAD, skip_load=False executes LOAD, load_from_plugins integration, and search_path always set. - Document skip_load in README under new "Managed PostgreSQL Usage" section for Azure/AWS RDS/etc. environments. - Fix syntax in existing load_from_plugins code example. Made-with: Cursor * Address review feedback: ValueError for contradictory flags, e2e test - Raise ValueError when skip_load=True and load_from_plugins=True are both set (contradictory combination) - Add end-to-end test verifying skip_load is forwarded through the full age.connect() → Age.connect() → setUpAge() call chain - Replace fragile string assertions with assert_called_with/assert_any_call - README: mention configure_connection() as the pool-based alternative for managed PostgreSQL environments Made-with: Cursor * Fix README: use setUpAge(skip_load=True) for pool example configure_connection() is not part of this PR; use the available setUpAge() API with skip_load=True for the connection pool example. Made-with: Cursor * fix(python-driver): use quoted $user in search_path and run TestSetUpAge in CI PostgreSQL treats single-quoted '$user' as a literal schema name; use "$user" so the session user schema is included. Include TestSetUpAge in the test_age_py __main__ suite so skip_load tests run in GitHub Actions. Made-with: Cursor (cherry picked from commit ce5004593ea5acb2cec3a2dc94a06241d7f23c17) * Fix OPTIONAL MATCH dropping null-preserving rows with subquery WHERE (#2380) * Fix OPTIONAL MATCH dropping null-preserving rows with subquery WHERE Cypher OPTIONAL MATCH semantics require that when no right-hand row survives the WHERE predicate, the outer row is still emitted with NULLs in the optional columns. Before this fix, a WHERE containing a list comprehension or sub-pattern predicate (EXISTS { ... }, COUNT { ... }) would take the transform_cypher_clause_with_where rewrite path, which detaches the WHERE, transforms the match clause as a subquery, and then attaches the WHERE as an outer filter on that subquery. For OPTIONAL MATCH, the inner subquery already produced a LATERAL LEFT JOIN with null-preserving rows; the outer filter then ran against those nulled rows and dropped them when the predicate evaluated NULL or false on the nulled side, producing zero rows where Cypher semantics require one null-filled row per outer match. Fix: in transform_cypher_match, the has_list_comp_or_subquery rewrite now only applies to non-optional MATCH. In the OPTIONAL MATCH path, transform_cypher_optional_match_clause detaches the WHERE from the cypher_match node before recursively transforming the right-hand side (so the inner transform does not double-apply or misresolve the predicate in a fresh namespace), and re-attaches the transformed predicate as the LEFT JOIN's ON condition after both sides are in the namespace. A LEFT JOIN with a failing ON condition correctly preserves left rows with null right columns, which matches Cypher OPTIONAL MATCH ... WHERE semantics. Regression tests cover: - EXISTS { (friend)-[...]->(...) } referencing the optional variable - EXISTS { (p)-[...]->(...) } referencing the outer variable - non-correlated EXISTS (previously-working guard) - plain scalar predicate on the optional variable (guard) - constant-false WHERE (guard) Fixes issue #2378. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthro…

jrgemignani requested a review from MuhammadTahaNaveed September 9, 2025 19:14

github-actions Bot added master override-stale To keep issues/PRs untouched from stale action labels Sep 9, 2025

jrgemignani force-pushed the fix_call_yield_bug branch from 004c66c to 729ff62 Compare September 9, 2025 22:49

MuhammadTahaNaveed approved these changes Sep 15, 2025

View reviewed changes

MuhammadTahaNaveed merged commit 94770a6 into apache:master Sep 15, 2025
7 checks passed

jrgemignani mentioned this pull request Jan 30, 2026

Update PG17 to master prior to 1.7.0 #2325

Merged

Navigation Menu

Fix issue with CALL/YIELD for user defined and qualified functions. - #2217

Fix issue with CALL/YIELD for user defined and qualified functions.#2217
MuhammadTahaNaveed merged 1 commit into
apache:masterfrom
jrgemignani:fix_call_yield_bug

jrgemignani commented Sep 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jrgemignani commented Sep 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants