feat: Lucene query parser with sql and dynamodb support

[Lutherwaves]

Implements Apache Lucene query syntax parser by extending https://github.com/grindlemire/go-lucene with custom drivers for SQL (JSONB) and DynamoDB PartiQL.

Supports field:value queries, wildcards, ranges, boolean operators, quoted phrases, fuzzy search, implicit search support for text fields only, and JSONB columns support. Includes field validation, security limits, and added tests.

---

PostgreSQL

pg_trgm - Required for fuzzy search functionality (similarity() function)

CREATE EXTENSION IF NOT EXISTS pg_trgm;

JSONB support - Built-in for PostgreSQL 9.4+ (no extension needed)

MySQL

JSON functions - Built-in for MySQL 5.7+ (no extension needed)
SOUNDEX() - Built-in function (no extension needed)

SQLite

JSON1 extension - Required for JSON field queries (JSON_EXTRACT())
    Usually compiled by default in SQLite 3.38.0+
    Verify with: SELECT json_valid('{}');
    If not available, recompile SQLite with -DSQLITE_ENABLE_JSON1
Note: Fuzzy search is not supported (no extension available)

DynamoDB

No extensions required (fully managed service)

---

Example Queries and SQL Translation

Basic Field Search

Query: name:john

PostgreSQL: "name" = $1 (params: ["john"])
MySQL: "name" = ? (params: ["john"])
SQLite: "name" = ? (params: ["john"])
DynamoDB: "name" = ? (params: ["john"])

Wildcard Search (Case-Insensitive)

Query: name:john*

PostgreSQL: "name"::text ILIKE $1 (params: ["john%"])
MySQL: LOWER("name") LIKE LOWER(?) (params: ["john%"])
SQLite: "name" LIKE ? (params: ["john%"])
DynamoDB: begins_with("name", 'john')

Wildcard Contains

Query: name:john

PostgreSQL: "name"::text ILIKE $1 (params: ["%john%"])
MySQL: LOWER("name") LIKE LOWER(?) (params: ["%john%"])
SQLite: "name" LIKE ? (params: ["%john%"])
DynamoDB: contains("name", 'john')

Fuzzy Search

Query: name:roam~2

PostgreSQL: similarity("name"::text, $1) > 0.3 (params: ["roam"])
MySQL: SOUNDEX("name") = SOUNDEX(?) (params: ["roam"])
SQLite: ❌ Error: "fuzzy search not supported with SQLite"
DynamoDB: ❌ Not supported

JSON Field Access

Query: labels.env:prod

PostgreSQL: labels->>'env' = $1 (params: ["prod"])
MySQL: JSON_UNQUOTE(JSON_EXTRACT(labels, '$.env')) = ? (params: ["prod"])
SQLite: JSON_EXTRACT(labels, '$.env') = ? (params: ["prod"])
DynamoDB: labels.env = ? (params: ["prod"])

JSON Field Wildcard

Query: labels.category:backend*

PostgreSQL: labels->>'category' ILIKE $1 (params: ["backend%"])
MySQL: LOWER(JSON_UNQUOTE(JSON_EXTRACT(labels, '$.category'))) LIKE LOWER(?) (params: ["backend%"])
SQLite: JSON_EXTRACT(labels, '$.category') LIKE ? (params: ["backend%"])
DynamoDB: begins_with(labels.category, 'backend')

Boolean AND

Query: name:john AND status:active

PostgreSQL: ("name" = $1) AND ("status" = $2) (params: ["john", "active"])
MySQL: ("name" = ?) AND ("status" = ?) (params: ["john", "active"])
SQLite: ("name" = ?) AND ("status" = ?) (params: ["john", "active"])
DynamoDB: ("name" = ?) AND ("status" = ?) (params: ["john", "active"])

Boolean OR

Query: status:active OR status:pending

PostgreSQL: ("status" = $1) OR ("status" = $2) (params: ["active", "pending"])
MySQL: ("status" = ?) OR ("status" = ?) (params: ["active", "pending"])
SQLite: ("status" = ?) OR ("status" = ?) (params: ["active", "pending"])
DynamoDB: ("status" = ?) OR ("status" = ?) (params: ["active", "pending"])

Range Query (Inclusive)

Query: age:[18 TO 65]

PostgreSQL: "age" BETWEEN $1 AND $2 (params: ["18", "65"])
MySQL: "age" BETWEEN ? AND ? (params: ["18", "65"])
SQLite: "age" BETWEEN ? AND ? (params: ["18", "65"])
DynamoDB: "age" BETWEEN ? AND ? (params: ["18", "65"])

Range Query (Open-Ended)

Query: age:[18 TO *]

PostgreSQL: "age" >= $1 (params: ["18"])
MySQL: "age" >= ? (params: ["18"])
SQLite: "age" >= ? (params: ["18"])
DynamoDB: "age" >= ? (params: ["18"])

Null Check

Query: parent_id:null

PostgreSQL: "parent_id" IS NULL
MySQL: "parent_id" IS NULL
SQLite: "parent_id" IS NULL
DynamoDB: ❌ Not supported

NOT Operator

Query: NOT status:deleted

PostgreSQL: NOT ("status" = $1) (params: ["deleted"])
MySQL: NOT ("status" = ?) (params: ["deleted"])
SQLite: NOT ("status" = ?) (params: ["deleted"])
DynamoDB: NOT ("status" = ?) (params: ["deleted"])

Complex Query

Query: (name:john* OR email:*@example.com) AND status:active

PostgreSQL: (("name"::text ILIKE $1) OR ("email"::text ILIKE $2)) AND ("status" = $3) (params: ["john%", "%@example.com", "active"])
MySQL: ((LOWER("name") LIKE LOWER(?)) OR (LOWER("email") LIKE LOWER(?))) AND ("status" = ?) (params: ["john%", "%@example.com", "active"])
SQLite: (("name" LIKE ?) OR ("email" LIKE ?)) AND ("status" = ?) (params: ["john%", "%@example.com", "active"])
DynamoDB: ((begins_with("name", 'john')) OR (contains("email", '@example.com'))) AND ("status" = ?) (params: ["active"])

[Lutherwaves]

Moving to draft to refactor a bit for better readability and consistency

[Lutherwaves]

@deanefrati @ayashjorden if anyone has some time to review, I also tested on a service using PSQL - all works as expected, see description for full examples.

[ayashjorden]

LGTM, Please add unit-tests

[Lutherwaves]

One additional thing I will be adding is the ability to instatiate the parser with some fields which are excluded from implicit search for multi-tenancy

[Lutherwaves]

LGTM, Please add unit-tests

@ayashjorden Ready

[ayashjorden]

Havan't read upstream docs, just make sure that we document this, both in func comments and in Magic README.md (or the appropriate sub-readme

[Lutherwaves]

Good catch, I think it would be better to error out if there are conflicting types, making hte query invalid.

[ayashjorden]

@Lutherwaves, This is a mammoth of a PR, I've added static-is comments suggesting simplifications or brevity.
Not sure how to go about approving it, @deanefrati as a production user, if you have an idea?

Updates were made, tests were added, I'm ok with the PR, but Dean needs to also take a look as he's using it in several places.

[Lutherwaves]

@ayashjorden thanks for the review, I will look into the suggestions. WRT the size - splitting this PR just to make it smaller, while I agree with the intention behind it, seems overkill in this case. This is a new, backwards compatible extension of the previous feature and the majority of lines come from the unit tests.

[ayashjorden]

I didn't hint to split it, just stated that it's a mammoth to review

…

On Sun, Jan 18, 2026, 10:23 PM Martin Yankov ***@***.***> wrote: *Lutherwaves* left a comment (tink3rlabs/magic#124) <#124 (comment)> @ayashjorden <https://github.com/ayashjorden> thanks for the review, I will look into the suggestions. WRT the size - splitting this PR just to make it smaller, while I agree with the intention behind it, seems overkill in this case. This is a new, backwards compatible extension of the previous feature and the majority of lines come from the unit tests. — Reply to this email directly, view it on GitHub <#124 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABACOU4G4YYQJIA7W4Y2IM34HRZ6NAVCNFSM6AAAAACQDYAUI2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTONRWGYYDENZWGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Lutherwaves]

It's literally an eye strain to review, I really appreciate you taking the time.

[Lutherwaves]

@ayashjorden to the best of my prompt & review skills, I addressed the review.

[deanefrati]

That's indeed a behemoth of a PR :) would you be able to add some usage instructions for developers for how to use this new functionality? I would like to test it in one of my services to make sure it works in real life...

[Lutherwaves]

That's indeed a behemoth of a PR :) would you be able to add some usage instructions for developers for how to use this new functionality? I would like to test it in one of my services to make sure it works in real life...

It is basically the StorageAdapter.Search method now accepts proper lucene. I have tested it extensively on postgresql and somewhat with sqlite, but not in dynamodb. Check the PR description for all examples. Happy to prompt up some docs before we merge if you think this would add value.