add-prefix-suffix-ngram-udf #438
Conversation
|
why there is |
fuction-1 include item docs listed in fuction |
|
|
||
| -- N-gram approach (fast - with prefiltering) | ||
| SELECT * FROM documents | ||
| WHERE NGRAM(content, 2) LIKE '%ap%' |
There was a problem hiding this comment.
Why would we use LIKE here? We can simply do NGRAM(content, 2) = 'ap'?
There was a problem hiding this comment.
thank you Ankit, would address this
| | -------- | | ||
| | fas,ast | | ||
|
|
||
| ### Prefix Operations |
There was a problem hiding this comment.
remove this - is this unrelated.
There was a problem hiding this comment.
not sure I follow, why is this unrelated? we also want to doc for prefix?
| | ------------ | | ||
| | Apache Pinot | | ||
|
|
||
| ### Advanced Text Matching with N-grams |
There was a problem hiding this comment.
i think this is unrelated.
|
|
||
| ## Signature | ||
|
|
||
| > NGRAM(col, n) |
|
|
||
| > NGRAM(col, n) | ||
| > | ||
| > PREFIX(col, prefixString) |
| > | ||
| > PREFIX(col, prefixString) | ||
| > | ||
| > SUFFIX(col, suffixString) |
| ```sql | ||
| SELECT NGRAM('Apache pinot', 2) AS bigrams | ||
| FROM myTable | ||
| ``` |
There was a problem hiding this comment.
add a tableconfig with "transformationConfig" instead of sql UDF
update the function ngram to unique_ngrams
musclemuller
force-pushed
the
muller/add-prefix-suffix-ngram-udf
branch
from
001a742 to
de5c295
Compare
|
|
||
| ## Context | ||
|
|
||
| We are onboarding a use case and trying to increase query throughput. We tested that QPS cannot be further improved with existing `REGEXP_LIKE` queries or `TEXT_MATCH` queries. The queries are as follows: |
There was a problem hiding this comment.
update this section to the following
-->
We were aiming to increase query throughput for a matching use case. Initial testing shows that the current approaches using REGEXP_LIKE and TEXT_MATCH queries have reached their performance limits, and further improvements in QPS cannot be achieved with these methods. The representative queries under evaluation are as follows:
| ``` | ||
|
|
||
| #### Query Examples | ||
|
|
There was a problem hiding this comment.
need to add two query example using generated ngrams
example1.
Select * from T where regexp_like(field, “*pino.*”) ==>
Select * from T
where ngram = 'pin' and ngram = 'ino' and ngram = 'not'
and regexp_like(field, “*pino.*”)
example2.
In some cases where the search string is shorter or equal to the ngram, we don’t need a validation stage.
Select * from T where regexp_like(field, “*pi.*”) ==>
Select * from T where ngram = 'pi’
9361aa7 to
fc04681
Compare
3 participants
New doc for how to add prefix, suffix and ngram UDFs
feature pr: apache/pinot#12392