Fix bm25query varlena detoasting for binary I/O and scoring #123
+127
−14
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tjgreen42
commented
Jan 9, 2026
tjgreen42
force-pushed
the
fix-bm25query-binary-io
branch
from
a6eb348 to
8ac399b
Compare
Collaborator
Author
|
Addressed - reverted the |
Several functions accessed TpQuery struct fields without properly detoasting the varlena, which caused data corruption when Postgres used short (1-byte) varlena headers instead of standard 4-byte headers. Fixed functions: - tpquery_out: use PG_DETOAST_DATUM for text output - tpquery_send: use PG_DETOAST_DATUM and proper pq_begintypsend - tpquery_recv: use pq_getmsgbyte for version (1 byte) - bm25_text_bm25query_score: use PG_DETOAST_DATUM for query - tpquery_eq: use PG_DETOAST_DATUM for both arguments Also fixed tpquery_send to use pq_begintypsend() instead of makeStringInfo() for proper binary protocol formatting. Testing: Added binary_io.sql test for COPY BINARY round-trip.
tjgreen42
force-pushed
the
fix-bm25query-binary-io
branch
from
8ac399b to
7213a78
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
tpquery_out,tpquery_send,bm25_text_bm25query_score, andtpquery_eqnow properly detoast varlena argumentstpquery_sendto usepq_begintypsend()for proper binary protocol formattingtpquery_recvto usepq_getmsgbyte()for version fieldBackground
When Postgres stores small varlena values, it may use a 1-byte header instead of the standard 4-byte header. Functions using
PG_GETARG_POINTERinstead ofPG_DETOAST_DATUMwould access struct fields at incorrect offsets, causing data corruption.This manifested as:
tpquery_outTesting
binary_io.sqltest that verifies COPY BINARY round-trip for bm25query type