Add initial native Rust runtime (packages/core-rust) for mounting and querying .knolo packs#31
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 290ba30b9a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| fn compact(s: &str) -> String { | ||
| s.chars().filter(|c| !c.is_whitespace()).collect() | ||
| } |
There was a problem hiding this comment.
parse_blocks calls compact(json) before extracting text, heading, docId, and namespace, and compact removes every whitespace character globally. Because block text is serialized as JSON strings from buildPack, this collapses normal spaces/newlines inside document content (e.g., "alpha beta" becomes "alphabeta"), corrupting returned snippets and fallback token-length calculation for packs without len values.
Useful? React with 👍 / 👎.
| if *ch == '{' { | ||
| if depth == 0 { | ||
| start = Some(i); | ||
| } | ||
| depth += 1; | ||
| } else if *ch == '}' { | ||
| depth -= 1; |
There was a problem hiding this comment.
split_top_level_objects increments/decrements depth on every {/} character without tracking whether the parser is currently inside a quoted string. If a block's text contains braces (common in code snippets like { "a": 1 }), object boundary detection becomes incorrect and can produce malformed object slices or mounting failures.
Useful? React with 👍 / 👎.
| let end = rest.find('"')?; | ||
| Some(unescape(&rest[..end])) |
There was a problem hiding this comment.
parse_string_or_null finds the first " after the opening quote and treats it as the end of the value. This breaks valid JSON strings containing escaped quotes (e.g., \"), truncating text/heading/docId/namespace at the first escaped quote and returning corrupted metadata and snippets.
Useful? React with 👍 / 👎.
1 participant
Motivation
.knolopacks for improved portability and performance.Description
packages/core-rustwithCargo.toml,src/lib.rs,README.md, and.gitignoreimplementingmount_pack_from_bytesandqueryalong with pack parsing formeta,lexicon,postings, andblocksformats.top_k,min_score,namespace, andsourcefilters.packages/core-rust/tests/core_rust_test.rsthat build a synthetic pack and verify mounting, ranking, and namespace filtering.README.mdto advertise the new Rust runtime and addtest:rusttopackage.jsonto run Rust tests viacargo test.Testing
cargo testinpackages/core-rustand the test suite passed.test:rustnpm script (cargo test --manifest-path packages/core-rust/Cargo.toml) was added for CI integration.Codex Task