Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Attempt to fix issue #7 #9

Merged
merged 6 commits into from
Dec 30, 2024
Merged

Attempt to fix issue #7 #9

merged 6 commits into from
Dec 30, 2024

Conversation

kkew3
Copy link
Owner

@kkew3 kkew3 commented Dec 30, 2024 •
edited
Loading

Introduce the following changes to fix #7:

  • Add support to spacing modifier letters, combining diacritical marks, and emoji in tokenization.
  • Breaking change: Simplify Chinese punctuation association rule. This involves keeping only a number of essential right-punctuation like Chinese comma and full-stop, and removing all others. The goal is to be more consistent with Vim's default behavior. Only WORD motions are affected.

kkew3 added 6 commits December 29, 2024 10:36
@kkew3
Add support to spacing modifier letters in tokenization
977a58e 
Related to issue #7.
@kkew3
Add support to combining diacritical marks in tokenization
11d0cf5 
Related to issue #7.
@kkew3
Refactor char grouping code to leverage rust's pattern matching exhau…
19b7611 
…stive check
@kkew3
Simplify Chinese punctuation association rule
e23c0b7 
Keep only some of the right punctuations, e.g. full-width comma and full-stop. All other punctuation are regarded "other type", e.g. full-width parenthesis and dashes. This way, emoji will then be considered similarly to non-word other type characters in a consistent manner.
@kkew3
Add support to emoji in tokenization
8b89339 
Related to issue #7.
@kkew3
Bump the version of jieba_vim_rs_core
699d9ba
@kkew3 kkew3 merged commit e3e8dfd into rust Dec 30, 2024
6 checks passed
@kkew3 kkew3 deleted the issue/7 branch December 30, 2024 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support spacing modifier letters, combining diacritical marks and emoji
1 participant
@kkew3