Skip to content

Commit 2ab9bb3

Browse files
committed
Address Greptile review on #669: single-source the CJK punctuation sets
The CJK terminator and closer codepoint lists were restated in the reconciler's phrase policy and again in SentenceBoundaryClassifier's closer walk, so adding a codepoint required parallel edits with no compiler enforcement. The two primitive sets are now one internal Character extension in the reconciler file, and both the phrase policy and the classifier compose from them.
1 parent 2acd2c6 commit 2ab9bb3

2 files changed

Lines changed: 31 additions & 30 deletions

File tree

Cotabby/Support/SentenceBoundaryClassifier.swift

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -39,12 +39,11 @@ enum SentenceBoundaryClassifier {
3939
switch text[lastIndex] {
4040
case "!", "?":
4141
return true
42-
// CJK sentence terminators: the ideographic full stop, fullwidth `!` `?`, and the halfwidth
43-
// ideographic stop. Unlike the ASCII period these are unambiguous (they never mark decimals,
44-
// list numbers, or abbreviations), so they are terminal without classifier disambiguation.
45-
// Without these a Japanese completion never registers a sentence end and generation always
46-
// runs to the token budget, which is why CJK suggestions came out so long.
47-
case "\u{3002}", "\u{FF01}", "\u{FF1F}", "\u{FF61}":
42+
// The shared CJK terminator set (see `Character.isCJKSentenceTerminator`): unambiguous, so
43+
// terminal without the period disambiguation below. Without these a Japanese completion never
44+
// registers a sentence end and generation always runs to the token budget, which is why CJK
45+
// suggestions came out so long.
46+
case let character where character.isCJKSentenceTerminator:
4847
return true
4948
case ".":
5049
return isTerminalPeriod(in: text, at: lastIndex)
@@ -104,14 +103,12 @@ enum SentenceBoundaryClassifier {
104103

105104
private extension Character {
106105
/// Closing punctuation that may follow a sentence terminator: straight and curly quotes,
107-
/// parentheses, square brackets, and braces, plus the CJK closers (corner brackets, fullwidth
108-
/// parenthesis, lenticular and angle brackets). `endsSentence` walks back past a run of these to
109-
/// find the real terminator underneath, so `"done."`, `(stop!)`, and `終わり。」` register as
106+
/// parentheses, square brackets, and braces, plus the shared CJK closer set (see
107+
/// `Character.isCJKClosingPunctuation`). `endsSentence` walks back past a run of these to find
108+
/// the real terminator underneath, so `"done."`, `(stop!)`, and `終わり。」` register as
110109
/// sentence ends.
111110
var isSentenceClosingPunctuation: Bool {
112111
self == "\"" || self == "'" || self == ")" || self == "]" || self == "}"
113-
|| self == "\u{201D}" || self == "\u{2019}"
114-
|| self == "\u{300D}" || self == "\u{300F}" || self == "\u{FF09}"
115-
|| self == "\u{3011}" || self == "\u{3009}" || self == "\u{300B}" || self == "\u{FF63}"
112+
|| self == "\u{201D}" || self == "\u{2019}" || isCJKClosingPunctuation
116113
}
117114
}

Cotabby/Support/SuggestionSessionReconciler.swift

Lines changed: 22 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -562,6 +562,28 @@ private extension String {
562562
}
563563
}
564564

565+
/// The CJK punctuation primitives, internal because they are the single source of truth shared by
566+
/// this file's acceptance policy and `SentenceBoundaryClassifier`'s sentence-end detection. Adding a
567+
/// codepoint here updates phrase boundaries, chunk binding, and the generation stop in one edit.
568+
extension Character {
569+
/// The CJK sentence terminators: ideographic full stop `。`, fullwidth `!` `?`, and the halfwidth
570+
/// ideographic stop `。`. Unlike the ASCII period these are unambiguous (they never mark decimals,
571+
/// list numbers, or abbreviations), so every consumer treats them as terminal without classifier
572+
/// disambiguation.
573+
var isCJKSentenceTerminator: Bool {
574+
self == "\u{3002}" || self == "\u{FF01}" || self == "\u{FF1F}" || self == "\u{FF61}"
575+
}
576+
577+
/// The CJK closing punctuation: corner brackets `」` `』` (and the halfwidth corner `」`),
578+
/// fullwidth parenthesis `)`, lenticular bracket `】`, and angle brackets `〉` `》`. Walk-backs
579+
/// skip a run of these to find the real terminator underneath, and chunk binding attaches them to
580+
/// the word they close.
581+
var isCJKClosingPunctuation: Bool {
582+
self == "\u{300D}" || self == "\u{300F}" || self == "\u{FF09}"
583+
|| self == "\u{3011}" || self == "\u{3009}" || self == "\u{300B}" || self == "\u{FF63}"
584+
}
585+
}
586+
565587
private extension Character {
566588
/// True when the character begins a word of a space-less script (Han, Hiragana, Katakana, Hangul,
567589
/// Thai, Lao, Khmer, Myanmar, ...). These scripts write words without separating spaces, so the
@@ -597,24 +619,6 @@ private extension Character {
597619
isLetter || isNumber
598620
}
599621

600-
/// The CJK sentence terminators: ideographic full stop `。`, fullwidth `!` `?`, and the halfwidth
601-
/// ideographic stop `。`. Declared once so `isPhraseSentenceTerminator` (phrase ends) and
602-
/// `bindsToPrecedingSpacelessWord` (chunk binding) share one list instead of each restating the
603-
/// four codepoints and silently drifting when one is updated.
604-
var isCJKSentenceTerminator: Bool {
605-
self == "\u{3002}" || self == "\u{FF01}" || self == "\u{FF1F}" || self == "\u{FF61}"
606-
}
607-
608-
/// The CJK closing punctuation: corner brackets `」` `』` (and the halfwidth corner `」`),
609-
/// fullwidth parenthesis `)`, lenticular bracket `】`, and angle brackets `〉` `》`. Declared
610-
/// once so `isPhraseClosingPunctuation` (the closer walk-back) and
611-
/// `bindsToPrecedingSpacelessWord` (chunk binding) share one list instead of each restating the
612-
/// codepoints.
613-
var isCJKClosingPunctuation: Bool {
614-
self == "\u{300D}" || self == "\u{300F}" || self == "\u{FF09}"
615-
|| self == "\u{3011}" || self == "\u{3009}" || self == "\u{300B}" || self == "\u{FF63}"
616-
}
617-
618622
/// The CJK opening brackets: corner brackets `「` `『` (and the halfwidth corner `「`), fullwidth
619623
/// parenthesis `(`, lenticular bracket `【`, and angle brackets `〈` `《`. These lead the word
620624
/// they quote, so the trailing-binding rule stops before them while the punctuation-led peel

0 commit comments

Comments
 (0)