Add CJK language support (Chinese, Japanese, Korean)

[benmaster82]

Add CJK language support (Chinese, Japanese, Korean)

Context

Kwipu currently supports 6 European languages (it, en, fr, de, es, pt). Adding CJK support would open the project to large communities of Obsidian and local-LLM users in East Asia.

What's needed

Chinese (zh)

• Stopwords list
• Word segmentation (e.g. jieba) - Chinese has no spaces between words
• Month names (一月, 二月, ... 十二月)
• Temporal keywords (会议, 决定, 结果, 完成, 截止日期, etc.)
• Relation patterns for wikilink inference

Japanese (ja)

• Stopwords list
• Word segmentation (e.g. fugashi or MeCab)
• Month names (1月, 2月, ... 12月)
• Temporal keywords (会議, 決定, 結果, 完了, etc.)
• Relation patterns

Korean (ko)

• Stopwords list
• Korean uses spaces between words, so the current tokenizer may work with minimal changes
• Month names (1월, 2월, ... 12월)
• Temporal keywords (회의, 결정, 결과, 완료, etc.)
• Relation patterns

Technical challenges

The current tokenizer in lang_config.py uses a regex ([a-zA-Z\u00C0-\u024F0-9]+) that doesn't match CJK characters. The tokenize() function needs a conditional path:

# Pseudocode
if detected_language in ("zh", "ja"):
    tokens = segmentation_library.cut(text)  # jieba, MeCab, etc.
else:
    tokens = regex_tokenize(text)  # current approach

Language detection (detect_language) also needs updating to recognize CJK character ranges (U+4E00-U+9FFF for CJK unified ideographs).

Suggested approach

1. Start with Chinese (largest community, most demand)
2. Add jieba as optional dependency
3. Extend tokenize() with CJK path
4. Add zh stopwords, month names, temporal keywords
5. Reuse the CJK infrastructure for Japanese and Korean