TinySegmenter

The super compact Japanese tokenizer in Gdscript for the Godot Engine.

This is a clone of TinySegmenter written in gdscript for godot engine, originally created by (c) 2008 Taku Kudo for javascript.

The code is based off/ported from https://github.com/leungwensen/tiny-segmenter

Example/Usage

var tinySegmenter = preload("res://tinySegmenter/tiny_segmenter.gd").new()
func _ready():
	print(tinySegmenter.segment("おなまえと　おでんわばんごう、おねがい　いたします。"))
	# outputs 
	# ["おなまえ", "と", "　", "おでんわ", "ばんごう", "、", "おね", "がい", "　", "いたし", "ます", "。"]