Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proposal: Add Golang Bindings for tokenizers #1751

Open
Nav31 opened this issue Mar 17, 2025 · 1 comment
Open

Proposal: Add Golang Bindings for tokenizers #1751

Nav31 opened this issue Mar 17, 2025 · 1 comment

Comments

@Nav31
Copy link

Nav31 commented Mar 17, 2025

  • Performance-Critical Applications – Many performance-intensive apps use Go for concurrency and efficiency. Native bindings allow seamless integration without overhead.
  • Server-Side NLP – Go is widely used in backend services. Golang bindings would make it easier to use Hugging Face tokenizers in APIs, microservices, and inference systems.
  • Better Compatibility – Many Go-based projects rely on Python/C++ workarounds for NLP tasks. Direct bindings would reduce complexity and improve developer experience.
  • Improved Inference Pipelines – Many real-time AI systems are built in Go. Native tokenizers would remove Python bottlenecks, making pipelines faster and more reliable
@strategy155
Copy link

strategy155 commented Mar 31, 2025 •
edited
Loading

How do you imagine that? extern and cgo combination?

Or something more high level, like uniffi-rs, rust2go or fcplug?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@strategy155 @Nav31