feat(context): tambah context::tokens dengan tiktoken-rs

Ganti tiga heuristik char-count (/3 di shortsend, /4 di loop turn, /4
di status bar) yang saling tidak konsisten dengan satu BPE tokenizer
nyata. tiktoken-rs membundel vocab lewat include_str! saat build, jadi
tidak ada akses jaringan saat runtime.
This commit is contained in:
asepharyana
2026-07-16 07:49:43 +07:00
parent 8bb697a53f
commit c219b6ec58
5 changed files with 112 additions and 0 deletions
+1
View File
@@ -58,6 +58,7 @@ chrono = { version = "0.4", features = ["serde"] }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
webbrowser = "1"
lsp-types = "0.97"
tiktoken-rs = "0.12"
[[bin]]
name = "zesdex"