feat(context): tambah context::tokens dengan tiktoken-rs
Ganti tiga heuristik char-count (/3 di shortsend, /4 di loop turn, /4 di status bar) yang saling tidak konsisten dengan satu BPE tokenizer nyata. tiktoken-rs membundel vocab lewat include_str! saat build, jadi tidak ada akses jaringan saat runtime.
This commit is contained in:
@@ -58,6 +58,7 @@ chrono = { version = "0.4", features = ["serde"] }
|
||||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||
webbrowser = "1"
|
||||
lsp-types = "0.97"
|
||||
tiktoken-rs = "0.12"
|
||||
|
||||
[[bin]]
|
||||
name = "zesdex"
|
||||
|
||||
Reference in New Issue
Block a user