MythEclipse/zesdex
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: tambah doc comment, logging, dan inline comments di semua 255 file

Browse Source 
Meliputi:
- File-level //! doc comment: tujuan file, alur kerja, komponen utama
- Function-level /// doc comment: apa, parameter, return, flow, edge cases
- Struct/enum/trait /// doc comment: peran, field docs
- Tracing logging (tracing::info!/debug!/trace!/warn!/error!) di setiap fungsi
- Inline comments untuk variable dan branching logic penting
- Seluruh 8 crates di workspace: zesdex-backend, zesdex-cms, zesdex-entities,
  zesdex-iam, zesdex-infra, zesdex-ipc, zesdex-middleware, zesdex-utils
- Build: 0 errors, 242/242 tests passed
This commit is contained in:
asepharyana
2026-07-19 17:05:47 +07:00
parent 6680795ce7
commit 5aaedbf787
255 changed files with 5666 additions and 743 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/zesdex-backend/src/app/runtime/stream/json_repair.rs
+38 -6
View File
@@ -5,9 +5,17 @@
//! we want tools to receive whatever arguments were already emitted so the
//! partial work can proceed.
//!
//! How: a single left-to-right scan pushes opening brackets/braces onto a
//! stack and pops them on matching closes, while tracking in-string/escape
//! state. At EOF, the algorithm:
//! 1. Removes a dangling escape backslash if present.
//! 2. Closes an unterminated string.
//! 3. Closes every unclosed bracket/brace in reverse (LIFO) order.
//!
//! Why LIFO vs. depth counters: `{` inside `[` must be closed with `}`
//! *before* the `]`, not after it. Simple depth counters get the order
//! wrong for nested heterogenous structures.
use tracing;
/// Try to repair truncated JSON by closing open strings, braces, and brackets.
///
@@ -17,17 +25,24 @@
/// At the end, if the last char was a backslash (start of an escape
/// sequence), remove it; if inside a string, append `"`; then close
/// every unclosed opener in reverse (LIFO) order.
///
/// Return: the input string with any missing closing delimiters appended.
/// If the input is already valid JSON, it is returned unchanged.
pub fn repair_incomplete_json(s: &str) -> String {
let original_len = s.len();
tracing::trace!(original_len, input_preview = &s[..original_len.min(80)], "repair_incomplete_json — start");
// LIFO stack of open brackets/braces encountered outside strings.
let mut stack: Vec<char> = Vec::new();
let mut in_string = false;
let mut in_string = false; // true between unescaped `"`
let mut prev_was_backslash = false;
// `true` only when the very last character consumed was a bare `\`
// True only when the very last character consumed was a bare `\`
// inside a string (i.e. the start of an escape that was never completed).
let mut ends_with_unclosed_escape = false;
for c in s.chars() {
if prev_was_backslash {
// Consume the character that was being escaped — the escape is
// This character is being escaped — the escape sequence is
// complete, so clear the unclosed-escape flag.
prev_was_backslash = false;
ends_with_unclosed_escape = false;
@@ -44,26 +59,33 @@ pub fn repair_incomplete_json(s: &str) -> String {
continue;
}
if in_string {
continue;
continue; // skip structural chars inside a string
}
match c {
'{' | '[' => stack.push(c),
'}' | ']' => {
// Pop the matching opener unconditionally. If the JSON is
// malformed (e.g. mismatched brackets), we still pop to keep
// the LIFO tracking as lossy — the repair phase will close
// whatever remains on the stack, which is good enough for
// our heuristic use case.
stack.pop();
}
_ => {}
}
}
// --- Build repaired output ---
let mut result = s.to_string();
if ends_with_unclosed_escape {
// The last character is a dangling backslash that started an escape
// The last char is a dangling backslash that started an escape
// but got cut off before the escaped char — remove it.
result.pop();
}
if in_string {
result.push('"');
result.push('"'); // close an unterminated string
}
// Close every unclosed bracket/brace in reverse (LIFO) order.
for &opener in stack.iter().rev() {
match opener {
'{' => result.push('}'),
@@ -71,11 +93,21 @@ pub fn repair_incomplete_json(s: &str) -> String {
_ => {}
}
}
let repaired_len = result.len();
tracing::debug!(
original_len,
repaired_len,
added_chars = (repaired_len - original_len),
"repair_incomplete_json — completed"
);
result
}
#[cfg(test)]
mod tests {
//! Unit tests for JSON repair: unclosed strings, unclosed braces,
//! nested structures, trailing backslashes, and escaped quotes.
use super::*;
#[test]
crates/zesdex-backend/src/app/runtime/stream/mod.rs
+9
View File
@@ -1,6 +1,15 @@
//! SSE stream parser: converts SSE- or JSON-chunked LLM responses into
//! typed `StreamEvent` variants (tokens, reasoning, tool calls, usage, done).
/// JSON-repair utilities for malformed streaming fragments (truncated JSON,
/// missing brackets, escaped newlines inside strings).
pub mod json_repair;
/// Turn-level streaming state machine: manages buffering, SSE parsing,
/// tool-call accumulation, and per-chunk event dispatch.
pub mod turn;
/// Re-export from `zesdex-entities` for convenience:
/// - `SseParser` — low-level SSE line/event parser
/// - `StreamEvent` — typed event variants yielded by the parser
pub use zesdex_entities::{SseParser, StreamEvent};
crates/zesdex-backend/src/app/runtime/stream/turn.rs
+109 -39
View File
@@ -1,12 +1,20 @@
//! Accumulates streaming LLM responses into complete message/tool-call
//! representation via `StreamedTurn`, and provides a standalone tool-call
//! accumulator in `tools::ToolCallAccumulator`.
//!
//! Flow: the caller feeds [`StreamEvent`] items (from `SseParser`) one by
//! one into [`StreamedTurn::apply_event`], which builds up content, reasoning,
//! and tool-call deltas incrementally. When the stream ends, call
//! [`StreamedTurn::build_assistant_message`] to produce a complete
//! `ChatMessage`. If the connection drops before `[DONE]`,
//! [`StreamedTurn::incomplete_tool_call`] detects truncated tool-call JSON.
use super::json_repair::repair_incomplete_json;
use super::StreamEvent;
use crate::dto::chat::message::ChatMessage;
use crate::dto::chat::tool::{ToolCall, ToolFunction};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use tracing;
/// Accumulates a single streaming assistant turn into its final
/// `ChatMessage` form, including tool-call deltas and content/reasoning.
@@ -29,33 +37,46 @@ pub struct ParsedToolCall {
pub is_complete: bool,
}
impl ParsedToolCall {}
impl ParsedToolCall {
// Placeholder for future convenience constructors or helpers.
// Today all field mutation happens inside `StreamedTurn::apply_event`;
// this block exists to reserve the namespace.
}
impl StreamedTurn {
/// Create an empty turn accumulator.
///
/// All fields start at their default (empty / false) state. The caller
/// then feeds [`StreamEvent`] items via [`apply_event`](Self::apply_event).
pub fn new() -> Self {
tracing::debug!("StreamedTurn::new — initialised empty accumulator");
StreamedTurn {
messages: Vec::new(),
tool_calls: Vec::new(),
is_complete: false,
done_received: false,
accumulated_content: String::new(),
accumulated_reasoning: String::new(),
is_complete: false, // set to true when [DONE] is received
done_received: false, // tracks whether a Done event was seen
accumulated_content: String::new(), // text tokens, growing
accumulated_reasoning: String::new(), // reasoning tokens, growing
}
}
/// Apply a `StreamEvent` to the turn, updating accumulated content,
/// reasoning, and tool-call deltas.
/// Apply a single `StreamEvent` to the in-progress accumulation.
///
/// Flow: match on variant — `Token` appends to `accumulated_content`,
/// `Reasoning` to `accumulated_reasoning`, `ToolCallDelta` fills or
/// grows the `tool_calls` vector, `Done` sets `is_complete = true`.
///
/// Any event variant not explicitly handled here (e.g. `Usage`) is
/// silently ignored, since only content/reasoning/tool-call state
/// is relevant for final message construction.
pub fn apply_event(&mut self, event: &StreamEvent) {
match event {
StreamEvent::Token(token) => {
tracing::trace!(len = token.len(), "apply_event: Token");
self.accumulated_content.push_str(token);
}
StreamEvent::Reasoning(reasoning) => {
tracing::trace!(len = reasoning.len(), "apply_event: Reasoning");
self.accumulated_reasoning.push_str(reasoning);
}
StreamEvent::ToolCallDelta {
@@ -64,6 +85,11 @@ impl StreamedTurn {
name,
arguments_delta,
} => {
tracing::trace!(
index, id, name, delta_len = arguments_delta.len(),
"apply_event: ToolCallDelta"
);
// Pad the tool_calls vector with stubs so we can index by `index`.
while self.tool_calls.len() <= *index {
self.tool_calls.push(ParsedToolCall {
id: String::new(),
@@ -73,6 +99,8 @@ impl StreamedTurn {
});
}
let tc = &mut self.tool_calls[*index];
// `id` and `name` are typically sent only on the first delta;
// subsequent deltas for the same index may omit them.
if let Some(new_id) = id {
if !new_id.is_empty() {
tc.id.clone_from(new_id);
@@ -83,12 +111,16 @@ impl StreamedTurn {
tc.name.clone_from(new_name);
}
}
// Accumulate argument JSON fragment-by-fragment.
tc.arguments.push_str(arguments_delta);
}
StreamEvent::Done => {
tracing::debug!("apply_event: Done — turn marked complete");
self.is_complete = true;
}
_ => {}
_ => {
tracing::trace!("apply_event: ignored {:?}", event);
}
}
}
@@ -101,42 +133,57 @@ impl StreamedTurn {
///
/// Return: a complete `ChatMessage` with role `Assistant`.
pub fn build_assistant_message(&self) -> ChatMessage {
tracing::debug!(
tool_calls = self.tool_calls.len(),
content_len = self.accumulated_content.len(),
reasoning_len = self.accumulated_reasoning.len(),
"build_assistant_message — assembling final ChatMessage"
);
// Build the assistant message, with or without tool calls.
let mut msg = if self.tool_calls.is_empty() {
// Plain text-only response — no tool calls to attach.
ChatMessage::assistant(None)
} else {
// Convert ParsedToolCall → DTO ToolCall, repairing truncated JSON.
let tool_dtos: Vec<ToolCall> = self
.tool_calls
.iter()
.filter(|tc| !tc.name.is_empty())
.filter(|tc| !tc.name.is_empty()) // skip unnamed stubs
.map(|tc| {
let args_value: serde_json::Value = match serde_json::from_str(&tc.arguments) {
Ok(v) => v,
Err(e) => {
let repaired = repair_incomplete_json(&tc.arguments);
match serde_json::from_str(&repaired) {
Ok(v) => {
tracing::warn!(
"[stream] tool call '{}' had truncated JSON \
arguments — repaired successfully: {}",
tc.name,
e,
);
v
}
Err(e2) => {
tracing::warn!(
"[stream] tool call '{}' has invalid JSON \
arguments: {} (after repair: {}) — falling \
back to raw string",
tc.name,
e,
e2,
);
serde_json::Value::String(tc.arguments.clone())
// Try to parse arguments as JSON. If the stream was cut
// short, the last tool call's arguments may be truncated.
let args_value: serde_json::Value =
match serde_json::from_str(&tc.arguments) {
Ok(v) => v,
Err(e) => {
let repaired = repair_incomplete_json(&tc.arguments);
match serde_json::from_str(&repaired) {
Ok(v) => {
tracing::warn!(
"[stream] tool call '{}' had truncated JSON \
arguments — repaired successfully: {}",
tc.name,
e,
);
v
}
Err(e2) => {
tracing::warn!(
"[stream] tool call '{}' has invalid JSON \
arguments: {} (after repair: {}) — falling \
back to raw string",
tc.name,
e,
e2,
);
// Last resort: store the raw string so the
// tool dispatcher can surface the error.
serde_json::Value::String(tc.arguments.clone())
}
}
}
}
};
};
ToolCall {
id: tc.id.clone(),
type_: "function".to_string(),
@@ -153,6 +200,8 @@ impl StreamedTurn {
}
msg
};
// Combine reasoning (inside <think> tags) with visible content.
let full_content = if self.accumulated_reasoning.is_empty() {
self.accumulated_content.clone()
} else {
@@ -161,12 +210,15 @@ impl StreamedTurn {
self.accumulated_reasoning, self.accumulated_content
)
};
let content = if full_content.is_empty() {
// Set content to None when empty so downstream code can distinguish
// "no content" from "empty string".
msg.content = if full_content.is_empty() {
tracing::debug!("build_assistant_message — no content after assembly; setting content=None");
None
} else {
Some(full_content)
};
msg.content = content;
msg
}
@@ -183,25 +235,43 @@ impl StreamedTurn {
/// Return: `Some((name, parse_error))` for the first bad tool call, or
/// `None` if every tool call's arguments are complete, parsable JSON.
pub fn incomplete_tool_call(&self) -> Option<(&str, String)> {
self.tool_calls
// Skip unnamed stubs — they indicate the stream never sent enough
// data to begin a real tool call at that index.
let result = self
.tool_calls
.iter()
.filter(|tc| !tc.name.is_empty())
.find_map(|tc| {
serde_json::from_str::<Value>(&tc.arguments)
.err()
.map(|e| (tc.name.as_str(), e.to_string()))
})
});
if let Some((name, ref err)) = result {
tracing::debug!(
tool_name = name, error = err.as_str(),
"incomplete_tool_call — found truncated tool arguments"
);
} else {
tracing::trace!("incomplete_tool_call — all tool calls have valid JSON");
}
result
}
}
impl Default for StreamedTurn {
/// Delegates to [`Self::new`]; exists so `StreamedTurn` can be used
/// as a default field value in other structs.
fn default() -> Self {
tracing::debug!("StreamedTurn::default — delegating to StreamedTurn::new");
Self::new()
}
}
#[cfg(test)]
mod tests {
//! Unit tests for streaming-turn accumulation, including JSON-repair
//! of truncated tool-call arguments and `incomplete_tool_call` detection.
use super::*;
fn tool_call(name: &str, arguments: &str) -> ParsedToolCall {