fix(chatbot): proper system role, tool catalog prompt, more rounds/history, tool failure observability

- Send system prompt as role:system instead of merging into the user message
  (models treat system messages as authoritative -> better tool selection)
- Add full 14-tool catalog + multi-hop instruction to the system prompt
- Raise MAX_TOOL_ROUNDS 4->6 so multi-hop queries complete
- Raise conversation history 3->8 exchanges (less repeat questioning)
- Append guild/channel scope hint to each user turn
- Add saturating toolErrors counter + structured warn log per tool failure
  so chatbot tool failures are observable instead of silently absorbed
This commit is contained in:
asepharyana
2026-09-04 13:36:51 +07:00
committed by asepharyana
parent c5219ee866
commit 9682abef82
2 changed files with 60 additions and 25 deletions
@@ -64,7 +64,7 @@ class ChatbotService {
private async getRecentConversationContext( private async getRecentConversationContext(
userId: string, userId: string,
): Promise<string[]> { ): Promise<string[]> {
const history = await chatbotRepository.getChatHistory(userId, 3); const history = await chatbotRepository.getChatHistory(userId, 8);
return history.flatMap((row) => [ return history.flatMap((row) => [
`User: ${row.user_message}`, `User: ${row.user_message}`,
`Bot: ${row.bot_response}`, `Bot: ${row.bot_response}`,
@@ -82,11 +82,27 @@ class ChatbotService {
${scopeLine} ${scopeLine}
TOOLS YANG TERSEDIA (panggil saat user tanya soal data server):
- get_server_stats → statistik server: total pesan, user aktif, flagged/warn/clean
- get_top_channels → channel paling aktif (jumlah pesan terbanyak)
- get_recent_activity → pesan terbaru (siapa, channel mana, jam berapa, isinya)
- get_top_flagged → pesan yang di-flag AI (alasan, severity, analysis)
- search_messages → cari pesan berdasarkan kata kunci (LIKE search)
- get_user_messages → pesan terbaru dari satu user tertentu
- get_user_profile → profil AI dari seorang user (pola perilaku, gaya bicara)
- get_channel_culture → norma/slang dari suatu channel
- get_message_detail → 1 pesan lengkap beserta hasil analisis AI-nya
- get_message_reviews → antrean review moderasi manual
- get_voice_recordings → rekaman suara terbaru
- get_moderation_timeline → tren harian: total vs flagged vs warn vs clean
- get_corrections → riwayat koreksi false-positive AI
ATURAN PENTING — JANGAN PAKAI KONTEKS STATIS: ATURAN PENTING — JANGAN PAKAI KONTEKS STATIS:
- Kamu TIDAK punya hafalan soal angka server (jumlah pesan, user aktif, flagged, dll). JANGAN tebak atau karang angka. - Kamu TIDAK punya hafalan soal angka server. JANGAN tebak atau karang angka.
- Untuk SEMUA pertanyaan soal data server (jumlah pesan, user aktif, channel ramai, aktivitas terbaru, pesan di-flag), WAJIB panggil tool yang sesuai (get_server_stats, get_top_channels, get_recent_activity, get_top_flagged). Jawab HANYA dari hasil tool. - Untuk SEMUA pertanyaan soal data server, WAJIB panggil tool yang sesuai. Jawab HANYA dari hasil tool.
- Tool otomatis di-scope ke guild/channel di atas — kalau argumen guildId/channelId kosong, biarkan kosong (sudah otomatis ter-isi). Jangan isi ID yang kamu tebak. - Tool otomatis di-scope ke guild/channel — kalau argumen kosong, biarkan kosong. Jangan isi ID tebakan.
- Kalau tool balas error atau kosong, bilang aja data lagi ga ketemu, jangan karang. - Kalau tool balik error atau kosong, bilang aja data lagi ga ketemu, jangan karang.
- Boleh panggil banyak tool dalam satu jawaban kalau pertanyaan butuh beberapa data (multi-hop).
Gaya ngobrol: Gaya ngobrol:
- Santai, hangat, kayak ngobrol sama temen - Santai, hangat, kayak ngobrol sama temen
@@ -127,27 +143,31 @@ Gaya ngobrol:
try { try {
const { default: axios } = await import("axios"); const { default: axios } = await import("axios");
// Gateway tidak handle role system — gabung konteks ke user message. const scopeHint = scope.guildId
// The system section stays visible to the model as the first user turn. ? ` (scope: guild ${scope.guildId}${scope.channelId ? `, channel ${scope.channelId}` : ""})`
const contextPrefixed = `${systemPrompt}\n\nPertanyaan user: ${userMessage}`; : "";
// Local message type that models the shapes the agent loop emits and
// Seed conversation: prior turns + current question. // accepts: plain system/user/assistant turns and assistant tool-calls +
const messages: Array< // tool results. role is a union string so history entries typed as
| { role: "user" | "assistant"; content: string } // `"user" | "assistant"` don't break the discriminant.
| { type ChatMsg = {
role: "assistant"; role: "system" | "user" | "assistant" | "tool";
content: string | null; content?: string | null;
tool_calls: Array<{ tool_call_id?: string;
id: string; tool_calls?: Array<{
type: "function"; id: string;
function: { name: string; arguments: string }; type: "function";
}>; function: { name: string; arguments: string };
} }>;
| { role: "tool"; tool_call_id: string; content: string } };
> = [...history, { role: "user", content: contextPrefixed }]; const messages: ChatMsg[] = [
{ role: "system", content: systemPrompt },
...history,
{ role: "user", content: `${userMessage}${scopeHint}` },
];
// ── Agentic tool loop ───────────────────────────────────────── // ── Agentic tool loop ─────────────────────────────────────────
const MAX_TOOL_ROUNDS = 4; const MAX_TOOL_ROUNDS = 6;
for (let round = 0; round <= MAX_TOOL_ROUNDS; round += 1) { for (let round = 0; round <= MAX_TOOL_ROUNDS; round += 1) {
const response = await axios.post( const response = await axios.post(
`${baseUrl}/chat/completions`, `${baseUrl}/chat/completions`,
@@ -1,4 +1,5 @@
import { and, desc, eq, like, sql } from "drizzle-orm"; import { and, desc, eq, like, sql } from "drizzle-orm";
import { createChildLogger } from "@/shared/logger/index";
import { getDatabase } from "../../shared/database/index.js"; import { getDatabase } from "../../shared/database/index.js";
import { import {
pgChannelCulturesTable, pgChannelCulturesTable,
@@ -19,6 +20,20 @@ import {
*/ */
export type ToolResult = string; export type ToolResult = string;
const logger = createChildLogger("chatbot.tools");
/** Saturating counter of chatbot tool execution errors (for observability). */
export let toolErrors = 0;
const TOOL_ERROR_CAP = 1000;
function toolExecError(err: unknown, name: string): string {
if (toolErrors < TOOL_ERROR_CAP) toolErrors++;
const detail = (err as Error)?.message ?? "unknown";
logger.warn({ tool: name, error: detail }, "Chatbot tool execution failed");
return `Tool ${name} gagal: ${detail}`;
}
/** Executes a tool call against the real DB and returns a readable result. */ /** Executes a tool call against the real DB and returns a readable result. */
export async function executeTool( export async function executeTool(
name: string, name: string,
@@ -90,7 +105,7 @@ export async function executeTool(
} }
} catch (error) { } catch (error) {
// Best-effort: if a tool fails, return readable error instead of crashing // Best-effort: if a tool fails, return readable error instead of crashing
return `Terjadi kesalahan saat ambil data: ${(error as Error).message ?? "unknown"}`; return toolExecError(error, name);
} }
} }