asepharyana
43e35a71dd
Add tests/llmE2e.test.ts — 7 end-to-end tests driving the REAL moderation prompt pipeline (buildSystemPrompt → XML payload → llmChat → parseModerationResponse) against a live model via omniroute. Covers: clean technical content (no false positives), harassment (flagged), username-only offenses including 'Pecinta Pria' + sexual/provocative usernames + SARA-in-username (always warn/low, NEVER delete — the nickname-reset path), and spam bursts. Gated behind AI_LLM_BASE_URL + AI_LLM_API_KEY: CI (no creds) skips the file → 216 unit tests stay green, zero LLM cost. Run locally via pnpm test:e2e:live (scripts/run-llm-e2e.sh injects creds from bws). Verified: 223/223 tests pass with live LLM, stability across 4 runs, typecheck + biome clean. docs: TESTING.md. ignore .hermes/ plans.
36 lines
1.6 KiB
Bash
Executable File
36 lines
1.6 KiB
Bash
Executable File
#!/bin/bash
|
|
# ────────────────────────────────────────────────────────────────────────────
|
|
# Run the LLM E2E moderation tests against the REAL model.
|
|
#
|
|
# These tests are gated behind AI_LLM_BASE_URL + AI_LLM_API_KEY, so plain
|
|
# `pnpm test` / CI skips them (zero cost, stays green). This script injects
|
|
# the gateway's live credentials from Bitwarden Secrets Manager and runs them.
|
|
#
|
|
# Usage (on the host that runs the gateway):
|
|
# bash scripts/run-llm-e2e.sh # all E2E tests, verbose
|
|
# bash scripts/run-llm-e2e.sh --watch # extra args go to vitest
|
|
#
|
|
# Requires: sudo access to `bws-env gmw` + /etc/bws-token.
|
|
# Cost: 7 real LLM calls (~30s, a few thousand tokens) per run.
|
|
# ────────────────────────────────────────────────────────────────────────────
|
|
set -euo pipefail
|
|
|
|
cd "$(dirname "$0")/.."
|
|
|
|
if [ ! -r /etc/bws-token ]; then
|
|
echo "ERROR: /etc/bws-token not readable — cannot fetch live LLM credentials." >&2
|
|
echo " Set AI_LLM_BASE_URL + AI_LLM_API_KEY manually instead." >&2
|
|
exit 1
|
|
fi
|
|
|
|
export BWS_ACCESS_TOKEN="$(tr -d '\r\n' < /etc/bws-token)"
|
|
|
|
sudo -n env BWS_ACCESS_TOKEN="$BWS_ACCESS_TOKEN" bash -c '
|
|
set -euo pipefail
|
|
export BWS_ACCESS_TOKEN="$BWS_ACCESS_TOKEN"
|
|
ENV=$(bws-env gmw 2>/dev/null)
|
|
set -a; eval "$ENV"; set +a
|
|
cd '"$PWD"'
|
|
exec ./node_modules/.bin/vitest run tests/llmE2e.test.ts --reporter=verbose "$@"
|
|
' -- "$@"
|