pnpm v10+ requires the built-in `node:sqlite` module which is only available
in Node 22+. All services now use node:22-alpine.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
`github.repository_owner` returns 'MythEclipse' (mixed case) which GHCR rejects.
Hardcode lowercase 'mytheclipse' to match GHCR requirements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract services into services/{frontend,backend,discord-gateway}
- Create packages/shared/ for shared logger, errors, utils, types
- Setup Modular MVC pattern in backend (controller→service→repository)
- Setup event-driven architecture in discord-gateway with Redis pub/sub
- Move Docker files to infra/docker/ with per-service Dockerfiles
- Update docker-compose.yml to use Traefik-only routing (no port exposes)
- Update GitHub Actions deploy workflow for multi-service matrix build
- Fix all import paths and resolve type errors across all services
- All 3 services pass tsc --noEmit clean
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Count only flagged messages in the active users Flag column
- Coerce PostgreSQL count results to numbers before calculating violation_score
- Prevent string concatenation in flagged*3 + warned scoring
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
API provider returns 400 error for these params. Removed from all LLM calls
in llmModerationClient.ts and indonesianTextNormalizer.ts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Remove LOCAL_BADWORDS array (25 hardcoded words) and FALSE_POSITIVE_WHITELISTS
- Remove detectLocalBadwords function — all detection now goes through API pipeline
- Fix DEFERRAL_ANALYSIS_PATTERN: remove overly broad patterns (admin perlu, bisa berpotensi, maaf/sorry, saya tidak yakin)
- Expand DEFERRAL_EXCEPTION_PATTERN to catch more decisive-deferral variations
- Update tests to reflect API-only detection (local fallback removed)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add XML delimiters to prevent prompt injection (R1)
- Use JSON Schema response format instead of json_object (R2)
- Add concurrency limiter via p-limit (R3)
- Add timeout per media analysis call (R4)
- Resize images with sharp before vision API (R5)
- Split text batches when exceeding batch size limit (R6)
- Add few-shot examples to system prompt (R7)
- Modularize system prompt builder (R8)
- Enhance deferral detection regex with exception patterns (R9)
- Sanitize error messages to avoid leaking internals (R10)
New files: concurrencyLimiter.ts, imageResizer.ts, moderationPrompt.ts
Updated: llmModerationClient.ts, config.ts, package.json, tests
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Extract custom emoji metadata with Discord CDN URLs
- Download and send emoji images to vision model for moderation analysis
- Render custom emoji as inline images in dashboard instead of raw <:name:id> text
- Add emoji vision cache with deterministic keying by emoji ID
- Add custom emoji vision prompt for context-aware moderation
The NixOS base image lacks sed and coreutils in PATH. Native Node.js
post-install scripts (node-pre-gyp, prebuild-install) require sed and
other core utilities. Installing gnused and coreutils-full provides the
necessary binaries for native module compilation.
Also switched back to single nix profile install call (instead of
sequential per-RUN) since all packages now reference the same pinned
nixpkgs commit, eliminating version drift conflicts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The node-pre-gyp and prebuild-install wrapper scripts require sed, which
is not in PATH in the NixOS base container. Install coreutils first to
provide sed, dirname, basename, and other core utilities needed by
native module post-install scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
pnpm's isolated store prevents native post-install scripts from finding
their dependency binaries (node-pre-gyp, prebuild-install). This causes
@discordjs/opus and @lng2004/node-datachannel to fail during pnpm install.
Enabling shamefully-hoist flattens node_modules so native scripts can
resolve their dependencies correctly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cacert 3.123 was installed first, but nodejs_22 pulls cacert 3.117 as a
transitive dependency, causing a file conflict on ca-bundle.crt.
Since all packages that need SSL certificates already pull cacert as a
dependency, installing it explicitly is redundant and causes conflicts.
Removing the explicit cacert install lets each package use its resolved
transitive cacert version.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two issues were causing Docker build failures:
1. Invalid nixpkgs commit hash (404 on GitHub archive)
2. Installing packages simultaneously caused version conflicts
(cacert 3.123 vs 3.117, git vs git-minimal)
Fix:
- Pin to valid nixos-unstable HEAD commit
- Install each package with sequential 'nix profile add' calls
so each package resolves dependencies before the next installs
- Remove explicit git (git-minimal pulled by yt-dlp is sufficient)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The nixos-unstable channel changes rapidly and causes file conflicts
between packages (cacert 3.123 vs 3.117, git vs git-minimal) because
different packages are resolved from different nixpkgs snapshots during
the same profile install.
Pinning to a specific commit ensures all packages are resolved from the
same nixpkgs snapshot, eliminating version drift and file conflicts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
yt-dlp already pulls git-minimal as a transitive dependency. Adding git
explicitly causes a file conflict on packinfo.pl. Removing git from the
install list resolves this since git-minimal provides all git functionality
needed for CI builds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Updated .env.example to include Groq API configuration.
- Enhanced config schema in config.ts to support Groq API keys and settings.
- Implemented Groq moderation API call in indonesianTextNormalizer.ts as a fallback for badword detection.
- Removed the Indonesian slang normalization function and related tests to streamline moderation logic.
- Updated tests to reflect changes in moderation strategy, focusing on emoji normalization and badword detection.
fix(redis): enhance Redis connection handling with fallback to in-memory storage
chore(sql): create missing messages and attachments tables with necessary constraints and indexes
- Updated test files to use a separate test database configuration.
- Introduced a new helper module for managing test database operations.
- Added a setup file to configure the environment for tests.
- Created new database migration scripts to optimize message indexing.
- Added a sample environment file for test database configuration.
- Removed SQLite support from the configuration and database initialization logic.
- Updated database migration scripts to focus solely on PostgreSQL migrations.
- Simplified logging messages to reflect PostgreSQL usage.
- Adjusted database schema definitions to remove SQLite-specific types and structures.
- Modified tests to ensure compatibility with PostgreSQL, including changes to table creation and data types.
- Cleaned up unused imports and code related to SQLite.
- Removed "recharts" from package.json and its associated entries from pnpm-lock.yaml.
- Cleaned up unnecessary dependencies and peer dependencies related to recharts and other unused packages.
- Added new API endpoints for daily trend data and activity heatmap in analyticsRoutes.ts.
- Created new frontend components: ActivityChart, ControlBar, Heatmap, SummaryCards, TopicList, TrendChart, UserTable, and ViolatorTable for displaying analytics data.
- Implemented loading and empty states in the new components.
- Enhanced the existing moderation tests with remote fallback handling for Indonesian text normalization.
- Analytics panel now refreshes immediately when messages are created, updated, deleted, or analyzed
- WebSocket events (message_created, message_updated, message_deleted, message_analyzed) trigger custom event 'analytics_refresh'
- Analytics hook listens to 'analytics_refresh' event for instant updates
- Auto-refresh interval lowered from 60s to 5s as fallback
- Two-tier refresh: real-time push + 5s polling safety net
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Remove Review tab completely (was redundant with Messages flagged view)
- UI state now client-side only (localStorage) — no server API calls for tab/channel/guild selection
- Fixes dashboard crash when server is down — now loads fully client-side
- Live panel still uses server API for voice/media operations (only what needs it)
- Analytics hourly chart labels now show Jakarta time (WIB/UTC+7) instead of UTC
- Analytics formatTimeAgo uses Jakarta time reference
- Reduced tabs to 3: Live, Messages, Analytics
- Removed unused uiState API imports and server-side state fetching
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New LivePanel component combines voice bridge, media player, screen share, and recordings
- Single page layout: voice controls → audio visualizer + active speakers → now playing → music/screen/recordings tabs
- Reduced tabs from 6 to 4: Live, Messages, Analytics, Review
- Sidebar updated with new tab structure
- Default tab changed from 'voice' to 'live'
- Cleaner compact layout with icon buttons and inline controls
- Recordings sub-panel with user avatars, status badges, and download buttons
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MessageCard:
- Add sticker image preview (renders actual sticker from URL, not just text name)
- Show attachment thumbnails inline (up to 4 with overflow counter)
- Remove dead columns (ai_policy_version, ai_evidence) from UI
- Add severity color coding (critical=red, high=orange, medium=yellow, low=blue)
- Add relative time display ('2h ago' instead of full datetime)
- Better hover effects and visual hierarchy
MessagesPanel:
- Add stats bar showing total/clean/warn/flagged/error/pending/deleted/edited counts
- Add AI status filter buttons (all, clean, warn, flagged, error, pending)
- Improve search UX with inline search icon and clear button
- Show filtered count in tab labels
ImageGrid:
- Include sticker images (was only attachments + embeds before)
- Add kind badge overlay (sticker/attachment/embed)
- Show user avatar next to image caption
- Better sticker rendering (object-contain with padding)
Header:
- Add shield icon per tab
- Improve tab titles and add descriptive subtitles
API client:
- Remove dead fields from MessageRecord (ai_moderation_raw, ai_policy_version, ai_evidence)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop 3 columns that were written but never read:
- ai_moderation_raw: raw LLM JSON response (~KB per message, never consumed)
- ai_policy_version: hardcoded 'default-2026-05-30', never used for decisions
- ai_evidence: JSON evidence array, never read after write
Changes:
- schema.ts: remove columns from both Postgres and SQLite table definitions
- messageStore.ts: remove from AIAnalysisUpdate interface and SET clauses
- aiAnalyzer.ts: remove from individual fallback update calls
- aiAnalysisWorker.ts: remove raw write, add missing fields (categories, severity, etc.)
- types.ts: remove from MessageRecord interface
- analysisRoutes.ts: remove from reset analysis call
- New migration: src/database/migrations/001_drop_unused_ai_columns.sql
- Migration applied to live DB: 27 columns → 24 columns
Kept ai_error (useful for future debugging, currently 0 non-null)
Kept metadata (1.9MB total, used for AI sticker/embed evidence analysis)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Adjusted formatting in conversationContext.ts for better token estimation readability.
- Enhanced readability in indonesianTextNormalizer.ts by formatting multiline replacements.
- Reformatted badword lists and whitelists in indonesianTextNormalizer.ts for consistency.
- Improved function signatures in messageStore.ts for clarity.
- Reformatted messageCapture.ts to enhance readability of channel ID checks.
- Cleaned up error logging in messageStore.ts and retentionManager.ts for better clarity.
- Reformatted indonesianSlangLexicon.ts for consistent object formatting.
- Enhanced URL fetching regex patterns in urlFetcher.ts for better readability.
- Simplified query parameter destructuring in analyticsRoutes.ts for cleaner code.
- Improved test readability in autoDeleteManager.test.ts and indonesianTextNormalizer.test.ts by formatting expectations.
- Cleaned up whitespace in messageCaptureFilter.test.ts for consistency.
- New sticker-specific vision prompt that tells LLM stickers are cartoon/meme art, not real photos
- New text-only warning for stickers that fail to download — prevents flagging based on name alone
- Updated system prompt with dedicated sticker guidance section (looser standards for cartoon content)
- Filesystem-backed sticker cache (keyed by name, 7-day TTL, 100MB max with LRU eviction)
- Config: STICKER_CACHE_DIR and STICKER_CACHE_MAX_SIZE_MB with defaults
- Updated .env.example with auto-delete + sticker config docs
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Added configuration options for NVIDIA Nemotron API key, model, and base URL.
- Refactored badword detection to utilize NVIDIA API, with a fallback to a local badword list.
- Updated moderation functions to handle asynchronous operations for text evidence generation.
- Removed dependency on the `indonesian-badwords` package and implemented custom detection logic.
- Enhanced tests to accommodate asynchronous behavior and validate new detection methods.
- Derive severity from status/score when ai_severity missing
- Derive recommended_action from severity/status when field missing
- Upgrade skip logs from debug to info so skip reasons are visible
- Allow both 'flagged' and 'warn' status for eligibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Implemented `useAnalytics` hook for fetching and managing analytics data.
- Created `analyticsStore.ts` to handle database queries for hourly stats, topic trends, user leaderboard, and moderation stats.
- Added Express routes for analytics endpoints including overview, hourly stats, topic trends, user leaderboard, moderation stats, and top violators.
- Introduced a utility function `filterHits` for filtering specific terms in text.
- Add indonesian-badwords dependency for local lexical signal
- Add Indonesian slang lexicon with woy/woi/hadeh as safe casual terms
- Normalize Discord custom emoji <:name:id> to [emoji:name] in prompts
- Wire normalization evidence into both conversationContext and llmModerationClient prompts
- Harden system prompt: woy/woi are casual greetings, not SARA/hate
- Add tests for emoji normalization, slang mapping, badword detection
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Added to safely extract and fetch up to 3 URLs per message (with SSRF protection, 5MB limit, and 8s timeout).
- Implemented recursive extraction to resolve Tenor/Giphy links from their HTML viewers to raw GIF binaries.
- In , fetched images are automatically injected as into the vision LLM context, and truncated webpage text is appended to the message string.
- In llmModerationClient.ts (inner retry), if OpenAI throws a 429 (or 401/403), throw p-retry's AbortError to immediately exit the 3-attempt inner retry loop.
- In aiAnalyzer.ts (outer retry), propagate the AbortError from runModerationAnalysis so the 2-attempt outer retry loop also aborts immediately.
- This ensures that a burst of 20 concurrent tasks hitting rate limits immediately returns the messages to the DB queue (as 'analysis_incomplete') and rapidly increments the individual circuit breaker, pausing processing and preventing a thundering herd instead of making 12 API calls per stuck message.
Infinite recovery loop (#new):
- Inside retryWithBackoff callback in processIndividualFallback, detect
'analysis_incomplete' in the LLM result and throw to trigger backoff.
- Track exhaustedOnIncomplete flag across retries.
- On final exhaustion: write terminal flag 'individual_analysis_exhausted'
to DB so the recovery query (which only looks for 'analysis_incomplete')
never picks this message up again.
- Transient failures (network/parse) are NOT written as exhausted; they
remain as 'analysis_incomplete' and are retried via the CB-throttled
recovery cycle.
Token budget zero-result deadlock (#10):
- If pickBatchWithinBudget returns [] because every candidate message
individually exceeds AI_ANALYSIS_MAX_TARGET_TOKENS, fall back to
messages.slice(0,1) so at least the first message is processed.
- Without this, messages would be permanently stuck as 'pending' because
every recovery tick would fetch them, trim to 0, and exit silently.
- Uses messages.slice(0,1) instead of messages[0]! to avoid the
forbidden noNonNullAssertion lint rule.
Stale state map memory leak (#9):
- startPendingAIAnalysisWorker now prunes conversationErrorCooldown and
conversationProcessing on every recovery interval tick.
- Cooldown entries past their expiry timestamp are deleted.
- Processing entries older than AI_ANALYSIS_PROCESSING_TIMEOUT_MS are
deleted (these represent stale locks from crashed processing runs).
- Prevents unbounded Map growth for long-running bots with many channels.
Batch/individual scheduling collision (#8):
- Build incompleteKeySet (Set<string>) from incompleteKeys before the
batch recovery loop.
- Batch recovery loop skips any key present in incompleteKeySet so a
conversation that has both 'pending' and 'analysis_incomplete' messages
is only targeted by the individual pipeline, not both simultaneously.
- Avoids the DB last-write-wins race where batch and individual pipelines
both update the same message rows concurrently.
#1+#5 - Individual fallback circuit breaker
- Add individualConsecutiveErrors + individualCooldownUntil (30s)
- On success: reset counter; on failure: increment + trip at
AI_ANALYSIS_INDIVIDUAL_CB_THRESHOLD (default 10) consecutive errors
- enqueueIndividualFallbacks checks CB before admitting any work
#1 - Individual fallback concurrency cap
- enqueueIndividualFallbacks enforces AI_ANALYSIS_INDIVIDUAL_MAX_CONCURRENT
(default 20); overflow stays as error/analysis_incomplete in DB and is
recovered by the recovery worker on the next interval
#3 - Unhandled rejection in async setTimeout
- scheduleConversationAnalysis no longer uses async arrow in setTimeout;
all async work is chained with .then()/.catch() explicitly
#4 - Recovery worker ignores individualInFlight
- Add individualInFlightByConversation Map<conversationKey, count>
- processIndividualFallback increments/decrements it in try/finally
- startPendingAIAnalysisWorker skips conversations present in the map
- Recovery worker also processes error/analysis_incomplete messages via
two new messageStore queries: getConversationKeysWithIncompleteAnalysis
and getIncompleteMessagesByConversation
#6 - pickBatchWithinBudget never called
- scheduleConversationAnalysis now calls pickBatchWithinBudget with
AI_ANALYSIS_MAX_TARGET_TOKENS (default 4000) + 50-token per-msg overhead
after fetching messages, before passing to processBatch
#7 - AI_PROCESSING_OVERLAP_MS 30s shorter than max LLM retry window
- Replace hardcoded 30 000 ms constant with configurable
AI_ANALYSIS_PROCESSING_TIMEOUT_MS (default 120 000 ms)
- LLM client: 30s timeout × 3 retries + backoff ≈ 90-100s; 120s is safe
New config keys:
AI_ANALYSIS_PROCESSING_TIMEOUT_MS (default: 120000)
AI_ANALYSIS_INDIVIDUAL_MAX_CONCURRENT (default: 20)
AI_ANALYSIS_INDIVIDUAL_CB_THRESHOLD (default: 10)
AI_ANALYSIS_MAX_TARGET_TOKENS (default: 4000)
New AnalysisQueueStatus fields:
activeIndividualRequests, individualInFlightCount,
individualCircuitBreakerActive
- After a batch LLM call, any result flagged analysis_incomplete is
immediately fanned out to an individual per-message fallback queue
- Batch hard-fail (result.ok=false) and unhandled exceptions now also
route all affected messages to the individual queue instead of waiting
behind the conversation error cooldown
- Individual queue runs fully parallel (fire-and-forget per message),
de-duplicated by a Set<messageId> so no double-processing
- processIndividualFallback runs in the main process (no worker pool IPC
overhead for a single-item call), with retryWithBackoff 2x/2-15s
- AnalysisQueueStatus gains activeIndividualRequests +
individualInFlightCount fields for dashboard observability
- Added @discordjs/opus and opusscript to package.json and pnpm-lock.yaml.
- Updated pnpm-workspace.yaml to allow builds for @discordjs/opus.
- Imported dotenv in config.ts for environment variable management.
- Created .npmrc to manage npm configurations.
Update all vendor submodules to use forks under MythEclipse account:
- vendor/discord-video-stream
- vendor/drizzle-orm
- vendor/better-sqlite3
- vendor/node-datachannel
This ensures all submodules are under your control and can be updated
independently without relying on upstream repositories.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ffmpeg -headers option was receiving the User-Agent value split across
multiple arguments due to spaces not being properly quoted. This caused ffmpeg
to interpret "Mozilla/5.0" as an output format, resulting in:
[NULL @ ...] Unable to find a suitable output format for 'Mozilla/5.0'
Fixed by wrapping the entire headers string in quotes so parseArgsStringToArgv
treats it as a single argument. The headers string is now properly passed to
ffmpeg as: -headers "User-Agent: ... \r\nConnection: ..."
The fix has been applied to vendor/discord-video-stream/src/media/newApi.ts
and compiled into dist/media/newApi.js. A patch file and documentation have
been added to the patches/ directory for reference.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ffmpeg -headers option was receiving the User-Agent value split across
multiple arguments due to spaces not being properly quoted. This caused ffmpeg
to interpret "Mozilla/5.0" as an output format, resulting in:
[NULL @ ...] Unable to find a suitable output format for 'Mozilla/5.0'
Fixed by wrapping the entire headers string in quotes so parseArgsStringToArgv
treats it as a single argument. The headers string is now properly passed to
ffmpeg as: -headers "User-Agent: ... \r\nConnection: ..."
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Replace flat imageParts prologue with per-message image map (messageImageMap)
keyed by message_id. Images are now inserted immediately after their owning
message's text part in the multimodal content array, giving the vision model
proper text+image co-context instead of a disconnected image dump before the
entire prompt.
- Rewrite moderationPrompt as Indonesian-first bilingual system prompt:
* Primary language: Bahasa Indonesia; English secondary
* Explicit Discord community context with Indonesian slang awareness
(anjay, wkwk, santuy, gw/lo abbreviations, etc.)
* SARA, hoaks, ujaran kebencian cultural context
* Charitable intent for ambiguous Indonesian phrasing
* Expanded flag taxonomy: sara, hoaks, nsfw_image, gore_image, doxxing, scam
* analysis field instructed in Bahasa Indonesia (maks 2 kalimat)
* Retry/correction messages also in Bahasa Indonesia
- Image instruction block conditionally injected into prompt only when
hasImages=true, explicitly telling model to treat image + preceding text
as one semantic unit and to OCR meme/screenshot text as message content.
