- Removed SQLite support from the configuration and database initialization logic.
- Updated database migration scripts to focus solely on PostgreSQL migrations.
- Simplified logging messages to reflect PostgreSQL usage.
- Adjusted database schema definitions to remove SQLite-specific types and structures.
- Modified tests to ensure compatibility with PostgreSQL, including changes to table creation and data types.
- Cleaned up unused imports and code related to SQLite.
- Removed "recharts" from package.json and its associated entries from pnpm-lock.yaml.
- Cleaned up unnecessary dependencies and peer dependencies related to recharts and other unused packages.
- Added new API endpoints for daily trend data and activity heatmap in analyticsRoutes.ts.
- Created new frontend components: ActivityChart, ControlBar, Heatmap, SummaryCards, TopicList, TrendChart, UserTable, and ViolatorTable for displaying analytics data.
- Implemented loading and empty states in the new components.
- Enhanced the existing moderation tests with remote fallback handling for Indonesian text normalization.
- Analytics panel now refreshes immediately when messages are created, updated, deleted, or analyzed
- WebSocket events (message_created, message_updated, message_deleted, message_analyzed) trigger custom event 'analytics_refresh'
- Analytics hook listens to 'analytics_refresh' event for instant updates
- Auto-refresh interval lowered from 60s to 5s as fallback
- Two-tier refresh: real-time push + 5s polling safety net
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Remove Review tab completely (was redundant with Messages flagged view)
- UI state now client-side only (localStorage) — no server API calls for tab/channel/guild selection
- Fixes dashboard crash when server is down — now loads fully client-side
- Live panel still uses server API for voice/media operations (only what needs it)
- Analytics hourly chart labels now show Jakarta time (WIB/UTC+7) instead of UTC
- Analytics formatTimeAgo uses Jakarta time reference
- Reduced tabs to 3: Live, Messages, Analytics
- Removed unused uiState API imports and server-side state fetching
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New LivePanel component combines voice bridge, media player, screen share, and recordings
- Single page layout: voice controls → audio visualizer + active speakers → now playing → music/screen/recordings tabs
- Reduced tabs from 6 to 4: Live, Messages, Analytics, Review
- Sidebar updated with new tab structure
- Default tab changed from 'voice' to 'live'
- Cleaner compact layout with icon buttons and inline controls
- Recordings sub-panel with user avatars, status badges, and download buttons
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MessageCard:
- Add sticker image preview (renders actual sticker from URL, not just text name)
- Show attachment thumbnails inline (up to 4 with overflow counter)
- Remove dead columns (ai_policy_version, ai_evidence) from UI
- Add severity color coding (critical=red, high=orange, medium=yellow, low=blue)
- Add relative time display ('2h ago' instead of full datetime)
- Better hover effects and visual hierarchy
MessagesPanel:
- Add stats bar showing total/clean/warn/flagged/error/pending/deleted/edited counts
- Add AI status filter buttons (all, clean, warn, flagged, error, pending)
- Improve search UX with inline search icon and clear button
- Show filtered count in tab labels
ImageGrid:
- Include sticker images (was only attachments + embeds before)
- Add kind badge overlay (sticker/attachment/embed)
- Show user avatar next to image caption
- Better sticker rendering (object-contain with padding)
Header:
- Add shield icon per tab
- Improve tab titles and add descriptive subtitles
API client:
- Remove dead fields from MessageRecord (ai_moderation_raw, ai_policy_version, ai_evidence)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop 3 columns that were written but never read:
- ai_moderation_raw: raw LLM JSON response (~KB per message, never consumed)
- ai_policy_version: hardcoded 'default-2026-05-30', never used for decisions
- ai_evidence: JSON evidence array, never read after write
Changes:
- schema.ts: remove columns from both Postgres and SQLite table definitions
- messageStore.ts: remove from AIAnalysisUpdate interface and SET clauses
- aiAnalyzer.ts: remove from individual fallback update calls
- aiAnalysisWorker.ts: remove raw write, add missing fields (categories, severity, etc.)
- types.ts: remove from MessageRecord interface
- analysisRoutes.ts: remove from reset analysis call
- New migration: src/database/migrations/001_drop_unused_ai_columns.sql
- Migration applied to live DB: 27 columns → 24 columns
Kept ai_error (useful for future debugging, currently 0 non-null)
Kept metadata (1.9MB total, used for AI sticker/embed evidence analysis)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Adjusted formatting in conversationContext.ts for better token estimation readability.
- Enhanced readability in indonesianTextNormalizer.ts by formatting multiline replacements.
- Reformatted badword lists and whitelists in indonesianTextNormalizer.ts for consistency.
- Improved function signatures in messageStore.ts for clarity.
- Reformatted messageCapture.ts to enhance readability of channel ID checks.
- Cleaned up error logging in messageStore.ts and retentionManager.ts for better clarity.
- Reformatted indonesianSlangLexicon.ts for consistent object formatting.
- Enhanced URL fetching regex patterns in urlFetcher.ts for better readability.
- Simplified query parameter destructuring in analyticsRoutes.ts for cleaner code.
- Improved test readability in autoDeleteManager.test.ts and indonesianTextNormalizer.test.ts by formatting expectations.
- Cleaned up whitespace in messageCaptureFilter.test.ts for consistency.
- New sticker-specific vision prompt that tells LLM stickers are cartoon/meme art, not real photos
- New text-only warning for stickers that fail to download — prevents flagging based on name alone
- Updated system prompt with dedicated sticker guidance section (looser standards for cartoon content)
- Filesystem-backed sticker cache (keyed by name, 7-day TTL, 100MB max with LRU eviction)
- Config: STICKER_CACHE_DIR and STICKER_CACHE_MAX_SIZE_MB with defaults
- Updated .env.example with auto-delete + sticker config docs
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Added configuration options for NVIDIA Nemotron API key, model, and base URL.
- Refactored badword detection to utilize NVIDIA API, with a fallback to a local badword list.
- Updated moderation functions to handle asynchronous operations for text evidence generation.
- Removed dependency on the `indonesian-badwords` package and implemented custom detection logic.
- Enhanced tests to accommodate asynchronous behavior and validate new detection methods.
- Derive severity from status/score when ai_severity missing
- Derive recommended_action from severity/status when field missing
- Upgrade skip logs from debug to info so skip reasons are visible
- Allow both 'flagged' and 'warn' status for eligibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Implemented `useAnalytics` hook for fetching and managing analytics data.
- Created `analyticsStore.ts` to handle database queries for hourly stats, topic trends, user leaderboard, and moderation stats.
- Added Express routes for analytics endpoints including overview, hourly stats, topic trends, user leaderboard, moderation stats, and top violators.
- Introduced a utility function `filterHits` for filtering specific terms in text.
- Add indonesian-badwords dependency for local lexical signal
- Add Indonesian slang lexicon with woy/woi/hadeh as safe casual terms
- Normalize Discord custom emoji <:name:id> to [emoji:name] in prompts
- Wire normalization evidence into both conversationContext and llmModerationClient prompts
- Harden system prompt: woy/woi are casual greetings, not SARA/hate
- Add tests for emoji normalization, slang mapping, badword detection
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Added to safely extract and fetch up to 3 URLs per message (with SSRF protection, 5MB limit, and 8s timeout).
- Implemented recursive extraction to resolve Tenor/Giphy links from their HTML viewers to raw GIF binaries.
- In , fetched images are automatically injected as into the vision LLM context, and truncated webpage text is appended to the message string.
- In llmModerationClient.ts (inner retry), if OpenAI throws a 429 (or 401/403), throw p-retry's AbortError to immediately exit the 3-attempt inner retry loop.
- In aiAnalyzer.ts (outer retry), propagate the AbortError from runModerationAnalysis so the 2-attempt outer retry loop also aborts immediately.
- This ensures that a burst of 20 concurrent tasks hitting rate limits immediately returns the messages to the DB queue (as 'analysis_incomplete') and rapidly increments the individual circuit breaker, pausing processing and preventing a thundering herd instead of making 12 API calls per stuck message.
Infinite recovery loop (#new):
- Inside retryWithBackoff callback in processIndividualFallback, detect
'analysis_incomplete' in the LLM result and throw to trigger backoff.
- Track exhaustedOnIncomplete flag across retries.
- On final exhaustion: write terminal flag 'individual_analysis_exhausted'
to DB so the recovery query (which only looks for 'analysis_incomplete')
never picks this message up again.
- Transient failures (network/parse) are NOT written as exhausted; they
remain as 'analysis_incomplete' and are retried via the CB-throttled
recovery cycle.
Token budget zero-result deadlock (#10):
- If pickBatchWithinBudget returns [] because every candidate message
individually exceeds AI_ANALYSIS_MAX_TARGET_TOKENS, fall back to
messages.slice(0,1) so at least the first message is processed.
- Without this, messages would be permanently stuck as 'pending' because
every recovery tick would fetch them, trim to 0, and exit silently.
- Uses messages.slice(0,1) instead of messages[0]! to avoid the
forbidden noNonNullAssertion lint rule.
Stale state map memory leak (#9):
- startPendingAIAnalysisWorker now prunes conversationErrorCooldown and
conversationProcessing on every recovery interval tick.
- Cooldown entries past their expiry timestamp are deleted.
- Processing entries older than AI_ANALYSIS_PROCESSING_TIMEOUT_MS are
deleted (these represent stale locks from crashed processing runs).
- Prevents unbounded Map growth for long-running bots with many channels.
Batch/individual scheduling collision (#8):
- Build incompleteKeySet (Set<string>) from incompleteKeys before the
batch recovery loop.
- Batch recovery loop skips any key present in incompleteKeySet so a
conversation that has both 'pending' and 'analysis_incomplete' messages
is only targeted by the individual pipeline, not both simultaneously.
- Avoids the DB last-write-wins race where batch and individual pipelines
both update the same message rows concurrently.
#1+#5 - Individual fallback circuit breaker
- Add individualConsecutiveErrors + individualCooldownUntil (30s)
- On success: reset counter; on failure: increment + trip at
AI_ANALYSIS_INDIVIDUAL_CB_THRESHOLD (default 10) consecutive errors
- enqueueIndividualFallbacks checks CB before admitting any work
#1 - Individual fallback concurrency cap
- enqueueIndividualFallbacks enforces AI_ANALYSIS_INDIVIDUAL_MAX_CONCURRENT
(default 20); overflow stays as error/analysis_incomplete in DB and is
recovered by the recovery worker on the next interval
#3 - Unhandled rejection in async setTimeout
- scheduleConversationAnalysis no longer uses async arrow in setTimeout;
all async work is chained with .then()/.catch() explicitly
#4 - Recovery worker ignores individualInFlight
- Add individualInFlightByConversation Map<conversationKey, count>
- processIndividualFallback increments/decrements it in try/finally
- startPendingAIAnalysisWorker skips conversations present in the map
- Recovery worker also processes error/analysis_incomplete messages via
two new messageStore queries: getConversationKeysWithIncompleteAnalysis
and getIncompleteMessagesByConversation
#6 - pickBatchWithinBudget never called
- scheduleConversationAnalysis now calls pickBatchWithinBudget with
AI_ANALYSIS_MAX_TARGET_TOKENS (default 4000) + 50-token per-msg overhead
after fetching messages, before passing to processBatch
#7 - AI_PROCESSING_OVERLAP_MS 30s shorter than max LLM retry window
- Replace hardcoded 30 000 ms constant with configurable
AI_ANALYSIS_PROCESSING_TIMEOUT_MS (default 120 000 ms)
- LLM client: 30s timeout × 3 retries + backoff ≈ 90-100s; 120s is safe
New config keys:
AI_ANALYSIS_PROCESSING_TIMEOUT_MS (default: 120000)
AI_ANALYSIS_INDIVIDUAL_MAX_CONCURRENT (default: 20)
AI_ANALYSIS_INDIVIDUAL_CB_THRESHOLD (default: 10)
AI_ANALYSIS_MAX_TARGET_TOKENS (default: 4000)
New AnalysisQueueStatus fields:
activeIndividualRequests, individualInFlightCount,
individualCircuitBreakerActive
- After a batch LLM call, any result flagged analysis_incomplete is
immediately fanned out to an individual per-message fallback queue
- Batch hard-fail (result.ok=false) and unhandled exceptions now also
route all affected messages to the individual queue instead of waiting
behind the conversation error cooldown
- Individual queue runs fully parallel (fire-and-forget per message),
de-duplicated by a Set<messageId> so no double-processing
- processIndividualFallback runs in the main process (no worker pool IPC
overhead for a single-item call), with retryWithBackoff 2x/2-15s
- AnalysisQueueStatus gains activeIndividualRequests +
individualInFlightCount fields for dashboard observability
- Added @discordjs/opus and opusscript to package.json and pnpm-lock.yaml.
- Updated pnpm-workspace.yaml to allow builds for @discordjs/opus.
- Imported dotenv in config.ts for environment variable management.
- Created .npmrc to manage npm configurations.
Update all vendor submodules to use forks under MythEclipse account:
- vendor/discord-video-stream
- vendor/drizzle-orm
- vendor/better-sqlite3
- vendor/node-datachannel
This ensures all submodules are under your control and can be updated
independently without relying on upstream repositories.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ffmpeg -headers option was receiving the User-Agent value split across
multiple arguments due to spaces not being properly quoted. This caused ffmpeg
to interpret "Mozilla/5.0" as an output format, resulting in:
[NULL @ ...] Unable to find a suitable output format for 'Mozilla/5.0'
Fixed by wrapping the entire headers string in quotes so parseArgsStringToArgv
treats it as a single argument. The headers string is now properly passed to
ffmpeg as: -headers "User-Agent: ... \r\nConnection: ..."
The fix has been applied to vendor/discord-video-stream/src/media/newApi.ts
and compiled into dist/media/newApi.js. A patch file and documentation have
been added to the patches/ directory for reference.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ffmpeg -headers option was receiving the User-Agent value split across
multiple arguments due to spaces not being properly quoted. This caused ffmpeg
to interpret "Mozilla/5.0" as an output format, resulting in:
[NULL @ ...] Unable to find a suitable output format for 'Mozilla/5.0'
Fixed by wrapping the entire headers string in quotes so parseArgsStringToArgv
treats it as a single argument. The headers string is now properly passed to
ffmpeg as: -headers "User-Agent: ... \r\nConnection: ..."
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Replace flat imageParts prologue with per-message image map (messageImageMap)
keyed by message_id. Images are now inserted immediately after their owning
message's text part in the multimodal content array, giving the vision model
proper text+image co-context instead of a disconnected image dump before the
entire prompt.
- Rewrite moderationPrompt as Indonesian-first bilingual system prompt:
* Primary language: Bahasa Indonesia; English secondary
* Explicit Discord community context with Indonesian slang awareness
(anjay, wkwk, santuy, gw/lo abbreviations, etc.)
* SARA, hoaks, ujaran kebencian cultural context
* Charitable intent for ambiguous Indonesian phrasing
* Expanded flag taxonomy: sara, hoaks, nsfw_image, gore_image, doxxing, scam
* analysis field instructed in Bahasa Indonesia (maks 2 kalimat)
* Retry/correction messages also in Bahasa Indonesia
- Image instruction block conditionally injected into prompt only when
hasImages=true, explicitly telling model to treat image + preceding text
as one semantic unit and to OCR meme/screenshot text as message content.
Three-layer defect chain causing 'cannot identify image file <_io.BytesIO object>':
1. attachmentUploader: hardcoded 'application/octet-stream' on Tele CDN upload
regardless of actual file MIME type — CDN stored images under wrong type.
2. messageCapture: processAttachmentUpload call site never forwarded
attachment.contentType into the options bag, so the fix in (1) would
have received undefined and fallen back to octet-stream anyway.
3. llmModerationClient: blindly trusted att.type from the DB record
(Discord-provided MIME) when constructing data: URLs, but validated
neither the HTTP status of the CDN re-fetch nor the actual byte content.
Stale/expired CDN URLs returning HTML error pages were base64-encoded
and sent to the model as 'image/jpeg', causing PIL to reject the stream.
Fixes:
- uploadAttachmentToTele now accepts contentType param (defaults to
application/octet-stream for non-image files)
- processAttachmentUpload options bag gains optional contentType field
- messageCapture forwards attachment.contentType at the call site
- Added sniffImageMimeType() using magic-byte probes for JPEG, PNG, GIF,
WebP, AVIF/HEIF — runs on every downloaded attachment buffer before
base64 encoding; skips the attachment (logs headerHex for diagnosis)
if bytes don't match a known image format
- data: URL now uses the sniffed MIME type, not the DB record
- Introduced `loggerSerialization.ts` to handle error serialization and log metadata formatting.
- Added `serializeError` function to convert Error objects into a structured format.
- Implemented `serializeLogValue` to handle various data types including Errors, Dates, RegExps, and plain objects.
- Created `formatLogMetadata` to format log metadata using the serialization functions.
feat(pagination): implement cursor encoding and decoding
- Added `pagination.ts` to manage cursor-based pagination.
- Implemented `encodeCursor` to convert cursor data into a base64 string.
- Developed `decodeCursor` to parse base64 strings back into cursor data, with error handling for invalid inputs.
The Dockerfile was only copying discord.js-selfbot-v13/package.json but not
discord-video-stream/package.json before running pnpm install. This caused
pnpm to fail with ERR_PNPM_WORKSPACE_PKG_NOT_FOUND since @dank074/discord-video-stream
is declared as a workspace dependency but its package.json wasn't available.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Added allowBuilds for specific packages in pnpm-workspace.yaml
- Excluded certain minimum release ages for packages
- Updated onlyBuiltDependencies list
- Marked discord.js-selfbot-v13 subproject as dirty
- Added `vendor/discord-video-stream` to pnpm workspace.
- Refactored `llmModerationClient.ts` for better readability and consistency.
- Adjusted imports in `recordingsRoutes.ts` for clarity.
- Updated `webserver.ts` to correctly import `createRecordingsRoutes`.
- Enhanced test cases in `llmModerationClient.test.ts` for improved readability.
- Updated submodule references for `better-sqlite3`, `discord-video-stream`, `discord.js-selfbot-v13`, `drizzle-orm`, and `node-datachannel`.
- Created documentation for deprecated dependency removal plan and design.
Design for migrating from Pino to Winston with hybrid console/file logging,
npm-standard log levels, and dependency cleanup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
MythEclipse