docs(gateway): consolidate README/ARCHITECTURE, drop stale MODULE_STRUCTURE
README.md was the extraction-era document (referenced winston, mock-crc.ts, llmModerationClient.ts, indonesianTextNormalizer.ts — all long gone) and duplicated ARCHITECTURE.md. Rewritten as a short run-the-service guide; layout/design lives only in ARCHITECTURE.md. MODULE_STRUCTURE.md deleted: it was a stale duplicate of ARCHITECTURE.md, referenced by nothing but itself. ARCHITECTURE.md updated to the post-refactor reality: app/ lifecycle split (bootstrap/lifecycle/process-guards/metrics-collector), ai-moderation recovery-worker + cache-prune, per-module index.ts facades, one-way dependency rule, corrected init/shutdown/observability sections.
This commit is contained in:
asepharyana
@@ -5,10 +5,9 @@ messages/attachments/reactions/threads/presence, runs LLM-based AI
|
|||||||
moderation, and publishes everything to Redis pub/sub for the backend to
|
moderation, and publishes everything to Redis pub/sub for the backend to
|
||||||
consume. The backend serves the HTTP/WS API to the frontend.
|
consume. The backend serves the HTTP/WS API to the frontend.
|
||||||
|
|
||||||
> NOTE: this doc is the source of truth for the module layout. The older
|
> NOTE: this doc is the source of truth for the module layout. The old
|
||||||
> `MODULE_STRUCTURE.md` was stale (referenced `winston`, `mock-crc.ts`,
|
> `MODULE_STRUCTURE.md` was a stale duplicate and has been removed. `README.md`
|
||||||
> `indonesianTextNormalizer.ts`, and `aiAnalysisWorker.ts`/`llmModerationClient.ts`
|
> only covers how to run the service.
|
||||||
> which were renamed/merged). If they disagree, this file wins.
|
|
||||||
|
|
||||||
## Top-level layout
|
## Top-level layout
|
||||||
|
|
||||||
@@ -16,33 +15,41 @@ consume. The backend serves the HTTP/WS API to the frontend.
|
|||||||
services/discord-gateway/
|
services/discord-gateway/
|
||||||
├── src/
|
├── src/
|
||||||
│ ├── index.ts # Entry point → initializeDiscordGateway()
|
│ ├── index.ts # Entry point → initializeDiscordGateway()
|
||||||
│ ├── app/
|
│ ├── app/ # Process lifecycle
|
||||||
│ │ ├── bootstrap.ts # Wires client, DB, Redis, workers, schedulers
|
│ │ ├── bootstrap.ts # Startup order: config → DB → services → metrics → login
|
||||||
│ │ ├── shutdown.ts # Graceful shutdown (SIGINT/SIGTERM + transient errors)
|
│ │ ├── lifecycle.ts # Everything wired on the Discord 'ready' hook
|
||||||
|
│ │ ├── process-guards.ts # SIGINT/SIGTERM + uncaught-error policy
|
||||||
|
│ │ ├── metrics-collector.ts # AI pipeline Prometheus gauges
|
||||||
|
│ │ ├── shutdown.ts # Graceful shutdown sequence
|
||||||
│ │ └── retention.ts # Expired-record cleanup scheduler
|
│ │ └── retention.ts # Expired-record cleanup scheduler
|
||||||
│ ├── shared/
|
│ ├── shared/ # Infrastructure — never imports from modules/
|
||||||
│ │ ├── config/ # Zod-validated env (index.ts = schema+loader)
|
│ │ ├── config/ # Zod-validated env (index.ts = schema+loader)
|
||||||
│ │ ├── database/ # Drizzle ORM + pg Pool + migrations
|
│ │ ├── database/ # Drizzle ORM + pg Pool + migrations
|
||||||
│ │ │ ├── init.ts drizzle.ts pool.ts migrate.ts migrateCli.ts
|
│ │ │ ├── init.ts drizzle.ts pool.ts migrate.ts migrateCli.ts
|
||||||
│ │ │ └── schema/ # messages, cache, meta, analytics
|
│ │ │ └── schema/ # messages, cache, meta, analytics
|
||||||
│ │ ├── logger/ # pino wrapper + createChildLogger()
|
│ │ ├── logger/ # pino wrapper + createChildLogger()
|
||||||
│ │ ├── errors/ # AppError / ConfigError ...
|
│ │ ├── errors/ # AppError / ConfigError ... + errorMessage()
|
||||||
|
│ │ │ # + isTransientStreamError()
|
||||||
│ │ ├── utils/ # retry, pagination
|
│ │ ├── utils/ # retry, pagination
|
||||||
│ │ ├── discord/clientOptions.ts # discord.js-selfbot-v13 client options
|
│ │ ├── discord/clientOptions.ts # discord.js-selfbot-v13 client options
|
||||||
│ │ ├── uploader.ts # Shared attachment upload helper
|
│ │ ├── uploader.ts # Shared attachment upload helper
|
||||||
│ │ ├── redis-channels.ts # Redis channel-name constants
|
│ │ ├── redis-channels.ts # Redis channel + command constants
|
||||||
│ │ └── moderation-types.ts # Shared AI analysis domain types
|
│ │ └── moderation-types.ts # Shared AI analysis domain types
|
||||||
│ └── modules/
|
│ └── modules/ # Feature modules, each with an index.ts facade
|
||||||
│ ├── message-capture/ # Discord event listeners + DB store
|
│ ├── message-capture/ # Discord event listeners + DB store
|
||||||
│ ├── ai-moderation/ # LLM moderation pipeline (see below)
|
│ ├── ai-moderation/ # LLM moderation pipeline (see below)
|
||||||
│ ├── attachment-upload/ # Download + (sharp) resize + upload
|
│ ├── attachment-upload/ # Download + (sharp) resize + upload
|
||||||
│ ├── event-broadcaster/ # RedisEventPublisher + EventBroadcaster
|
│ ├── event-broadcaster/ # RedisEventPublisher + EventBroadcaster
|
||||||
│ ├── command-handler/ # Redis-subscribed backend→gateway commands
|
│ ├── command-handler/ # Redis-subscribed backend→gateway commands
|
||||||
│ ├── reaction-tracking/ thread-tracking/ user-presence/
|
│ ├── reaction-tracking/ thread-tracking/ user-presence/
|
||||||
│ ├── channel-topic/ guild-member-events/
|
│ ├── channel-topic/ guild-member-events/ monitor/
|
||||||
│ └── gateway-metrics/ # Prometheus /metrics endpoint (port 4016)
|
│ └── gateway-metrics/ # Prometheus /metrics endpoint (port 4016)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Dependency direction is one-way: `index.ts` → `app/` → `modules/` → `shared/`.
|
||||||
|
Code outside a module imports its `index.ts` facade, never an internal file;
|
||||||
|
deep imports stay valid inside the module itself.
|
||||||
|
|
||||||
## AI moderation pipeline (`ai-moderation/`)
|
## AI moderation pipeline (`ai-moderation/`)
|
||||||
|
|
||||||
LLM-only judge — no regex/heuristic classification. One orchestrator call
|
LLM-only judge — no regex/heuristic classification. One orchestrator call
|
||||||
@@ -54,8 +61,15 @@ concurrency semaphores. The text lane frees its lock and saves+broadcasts the
|
|||||||
moment text analysis finishes — it never waits on a slow vision/media batch
|
moment text analysis finishes — it never waits on a slow vision/media batch
|
||||||
of the same conversation, and vice versa.
|
of the same conversation, and vice versa.
|
||||||
|
|
||||||
- `aiAnalyzer.ts` — public API: `queueMessageAnalysis`, `getAnalysisQueueStatus`,
|
- `aiAnalyzer.ts` — public API: `queueMessageAnalysis`, `queueConversationAnalysis`,
|
||||||
`startPendingAIAnalysisWorker` (recovery worker + cache-prune).
|
`getAnalysisQueueStatus`, `startPendingAIAnalysisWorker`. Short-circuits
|
||||||
|
age-restricted and skip-list messages before any LLM work.
|
||||||
|
- `recovery-worker.ts` — periodic sweep for stranded `pending` messages
|
||||||
|
(re-scheduled per lane) and `error`/`analysis_incomplete` messages
|
||||||
|
(individual fallback queue); prunes stale lane locks, per-conversation CB
|
||||||
|
counters and individual in-flight markers.
|
||||||
|
- `cache-prune.ts` — throttled (6h) expired-verdict sweep across Postgres and
|
||||||
|
Qdrant, driven from the recovery interval.
|
||||||
- `batchScheduler.ts` — per-conversation per-LANE debounce → `processBatch`
|
- `batchScheduler.ts` — per-conversation per-LANE debounce → `processBatch`
|
||||||
(lane-aware). `splitMessagesByLane` / `laneOfMessage` live in
|
(lane-aware). `splitMessagesByLane` / `laneOfMessage` live in
|
||||||
`analysisLanes.ts` (pure, unit-testable).
|
`analysisLanes.ts` (pure, unit-testable).
|
||||||
@@ -123,28 +137,51 @@ See `src/shared/redis-channels.ts` for the canonical names.
|
|||||||
|
|
||||||
## Initialization flow
|
## Initialization flow
|
||||||
|
|
||||||
|
`bootstrap.ts` runs these steps in order (each is a named function):
|
||||||
|
|
||||||
1. Validate env (Zod). Refuse to start if `AI_ANALYSIS_ENABLED` but no key.
|
1. Validate env (Zod). Refuse to start if `AI_ANALYSIS_ENABLED` but no key.
|
||||||
2. `AUTO_MIGRATE_ON_STARTUP` → run pending Drizzle migrations.
|
→ `assertConfigIsUsable()`
|
||||||
3. `initializeDatabase()` (pg Pool, min 0).
|
2. Build long-lived services: Discord client, `RedisEventPublisher` +
|
||||||
4. Create discord.js-selfbot-v13 client; register listeners on `ready`.
|
`EventBroadcaster`, `CommandHandler`; install the shutdown handler.
|
||||||
5. Start `gmw-discord-gateway` metrics server (port `METRICS_PORT`, default 4016).
|
3. Connect infrastructure → `connectDatabase()`:
|
||||||
6. `client.login(token)`.
|
`AUTO_MIGRATE_ON_STARTUP` runs pending Drizzle migrations, then
|
||||||
|
`initializeDatabase()` (pg Pool, min 0).
|
||||||
|
4. `registerClientDebugLogging()` — only client debug lines carrying signal.
|
||||||
|
5. Install process guards (`registerProcessGuards`).
|
||||||
|
6. Register pipeline gauges + start the metrics server (port `METRICS_PORT`,
|
||||||
|
default 4016).
|
||||||
|
7. `client.login(token)`.
|
||||||
|
|
||||||
|
On the Discord `ready` event, `lifecycle.ts` runs `startGatewayLifecycle()`:
|
||||||
|
|
||||||
|
1. Inject the event broadcaster into message-capture and moderation-actions
|
||||||
|
(before any listener can fire).
|
||||||
|
2. Register Discord listeners: message-capture, reaction, thread, presence,
|
||||||
|
channel-topic, guild-member.
|
||||||
|
3. Start background work: AI analysis worker + recovery worker, command
|
||||||
|
handler, retention cleanup, weekly digest.
|
||||||
|
|
||||||
## Graceful shutdown
|
## Graceful shutdown
|
||||||
|
|
||||||
`SIGINT`/`SIGTERM` (and uncaught transient stream errors: EPIPE / ECONNRESET /
|
`process-guards.ts` owns the policy. `SIGINT`/`SIGTERM` and non-transient
|
||||||
ERR_STREAM_DESTROYED / ERR_STREAM_WRITE_AFTER_END are treated as non-fatal):
|
uncaught exceptions/rejections run `shutdown.ts`; transient stream errors
|
||||||
stop metrics → close event broadcaster (Redis) → close command handler →
|
(EPIPE / ECONNRESET / ERR_STREAM_DESTROYED / ERR_STREAM_WRITE_AFTER_END, see
|
||||||
close DB → destroy client → exit.
|
`isTransientStreamError()`) are logged and IGNORED so the bot stays online.
|
||||||
|
|
||||||
|
Shutdown order: stop metrics → close event broadcaster (Redis) → close command
|
||||||
|
handler → close DB → destroy client → exit.
|
||||||
|
|
||||||
## Observability
|
## Observability
|
||||||
|
|
||||||
Prometheus scrapes `127.0.0.1:4016/metrics` (`bete_*` prefix). Collectors run
|
Prometheus scrapes `127.0.0.1:4016/metrics` (`bete_*` prefix). Collectors run
|
||||||
per-scrape and expose: process memory/uptime, and (when AI analysis is on) live
|
per-scrape and expose: process memory/uptime, and (when AI analysis is on) live
|
||||||
pipeline gauges — `ai_analysis_queued_conversations`,
|
pipeline gauges registered by `app/metrics-collector.ts` —
|
||||||
`ai_analysis_active_batch_requests`, `ai_analysis_active_individual_requests`,
|
`ai_analysis_queued_conversations`, `ai_analysis_active_batch_requests`,
|
||||||
`ai_analysis_individual_in_flight`, `ai_analysis_individual_circuit_breaker_active`,
|
`ai_analysis_active_text_requests`, `ai_analysis_active_media_requests`,
|
||||||
`ai_analysis_worker_threads`, `ai_analysis_worker_threads_active`.
|
`ai_analysis_active_individual_requests`, `ai_analysis_individual_in_flight`,
|
||||||
|
`ai_analysis_individual_circuit_breaker_active`,
|
||||||
|
`ai_analysis_worker_threads_{text,media}`,
|
||||||
|
`ai_analysis_worker_threads_active_{text,media}`.
|
||||||
|
|
||||||
## Key invariants (do not break)
|
## Key invariants (do not break)
|
||||||
|
|
||||||
|
|||||||
@@ -1,73 +0,0 @@
|
|||||||
# Discord Gateway Service — Module Structure
|
|
||||||
|
|
||||||
> Kept as a compact module map. For the authoritative layout, design
|
|
||||||
> decisions, and invariants, see `ARCHITECTURE.md`. This file was rewritten
|
|
||||||
> on 2026-08-16 to fix stale references (`winston` → pino,
|
|
||||||
> `mock-crc.ts`/`indonesianTextNormalizer.ts` removed,
|
|
||||||
> `aiAnalysisWorker.ts` → `ai-analysis-worker.ts`,
|
|
||||||
> `llmModerationClient.ts` → `llmClient.ts`).
|
|
||||||
|
|
||||||
## Top-level
|
|
||||||
|
|
||||||
```
|
|
||||||
services/discord-gateway/
|
|
||||||
├── src/
|
|
||||||
│ ├── index.ts # Entry point
|
|
||||||
│ ├── app/ # bootstrap, shutdown, retention
|
|
||||||
│ ├── shared/ # config, database, logger, errors, utils, discord, uploader
|
|
||||||
│ └── modules/
|
|
||||||
│ ├── message-capture/ # Discord listeners + DB store + metadata
|
|
||||||
│ ├── ai-moderation/ # LLM moderation pipeline (largest module)
|
|
||||||
│ ├── attachment-upload/ # Download + sharp resize + upload
|
|
||||||
│ ├── event-broadcaster/ # RedisEventPublisher + EventBroadcaster
|
|
||||||
│ ├── command-handler/ # Backend→gateway Redis commands
|
|
||||||
│ ├── reaction-tracking/ thread-tracking/ user-presence/
|
|
||||||
│ ├── channel-topic/ guild-member-events/
|
|
||||||
│ └── gateway-metrics/ # Prometheus /metrics (port 4016)
|
|
||||||
├── tests/ # Vitest suites (129 tests)
|
|
||||||
├── drizzle/ # Drizzle migration SQL + journal
|
|
||||||
├── ARCHITECTURE.md README.md package.json tsconfig.json vitest.config.ts
|
|
||||||
```
|
|
||||||
|
|
||||||
## Module responsibilities (summary)
|
|
||||||
|
|
||||||
### message-capture
|
|
||||||
Captures `messageCreate`/`messageUpdate`/`messageDelete`, extracts metadata,
|
|
||||||
stores to Postgres, publishes to Redis. Controller–Service–Repository split:
|
|
||||||
`messageCapture.ts` (listener) → `messageStore.ts` (DB) + `messageMetadata.ts`
|
|
||||||
(service).
|
|
||||||
|
|
||||||
### ai-moderation
|
|
||||||
LLM-only moderation. Entry: `aiAnalyzer.ts` (`queueMessageAnalysis`,
|
|
||||||
`startPendingAIAnalysisWorker`, `getAnalysisQueueStatus`). Scheduling:
|
|
||||||
`batchScheduler.ts` → `batchProcessor.ts` (batch lock + circuit breaker) →
|
|
||||||
`individualFallbackProcessor.ts` (per-message retry). Heavy work runs in the
|
|
||||||
Piscina pool via `ai-analysis-worker.ts` (jobs `batch` / `individual`).
|
|
||||||
Orchestration/caching: `moderationOrchestrator.ts` (exact hash → batched
|
|
||||||
semantic Qdrant → LLM), `textBatchProcessor.ts` / `mediaBatchProcessor.ts`
|
|
||||||
(one LLM call per sub-batch), `llmClient.ts` (central streaming client),
|
|
||||||
`embeddingClient.ts` + `qdrantClient.ts` (semantic cache), plus
|
|
||||||
`channelCultureStore.ts` / `userProfileStore.ts`.
|
|
||||||
|
|
||||||
### attachment-upload
|
|
||||||
`attachmentUploader.ts` (download → upload to storage) + `imageResizer.ts`
|
|
||||||
(sharp resize). Emits `discord:attachment:*`.
|
|
||||||
|
|
||||||
### event-broadcaster
|
|
||||||
`RedisEventPublisher` (ioredis publish) + `EventBroadcaster` (typed methods).
|
|
||||||
Channel names in `src/shared/redis-channels.ts`.
|
|
||||||
|
|
||||||
### gateway-metrics
|
|
||||||
`metrics.ts` Prometheus HTTP server on `METRICS_PORT` (4016). Collectors run
|
|
||||||
per scrape; live pipeline gauges registered in `bootstrap.ts`.
|
|
||||||
|
|
||||||
## Shared infrastructure
|
|
||||||
- **config** — Zod schema in `shared/config/index.ts` (single source of truth).
|
|
||||||
- **database** — Drizzle ORM over `pg`; pool `min:0` (`shared/config`).
|
|
||||||
- **logger** — `pino` wrapper, `createChildLogger()` for context loggers.
|
|
||||||
- **errors** — `AppError` hierarchy (`ConfigError`, …).
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
- No HTTP server (other than the metrics endpoint). Pure event-driven.
|
|
||||||
- `MODULE_STRUCTURE.md` is intentionally a sketch; `ARCHITECTURE.md` is the
|
|
||||||
detailed reference. When they diverge, `ARCHITECTURE.md` wins.
|
|
||||||
@@ -1,319 +1,63 @@
|
|||||||
# Discord Gateway Service - Extraction Complete
|
# Discord Gateway
|
||||||
|
|
||||||
## Overview
|
Event-driven selfbot service: captures Discord events, runs LLM moderation,
|
||||||
|
publishes everything to Redis for the backend to consume.
|
||||||
|
|
||||||
Successfully extracted Discord Gateway service with **Modular MVC + Event-Driven Architecture** using Redis pub/sub for inter-service communication.
|
> Architecture, invariants and the AI pipeline are documented in
|
||||||
|
> **`ARCHITECTURE.md`** — that file is the source of truth. This README only
|
||||||
|
> covers how to run it.
|
||||||
|
|
||||||
## Directory Structure
|
## Commands
|
||||||
|
|
||||||
```
|
```bash
|
||||||
services/discord-gateway/
|
pnpm install
|
||||||
├── src/
|
pnpm typecheck # tsc --noEmit
|
||||||
│ ├── app/
|
pnpm lint # biome check --diagnostic-level=error .
|
||||||
│ │ ├── bootstrap.ts # Service initialization (Discord client, DB, Redis)
|
pnpm test # vitest run (138 tests)
|
||||||
│ │ └── shutdown.ts # Graceful shutdown handler
|
pnpm build # tsc — CI/prod builds run this inside nix, which also
|
||||||
│ ├── shared/ # Shared infrastructure layer
|
# runs scripts/fix-imports.mjs to rewrite @/ aliases and
|
||||||
│ │ ├── config/
|
# extensionless imports for Node ESM
|
||||||
│ │ │ └── config.ts # Zod-validated environment config
|
pnpm dev # tsx watch src/index.ts
|
||||||
│ │ ├── database/
|
pnpm start # node dist/index.js
|
||||||
│ │ │ ├── schema.ts # Drizzle ORM schema
|
|
||||||
│ │ │ ├── drizzle.ts # PostgreSQL connection
|
|
||||||
│ │ │ ├── migrate.ts # Migration runner
|
|
||||||
│ │ ├── errors/
|
|
||||||
│ │ │ └── errors.ts # Custom error classes
|
|
||||||
│ │ ├── logger/
|
|
||||||
│ │ │ ├── logger.ts # Winston logger wrapper
|
|
||||||
│ │ │ └── serialization.ts # Log serialization
|
|
||||||
│ │ ├── utils/
|
|
||||||
│ │ │ └── retry.ts # Retry with exponential backoff
|
|
||||||
│ │ └── discord/
|
|
||||||
│ │ └── clientOptions.ts # Discord.js client config
|
|
||||||
│ ├── modules/ # Feature modules (Modular MVC)
|
|
||||||
│ │ ├── message-capture/ # Controller-Service-Repository
|
|
||||||
│ │ │ ├── messageCapture.ts # Controller: Discord event listeners
|
|
||||||
│ │ │ ├── messageStore.ts # Repository: DB operations
|
|
||||||
│ │ │ ├── messageMetadata.ts # Service: Metadata extraction
|
|
||||||
│ │ │ ├── types.ts # Domain types
|
|
||||||
│ │ │ └── index.ts # Module exports
|
|
||||||
│ │ ├── ai-moderation/ # Controller-Service-Repository
|
|
||||||
│ │ │ ├── aiAnalyzer.ts # Controller: Analysis orchestration
|
|
||||||
│ │ │ ├── llmModerationClient.ts # Service: LLM API client
|
|
||||||
│ │ │ ├── aiAnalysisWorker.ts # Service: Worker pool
|
|
||||||
│ │ │ ├── indonesianTextNormalizer.ts # Service: Text normalization
|
|
||||||
│ │ │ ├── moderationPrompt.ts # Service: Prompt generation
|
|
||||||
│ │ │ └── index.ts # Module exports
|
|
||||||
│ │ ├── attachment-upload/ # Controller-Service-Repository
|
|
||||||
│ │ │ ├── attachmentUploader.ts # Service: Upload orchestration
|
|
||||||
│ │ │ ├── imageResizer.ts # Service: Image resizing
|
|
||||||
│ │ │ └── index.ts # Module exports
|
|
||||||
│ │ └── event-broadcaster/ # Event-driven layer
|
|
||||||
│ │ ├── eventBroadcaster.ts # Service: Redis pub/sub publisher
|
|
||||||
│ │ ├── eventTypes.ts # Domain: Event type definitions
|
|
||||||
│ │ └── index.ts # Module exports
|
|
||||||
│ ├── mock-crc.ts # CRC polyfill for discord.js
|
|
||||||
│ └── index.ts # Service entry point
|
|
||||||
├── ARCHITECTURE.md # Detailed architecture documentation
|
|
||||||
├── package.json # Service dependencies
|
|
||||||
└── tsconfig.json # TypeScript configuration (inherited)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Architecture Patterns
|
Deployment is CI-only: `nix build .#discord-gateway` → Attic cache → systemd
|
||||||
|
restart on the VPS. Do not build/hand-copy the artifact.
|
||||||
|
|
||||||
### 1. Modular MVC Structure
|
## Layout
|
||||||
Each feature module follows **Controller-Service-Repository** pattern:
|
|
||||||
|
|
||||||
**Message Capture Module**:
|
|
||||||
- **Controller** (`messageCapture.ts`): Listens to Discord events (messageCreate, messageUpdate, messageDelete)
|
|
||||||
- **Service** (`messageMetadata.ts`): Extracts and normalizes message metadata
|
|
||||||
- **Repository** (`messageStore.ts`): Database CRUD operations
|
|
||||||
|
|
||||||
**AI Moderation Module**:
|
|
||||||
- **Controller** (`aiAnalyzer.ts`): Orchestrates analysis workflow
|
|
||||||
- **Service** (`llmModerationClient.ts`): LLM API integration
|
|
||||||
- **Service** (`aiAnalysisWorker.ts`): Worker pool management
|
|
||||||
- **Service** (`indonesianTextNormalizer.ts`): Text preprocessing
|
|
||||||
|
|
||||||
**Attachment Upload Module**:
|
|
||||||
- **Service** (`attachmentUploader.ts`): Upload orchestration
|
|
||||||
- **Service** (`imageResizer.ts`): Image processing
|
|
||||||
|
|
||||||
### 2. Event-Driven Architecture
|
|
||||||
**Redis Pub/Sub** replaces WebSocket broadcaster:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
Discord Events → Discord Gateway Service → Redis Pub/Sub → Backend Service
|
src/
|
||||||
↓
|
├── index.ts # entry → initializeDiscordGateway()
|
||||||
Event Channels:
|
├── app/ # process lifecycle
|
||||||
- discord:message:created
|
│ ├── bootstrap.ts # startup order: config → DB → services → metrics → login
|
||||||
- discord:message:updated
|
│ ├── lifecycle.ts # everything wired on the Discord 'ready' hook
|
||||||
- discord:message:deleted
|
│ ├── process-guards.ts # SIGINT/SIGTERM + uncaught error policy
|
||||||
- discord:message:analyzed
|
│ ├── metrics-collector.ts # AI pipeline Prometheus gauges
|
||||||
- discord:attachment:created
|
│ ├── shutdown.ts # graceful shutdown sequence
|
||||||
- discord:attachment:uploaded
|
│ └── retention.ts # expired-record cleanup scheduler
|
||||||
- discord:analysis:queue_status
|
├── shared/ # infrastructure — never imports from modules/
|
||||||
|
│ ├── config/ database/ logger/ errors/ utils/
|
||||||
|
│ ├── discord/clientOptions.ts
|
||||||
|
│ ├── redis-channels.ts # canonical Redis channel + command constants
|
||||||
|
│ └── moderation-types.ts # domain types shared across services
|
||||||
|
└── modules/ # feature modules (each exposes an index.ts facade)
|
||||||
|
├── ai-moderation/ # LLM moderation pipeline (largest module)
|
||||||
|
├── message-capture/ # Discord listeners + message/attachment DB
|
||||||
|
├── attachment-upload/ # download → resize → upload
|
||||||
|
├── event-broadcaster/ # Redis pub/sub publisher
|
||||||
|
├── command-handler/ # backend → gateway commands over Redis
|
||||||
|
├── gateway-metrics/ # Prometheus /metrics (port 4016)
|
||||||
|
├── monitor/ # weekly digest scheduler
|
||||||
|
└── reaction-tracking/ thread-tracking/ user-presence/
|
||||||
|
channel-topic/ guild-member-events/
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Shared Infrastructure Layer
|
Dependency direction is one-way: `index.ts` → `app/` → `modules/` → `shared/`.
|
||||||
Centralized, reusable components:
|
Callers outside a module import its `index.ts` facade, never an internal file.
|
||||||
- **Config**: Zod-validated environment variables
|
|
||||||
- **Logger**: Winston logger with context support
|
|
||||||
- **Database**: Drizzle ORM with PostgreSQL
|
|
||||||
- **Errors**: Custom error classes with codes and HTTP status codes
|
|
||||||
- **Utils**: Retry logic with exponential backoff
|
|
||||||
- **Discord**: Client configuration and options
|
|
||||||
|
|
||||||
### 4. No HTTP Server
|
## Testing
|
||||||
- **Event-driven only**: No Express, WebSocket, or HTTP routes
|
|
||||||
- **Redis pub/sub**: All inter-service communication via Redis
|
|
||||||
- **Backend service**: Consumes events and serves HTTP API
|
|
||||||
- **Frontend**: Continues to use Backend HTTP API
|
|
||||||
|
|
||||||
## Key Features
|
Vitest, tests in `tests/`. Config supplies dummy env vars so the suite runs
|
||||||
|
without live Postgres/Redis/Qdrant; external services are mocked. `llmE2e.test.ts`
|
||||||
### Message Capture
|
is skipped by default and needs real credentials (`pnpm test:e2e:live`).
|
||||||
1. Discord emits `messageCreate`, `messageUpdate`, `messageDelete` events
|
|
||||||
2. `messageCapture.ts` listener receives and validates event
|
|
||||||
3. Extract metadata: user, channel, content, timestamp, attachments
|
|
||||||
4. `messageStore.ts` inserts into PostgreSQL
|
|
||||||
5. `eventBroadcaster.messageCreated()` publishes to Redis
|
|
||||||
6. Backend service subscribes and processes
|
|
||||||
|
|
||||||
### AI Moderation
|
|
||||||
1. `aiAnalyzer.ts` queues messages for analysis
|
|
||||||
2. `llmModerationClient.ts` calls LLM API with context
|
|
||||||
3. `indonesianTextNormalizer.ts` preprocesses text
|
|
||||||
4. Results stored in database
|
|
||||||
5. `eventBroadcaster.messageAnalyzed()` publishes results
|
|
||||||
6. Backend service receives and updates UI
|
|
||||||
|
|
||||||
### Attachment Upload
|
|
||||||
1. `messageCapture.ts` detects attachments
|
|
||||||
2. `attachmentUploader.ts` downloads from Discord
|
|
||||||
3. `imageResizer.ts` resizes images if needed
|
|
||||||
4. Upload to external storage with retry logic
|
|
||||||
5. `eventBroadcaster.attachmentUploaded()` publishes
|
|
||||||
6. Backend service stores metadata
|
|
||||||
|
|
||||||
## Initialization Flow
|
|
||||||
|
|
||||||
```
|
|
||||||
1. Load environment config (Zod validation)
|
|
||||||
↓
|
|
||||||
2. Initialize PostgreSQL connection
|
|
||||||
↓
|
|
||||||
3. Run pending database migrations
|
|
||||||
↓
|
|
||||||
4. Create Discord client with optimized cache
|
|
||||||
↓
|
|
||||||
5. Initialize Redis event broadcaster
|
|
||||||
↓
|
|
||||||
6. Register Discord event listeners
|
|
||||||
- messageCapture (message events)
|
|
||||||
- aiAnalyzer (analysis worker)
|
|
||||||
↓
|
|
||||||
7. Login to Discord
|
|
||||||
↓
|
|
||||||
8. Listen for graceful shutdown signals
|
|
||||||
```
|
|
||||||
|
|
||||||
## Graceful Shutdown
|
|
||||||
|
|
||||||
On SIGINT/SIGTERM/uncaughtException/unhandledRejection:
|
|
||||||
1. Close PostgreSQL connection
|
|
||||||
2. Close Redis connection
|
|
||||||
3. Destroy Discord client
|
|
||||||
4. Exit process (code 0 for clean, 1 for error)
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
|
|
||||||
**Core Discord**:
|
|
||||||
- `discord.js-selfbot-v13` — Discord client (selfbot variant)
|
|
||||||
|
|
||||||
**Media Processing**:
|
|
||||||
- `sharp` — Image resizing
|
|
||||||
|
|
||||||
**Data & Config**:
|
|
||||||
- `drizzle-orm` — Type-safe ORM
|
|
||||||
- `pg` — PostgreSQL driver
|
|
||||||
- `zod` — Config validation
|
|
||||||
- `ioredis` — Redis client
|
|
||||||
|
|
||||||
**Logging & Utilities**:
|
|
||||||
- `winston` — Structured logging
|
|
||||||
- `p-retry` — Retry with backoff
|
|
||||||
- `p-limit` — Concurrency limiting
|
|
||||||
- `piscina` — Worker pool
|
|
||||||
|
|
||||||
## No Breaking Changes
|
|
||||||
|
|
||||||
- Original `src/` remains untouched
|
|
||||||
- Discord Gateway is a **new service** in `services/discord-gateway/`
|
|
||||||
- Can run alongside existing monolith during transition
|
|
||||||
- Backend service will consume Redis events
|
|
||||||
- Frontend continues to use Backend HTTP API
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
1. **Create Backend service** (`services/backend/`)
|
|
||||||
- HTTP API endpoints
|
|
||||||
- Redis event subscribers
|
|
||||||
- Database models
|
|
||||||
- WebSocket broadcaster
|
|
||||||
|
|
||||||
2. **Update Frontend** (`frontend/`)
|
|
||||||
- Connect to Backend HTTP API
|
|
||||||
- Subscribe to WebSocket events
|
|
||||||
|
|
||||||
3. **Nix & CI/CD**
|
|
||||||
- flake.nix package for Discord Gateway
|
|
||||||
- systemd services (gmw-backend, gmw-discord-gateway)
|
|
||||||
- GitHub Actions for build/deploy (nix copy → systemctl restart)
|
|
||||||
|
|
||||||
4. **Documentation**
|
|
||||||
- API documentation
|
|
||||||
- Event schema documentation
|
|
||||||
- Deployment guide
|
|
||||||
|
|
||||||
## Files Created
|
|
||||||
|
|
||||||
**Total: 43 files**
|
|
||||||
|
|
||||||
### Shared Infrastructure (9 files)
|
|
||||||
- `src/shared/config/config.ts`
|
|
||||||
- `src/shared/database/` (5 files)
|
|
||||||
- `@bete/shared/errors` (shared package)
|
|
||||||
- `src/shared/logger/logger.ts`
|
|
||||||
- `src/shared/logger/serialization.ts`
|
|
||||||
- `src/shared/utils/retry.ts`
|
|
||||||
- `src/shared/discord/clientOptions.ts`
|
|
||||||
|
|
||||||
### Modules (28 files)
|
|
||||||
- `src/modules/message-capture/` (5 files)
|
|
||||||
- `src/modules/ai-moderation/` (6 files)
|
|
||||||
- `src/modules/attachment-upload/` (3 files)
|
|
||||||
- `src/modules/event-broadcaster/` (3 files)
|
|
||||||
|
|
||||||
### App & Entry (4 files)
|
|
||||||
- `src/app/bootstrap.ts`
|
|
||||||
- `src/app/shutdown.ts`
|
|
||||||
- `src/index.ts`
|
|
||||||
- `src/mock-crc.ts`
|
|
||||||
|
|
||||||
### Configuration (2 files)
|
|
||||||
- `package.json`
|
|
||||||
- `ARCHITECTURE.md`
|
|
||||||
|
|
||||||
## Verification Checklist
|
|
||||||
|
|
||||||
✅ Directory structure created
|
|
||||||
✅ Shared infrastructure migrated
|
|
||||||
✅ Message capture module migrated
|
|
||||||
✅ AI moderation module migrated
|
|
||||||
✅ Attachment upload module migrated
|
|
||||||
✅ Event broadcaster module created (Redis pub/sub)
|
|
||||||
✅ Bootstrap and entry point created
|
|
||||||
✅ Package.json with dependencies
|
|
||||||
✅ No HTTP server code (Express, WebSocket removed)
|
|
||||||
✅ Event-driven architecture implemented
|
|
||||||
✅ Graceful shutdown handler
|
|
||||||
✅ Module index files for clean exports
|
|
||||||
✅ Architecture documentation
|
|
||||||
|
|
||||||
## Event Flow Diagram
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Discord Gateway Service │
|
|
||||||
├─────────────────────────────────────────────────────────────────┤
|
|
||||||
│ │
|
|
||||||
│ ┌────────────────────────────┐ ┌────────────────────────────┐ │
|
|
||||||
│ │ Message Capture │ │ AI Moderation │ │
|
|
||||||
│ │ (Controller) │ │ (Controller) │ │
|
|
||||||
│ └──────────────┬─────────────┘ └──────────────┬─────────────┘ │
|
|
||||||
│ │ │ │
|
|
||||||
│ ├───────────────────────────────┤ │
|
|
||||||
│ │ │ │
|
|
||||||
│ ▼ ▼ │
|
|
||||||
│ ┌───────────────────────────────────────────────────────────┐ │
|
|
||||||
│ │ Event Broadcaster (Redis Pub/Sub) │ │
|
|
||||||
│ │ - discord:message:created │ │
|
|
||||||
│ │ - discord:message:updated │ │
|
|
||||||
│ │ - discord:message:deleted │ │
|
|
||||||
│ │ - discord:message:analyzed │ │
|
|
||||||
│ │ - discord:attachment:created │ │
|
|
||||||
│ │ - discord:attachment:uploaded │ │
|
|
||||||
│ │ - discord:analysis:queue_status │ │
|
|
||||||
│ └───────────────────────────────────────────────────────────┘ │
|
|
||||||
│ │ │
|
|
||||||
└────────────────────────────────┼────────────────────────────────┘
|
|
||||||
│
|
|
||||||
│ Redis Pub/Sub
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Backend Service │
|
|
||||||
│ (Subscribes to events, serves HTTP API, manages WebSocket) │
|
|
||||||
└─────────────────────────────────────────────────────────────────┘
|
|
||||||
│
|
|
||||||
│ HTTP API
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Frontend Application │
|
|
||||||
│ (React SPA, real-time updates via WebSocket) │
|
|
||||||
└─────────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
The Discord Gateway service has been successfully extracted with:
|
|
||||||
- **Modular MVC architecture** for clean separation of concerns
|
|
||||||
- **Event-driven design** using Redis pub/sub for inter-service communication
|
|
||||||
- **Shared infrastructure layer** for reusable components
|
|
||||||
- **No HTTP server** — pure event-driven service
|
|
||||||
- **Graceful shutdown** handling
|
|
||||||
- **Type-safe configuration** with Zod validation
|
|
||||||
- **Structured logging** with Winston
|
|
||||||
- **PostgreSQL integration** with Drizzle ORM
|
|
||||||
|
|
||||||
The service is ready for integration with the Backend service, which will consume Redis events and serve the HTTP API to the Frontend.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user