MythEclipse/GMW
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(gateway): consolidate README/ARCHITECTURE, drop stale MODULE_STRUCTURE

Browse Source 
README.md was the extraction-era document (referenced winston, mock-crc.ts,
llmModerationClient.ts, indonesianTextNormalizer.ts — all long gone) and
duplicated ARCHITECTURE.md. Rewritten as a short run-the-service guide;
layout/design lives only in ARCHITECTURE.md.

MODULE_STRUCTURE.md deleted: it was a stale duplicate of ARCHITECTURE.md,
referenced by nothing but itself.

ARCHITECTURE.md updated to the post-refactor reality: app/ lifecycle split
(bootstrap/lifecycle/process-guards/metrics-collector), ai-moderation
recovery-worker + cache-prune, per-module index.ts facades, one-way
dependency rule, corrected init/shutdown/observability sections.
This commit is contained in:
asepharyana
2026-09-24 15:09:13 +07:00
parent 494e16b3b3
commit c57ee12da1
3 changed files with 115 additions and 407 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/discord-gateway/README.md
+50 -306
View File
@@ -1,319 +1,63 @@
# Discord Gateway Service - Extraction Complete
# Discord Gateway
## Overview
Event-driven selfbot service: captures Discord events, runs LLM moderation,
publishes everything to Redis for the backend to consume.
Successfully extracted Discord Gateway service with **Modular MVC + Event-Driven Architecture** using Redis pub/sub for inter-service communication.
> Architecture, invariants and the AI pipeline are documented in
> **`ARCHITECTURE.md`** — that file is the source of truth. This README only
> covers how to run it.
## Directory Structure
## Commands
```
services/discord-gateway/
├── src/
│ ├── app/
│ │ ├── bootstrap.ts # Service initialization (Discord client, DB, Redis)
│ │ └── shutdown.ts # Graceful shutdown handler
│ ├── shared/ # Shared infrastructure layer
│ │ ├── config/
│ │ │ └── config.ts # Zod-validated environment config
│ │ ├── database/
│ │ │ ├── schema.ts # Drizzle ORM schema
│ │ │ ├── drizzle.ts # PostgreSQL connection
│ │ │ ├── migrate.ts # Migration runner
│ │ ├── errors/
│ │ │ └── errors.ts # Custom error classes
│ │ ├── logger/
│ │ │ ├── logger.ts # Winston logger wrapper
│ │ │ └── serialization.ts # Log serialization
│ │ ├── utils/
│ │ │ └── retry.ts # Retry with exponential backoff
│ │ └── discord/
│ │ └── clientOptions.ts # Discord.js client config
│ ├── modules/ # Feature modules (Modular MVC)
│ │ ├── message-capture/ # Controller-Service-Repository
│ │ │ ├── messageCapture.ts # Controller: Discord event listeners
│ │ │ ├── messageStore.ts # Repository: DB operations
│ │ │ ├── messageMetadata.ts # Service: Metadata extraction
│ │ │ ├── types.ts # Domain types
│ │ │ └── index.ts # Module exports
│ │ ├── ai-moderation/ # Controller-Service-Repository
│ │ │ ├── aiAnalyzer.ts # Controller: Analysis orchestration
│ │ │ ├── llmModerationClient.ts # Service: LLM API client
│ │ │ ├── aiAnalysisWorker.ts # Service: Worker pool
│ │ │ ├── indonesianTextNormalizer.ts # Service: Text normalization
│ │ │ ├── moderationPrompt.ts # Service: Prompt generation
│ │ │ └── index.ts # Module exports
│ │ ├── attachment-upload/ # Controller-Service-Repository
│ │ │ ├── attachmentUploader.ts # Service: Upload orchestration
│ │ │ ├── imageResizer.ts # Service: Image resizing
│ │ │ └── index.ts # Module exports
│ │ └── event-broadcaster/ # Event-driven layer
│ │ ├── eventBroadcaster.ts # Service: Redis pub/sub publisher
│ │ ├── eventTypes.ts # Domain: Event type definitions
│ │ └── index.ts # Module exports
│ ├── mock-crc.ts # CRC polyfill for discord.js
│ └── index.ts # Service entry point
├── ARCHITECTURE.md # Detailed architecture documentation
├── package.json # Service dependencies
└── tsconfig.json # TypeScript configuration (inherited)
```bash
pnpm install
pnpm typecheck # tsc --noEmit
pnpm lint # biome check --diagnostic-level=error .
pnpm test # vitest run (138 tests)
pnpm build # tsc — CI/prod builds run this inside nix, which also
# runs scripts/fix-imports.mjs to rewrite @/ aliases and
# extensionless imports for Node ESM
pnpm dev # tsx watch src/index.ts
pnpm start # node dist/index.js
```
## Architecture Patterns
Deployment is CI-only: `nix build .#discord-gateway` → Attic cache → systemd
restart on the VPS. Do not build/hand-copy the artifact.
### 1. Modular MVC Structure
Each feature module follows **Controller-Service-Repository** pattern:
**Message Capture Module**:
- **Controller** (`messageCapture.ts`): Listens to Discord events (messageCreate, messageUpdate, messageDelete)
- **Service** (`messageMetadata.ts`): Extracts and normalizes message metadata
- **Repository** (`messageStore.ts`): Database CRUD operations
**AI Moderation Module**:
- **Controller** (`aiAnalyzer.ts`): Orchestrates analysis workflow
- **Service** (`llmModerationClient.ts`): LLM API integration
- **Service** (`aiAnalysisWorker.ts`): Worker pool management
- **Service** (`indonesianTextNormalizer.ts`): Text preprocessing
**Attachment Upload Module**:
- **Service** (`attachmentUploader.ts`): Upload orchestration
- **Service** (`imageResizer.ts`): Image processing
### 2. Event-Driven Architecture
**Redis Pub/Sub** replaces WebSocket broadcaster:
## Layout
```
Discord Events → Discord Gateway Service → Redis Pub/Sub → Backend Service
↓
Event Channels:
- discord:message:created
- discord:message:updated
- discord:message:deleted
- discord:message:analyzed
- discord:attachment:created
- discord:attachment:uploaded
- discord:analysis:queue_status
src/
├── index.ts # entry → initializeDiscordGateway()
├── app/ # process lifecycle
│ ├── bootstrap.ts # startup order: config → DB → services → metrics → login
│ ├── lifecycle.ts # everything wired on the Discord 'ready' hook
│ ├── process-guards.ts # SIGINT/SIGTERM + uncaught error policy
│ ├── metrics-collector.ts # AI pipeline Prometheus gauges
│ ├── shutdown.ts # graceful shutdown sequence
│ └── retention.ts # expired-record cleanup scheduler
├── shared/ # infrastructure — never imports from modules/
│ ├── config/ database/ logger/ errors/ utils/
│ ├── discord/clientOptions.ts
│ ├── redis-channels.ts # canonical Redis channel + command constants
│ └── moderation-types.ts # domain types shared across services
└── modules/ # feature modules (each exposes an index.ts facade)
├── ai-moderation/ # LLM moderation pipeline (largest module)
├── message-capture/ # Discord listeners + message/attachment DB
├── attachment-upload/ # download → resize → upload
├── event-broadcaster/ # Redis pub/sub publisher
├── command-handler/ # backend → gateway commands over Redis
├── gateway-metrics/ # Prometheus /metrics (port 4016)
├── monitor/ # weekly digest scheduler
└── reaction-tracking/ thread-tracking/ user-presence/
channel-topic/ guild-member-events/
```
### 3. Shared Infrastructure Layer
Centralized, reusable components:
- **Config**: Zod-validated environment variables
- **Logger**: Winston logger with context support
- **Database**: Drizzle ORM with PostgreSQL
- **Errors**: Custom error classes with codes and HTTP status codes
- **Utils**: Retry logic with exponential backoff
- **Discord**: Client configuration and options
Dependency direction is one-way: `index.ts` → `app/` → `modules/` → `shared/`.
Callers outside a module import its `index.ts` facade, never an internal file.
### 4. No HTTP Server
- **Event-driven only**: No Express, WebSocket, or HTTP routes
- **Redis pub/sub**: All inter-service communication via Redis
- **Backend service**: Consumes events and serves HTTP API
- **Frontend**: Continues to use Backend HTTP API
## Testing
## Key Features
### Message Capture
1. Discord emits `messageCreate`, `messageUpdate`, `messageDelete` events
2. `messageCapture.ts` listener receives and validates event
3. Extract metadata: user, channel, content, timestamp, attachments
4. `messageStore.ts` inserts into PostgreSQL
5. `eventBroadcaster.messageCreated()` publishes to Redis
6. Backend service subscribes and processes
### AI Moderation
1. `aiAnalyzer.ts` queues messages for analysis
2. `llmModerationClient.ts` calls LLM API with context
3. `indonesianTextNormalizer.ts` preprocesses text
4. Results stored in database
5. `eventBroadcaster.messageAnalyzed()` publishes results
6. Backend service receives and updates UI
### Attachment Upload
1. `messageCapture.ts` detects attachments
2. `attachmentUploader.ts` downloads from Discord
3. `imageResizer.ts` resizes images if needed
4. Upload to external storage with retry logic
5. `eventBroadcaster.attachmentUploaded()` publishes
6. Backend service stores metadata
## Initialization Flow
```
1. Load environment config (Zod validation)
↓
2. Initialize PostgreSQL connection
↓
3. Run pending database migrations
↓
4. Create Discord client with optimized cache
↓
5. Initialize Redis event broadcaster
↓
6. Register Discord event listeners
- messageCapture (message events)
- aiAnalyzer (analysis worker)
↓
7. Login to Discord
↓
8. Listen for graceful shutdown signals
```
## Graceful Shutdown
On SIGINT/SIGTERM/uncaughtException/unhandledRejection:
1. Close PostgreSQL connection
2. Close Redis connection
3. Destroy Discord client
4. Exit process (code 0 for clean, 1 for error)
## Dependencies
**Core Discord**:
- `discord.js-selfbot-v13` — Discord client (selfbot variant)
**Media Processing**:
- `sharp` — Image resizing
**Data & Config**:
- `drizzle-orm` — Type-safe ORM
- `pg` — PostgreSQL driver
- `zod` — Config validation
- `ioredis` — Redis client
**Logging & Utilities**:
- `winston` — Structured logging
- `p-retry` — Retry with backoff
- `p-limit` — Concurrency limiting
- `piscina` — Worker pool
## No Breaking Changes
- Original `src/` remains untouched
- Discord Gateway is a **new service** in `services/discord-gateway/`
- Can run alongside existing monolith during transition
- Backend service will consume Redis events
- Frontend continues to use Backend HTTP API
## Next Steps
1. **Create Backend service** (`services/backend/`)
- HTTP API endpoints
- Redis event subscribers
- Database models
- WebSocket broadcaster
2. **Update Frontend** (`frontend/`)
- Connect to Backend HTTP API
- Subscribe to WebSocket events
3. **Nix & CI/CD**
- flake.nix package for Discord Gateway
- systemd services (gmw-backend, gmw-discord-gateway)
- GitHub Actions for build/deploy (nix copy → systemctl restart)
4. **Documentation**
- API documentation
- Event schema documentation
- Deployment guide
## Files Created
**Total: 43 files**
### Shared Infrastructure (9 files)
- `src/shared/config/config.ts`
- `src/shared/database/` (5 files)
- `@bete/shared/errors` (shared package)
- `src/shared/logger/logger.ts`
- `src/shared/logger/serialization.ts`
- `src/shared/utils/retry.ts`
- `src/shared/discord/clientOptions.ts`
### Modules (28 files)
- `src/modules/message-capture/` (5 files)
- `src/modules/ai-moderation/` (6 files)
- `src/modules/attachment-upload/` (3 files)
- `src/modules/event-broadcaster/` (3 files)
### App & Entry (4 files)
- `src/app/bootstrap.ts`
- `src/app/shutdown.ts`
- `src/index.ts`
- `src/mock-crc.ts`
### Configuration (2 files)
- `package.json`
- `ARCHITECTURE.md`
## Verification Checklist
✅ Directory structure created
✅ Shared infrastructure migrated
✅ Message capture module migrated
✅ AI moderation module migrated
✅ Attachment upload module migrated
✅ Event broadcaster module created (Redis pub/sub)
✅ Bootstrap and entry point created
✅ Package.json with dependencies
✅ No HTTP server code (Express, WebSocket removed)
✅ Event-driven architecture implemented
✅ Graceful shutdown handler
✅ Module index files for clean exports
✅ Architecture documentation
## Event Flow Diagram
```
┌─────────────────────────────────────────────────────────────────┐
│ Discord Gateway Service │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌────────────────────────────┐ ┌────────────────────────────┐ │
│ │ Message Capture │ │ AI Moderation │ │
│ │ (Controller) │ │ (Controller) │ │
│ └──────────────┬─────────────┘ └──────────────┬─────────────┘ │
│ │ │ │
│ ├───────────────────────────────┤ │
│ │ │ │
│ ▼ ▼ │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ Event Broadcaster (Redis Pub/Sub) │ │
│ │ - discord:message:created │ │
│ │ - discord:message:updated │ │
│ │ - discord:message:deleted │ │
│ │ - discord:message:analyzed │ │
│ │ - discord:attachment:created │ │
│ │ - discord:attachment:uploaded │ │
│ │ - discord:analysis:queue_status │ │
│ └───────────────────────────────────────────────────────────┘ │
│ │ │
└────────────────────────────────┼────────────────────────────────┘
│
│ Redis Pub/Sub
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Backend Service │
│ (Subscribes to events, serves HTTP API, manages WebSocket) │
└─────────────────────────────────────────────────────────────────┘
│
│ HTTP API
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Frontend Application │
│ (React SPA, real-time updates via WebSocket) │
└─────────────────────────────────────────────────────────────────┘
```
## Summary
The Discord Gateway service has been successfully extracted with:
- **Modular MVC architecture** for clean separation of concerns
- **Event-driven design** using Redis pub/sub for inter-service communication
- **Shared infrastructure layer** for reusable components
- **No HTTP server** — pure event-driven service
- **Graceful shutdown** handling
- **Type-safe configuration** with Zod validation
- **Structured logging** with Winston
- **PostgreSQL integration** with Drizzle ORM
The service is ready for integration with the Backend service, which will consume Redis events and serve the HTTP API to the Frontend.
Vitest, tests in `tests/`. Config supplies dummy env vars so the suite runs
without live Postgres/Redis/Qdrant; external services are mocked. `llmE2e.test.ts`
is skipped by default and needs real credentials (`pnpm test:e2e:live`).