364 lines
16 KiB
Markdown
364 lines
16 KiB
Markdown
# Discord Gateway Service - Extraction Complete
|
|
|
|
## Overview
|
|
|
|
Successfully extracted Discord Gateway service with **Modular MVC + Event-Driven Architecture** using Redis pub/sub for inter-service communication.
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
services/discord-gateway/
|
|
├── src/
|
|
│ ├── app/
|
|
│ │ ├── bootstrap.ts # Service initialization (Discord client, DB, Redis)
|
|
│ │ └── shutdown.ts # Graceful shutdown handler
|
|
│ ├── shared/ # Shared infrastructure layer
|
|
│ │ ├── config/
|
|
│ │ │ └── config.ts # Zod-validated environment config
|
|
│ │ ├── database/
|
|
│ │ │ ├── schema.ts # Drizzle ORM schema
|
|
│ │ │ ├── drizzle.ts # PostgreSQL connection
|
|
│ │ │ ├── migrate.ts # Migration runner
|
|
│ │ │ └── voiceRecordingRepo.ts
|
|
│ │ ├── errors/
|
|
│ │ │ └── errors.ts # Custom error classes
|
|
│ │ ├── logger/
|
|
│ │ │ ├── logger.ts # Winston logger wrapper
|
|
│ │ │ └── serialization.ts # Log serialization
|
|
│ │ ├── utils/
|
|
│ │ │ └── retry.ts # Retry with exponential backoff
|
|
│ │ └── discord/
|
|
│ │ └── clientOptions.ts # Discord.js client config
|
|
│ ├── modules/ # Feature modules (Modular MVC)
|
|
│ │ ├── message-capture/ # Controller-Service-Repository
|
|
│ │ │ ├── messageCapture.ts # Controller: Discord event listeners
|
|
│ │ │ ├── messageStore.ts # Repository: DB operations
|
|
│ │ │ ├── messageMetadata.ts # Service: Metadata extraction
|
|
│ │ │ ├── types.ts # Domain types
|
|
│ │ │ └── index.ts # Module exports
|
|
│ │ ├── ai-moderation/ # Controller-Service-Repository
|
|
│ │ │ ├── aiAnalyzer.ts # Controller: Analysis orchestration
|
|
│ │ │ ├── llmModerationClient.ts # Service: LLM API client
|
|
│ │ │ ├── aiAnalysisWorker.ts # Service: Worker pool
|
|
│ │ │ ├── indonesianTextNormalizer.ts # Service: Text normalization
|
|
│ │ │ ├── moderationPrompt.ts # Service: Prompt generation
|
|
│ │ │ └── index.ts # Module exports
|
|
│ │ ├── voice-recording/ # Controller-Service-Repository
|
|
│ │ │ ├── voiceController.ts # Controller: Voice connection mgmt
|
|
│ │ │ ├── recorder.ts # Service: Recording orchestration
|
|
│ │ │ ├── recorder/ # Sub-services
|
|
│ │ │ │ ├── audioStream.ts # Audio stream subscription
|
|
│ │ │ │ ├── decoder.ts # Opus decoding
|
|
│ │ │ │ ├── segment.ts # OGG segment rotation
|
|
│ │ │ │ ├── metadata.ts # Segment metadata
|
|
│ │ │ │ ├── sessionRecording.ts # Session management
|
|
│ │ │ │ └── uploader.ts # Segment upload
|
|
│ │ │ └── index.ts # Module exports
|
|
│ │ ├── attachment-upload/ # Controller-Service-Repository
|
|
│ │ │ ├── attachmentUploader.ts # Service: Upload orchestration
|
|
│ │ │ ├── imageResizer.ts # Service: Image resizing
|
|
│ │ │ └── index.ts # Module exports
|
|
│ │ └── event-broadcaster/ # Event-driven layer
|
|
│ │ ├── eventBroadcaster.ts # Service: Redis pub/sub publisher
|
|
│ │ ├── eventTypes.ts # Domain: Event type definitions
|
|
│ │ └── index.ts # Module exports
|
|
│ ├── mock-crc.ts # CRC polyfill for discord.js
|
|
│ └── index.ts # Service entry point
|
|
├── ARCHITECTURE.md # Detailed architecture documentation
|
|
├── package.json # Service dependencies
|
|
└── tsconfig.json # TypeScript configuration (inherited)
|
|
```
|
|
|
|
## Architecture Patterns
|
|
|
|
### 1. Modular MVC Structure
|
|
Each feature module follows **Controller-Service-Repository** pattern:
|
|
|
|
**Message Capture Module**:
|
|
- **Controller** (`messageCapture.ts`): Listens to Discord events (messageCreate, messageUpdate, messageDelete)
|
|
- **Service** (`messageMetadata.ts`): Extracts and normalizes message metadata
|
|
- **Repository** (`messageStore.ts`): Database CRUD operations
|
|
|
|
**AI Moderation Module**:
|
|
- **Controller** (`aiAnalyzer.ts`): Orchestrates analysis workflow
|
|
- **Service** (`llmModerationClient.ts`): LLM API integration
|
|
- **Service** (`aiAnalysisWorker.ts`): Worker pool management
|
|
- **Service** (`indonesianTextNormalizer.ts`): Text preprocessing
|
|
|
|
**Voice Recording Module**:
|
|
- **Controller** (`voiceController.ts`): Voice channel connection management
|
|
- **Service** (`recorder.ts`): Recording orchestration
|
|
- **Sub-services** (`recorder/*`): Audio stream, decoding, segmentation, upload
|
|
|
|
**Attachment Upload Module**:
|
|
- **Service** (`attachmentUploader.ts`): Upload orchestration
|
|
- **Service** (`imageResizer.ts`): Image processing
|
|
|
|
### 2. Event-Driven Architecture
|
|
**Redis Pub/Sub** replaces WebSocket broadcaster:
|
|
|
|
```
|
|
Discord Events → Discord Gateway Service → Redis Pub/Sub → Backend Service
|
|
↓
|
|
Event Channels:
|
|
- discord:message:created
|
|
- discord:message:updated
|
|
- discord:message:deleted
|
|
- discord:message:analyzed
|
|
- discord:attachment:created
|
|
- discord:attachment:uploaded
|
|
- discord:voice:started
|
|
- discord:voice:stopped
|
|
- discord:voice:uploaded
|
|
- discord:analysis:queue_status
|
|
```
|
|
|
|
### 3. Shared Infrastructure Layer
|
|
Centralized, reusable components:
|
|
- **Config**: Zod-validated environment variables
|
|
- **Logger**: Winston logger with context support
|
|
- **Database**: Drizzle ORM with PostgreSQL
|
|
- **Errors**: Custom error classes with codes and HTTP status codes
|
|
- **Utils**: Retry logic with exponential backoff
|
|
- **Discord**: Client configuration and options
|
|
|
|
### 4. No HTTP Server
|
|
- **Event-driven only**: No Express, WebSocket, or HTTP routes
|
|
- **Redis pub/sub**: All inter-service communication via Redis
|
|
- **Backend service**: Consumes events and serves HTTP API
|
|
- **Frontend**: Continues to use Backend HTTP API
|
|
|
|
## Key Features
|
|
|
|
### Message Capture
|
|
1. Discord emits `messageCreate`, `messageUpdate`, `messageDelete` events
|
|
2. `messageCapture.ts` listener receives and validates event
|
|
3. Extract metadata: user, channel, content, timestamp, attachments
|
|
4. `messageStore.ts` inserts into PostgreSQL
|
|
5. `eventBroadcaster.messageCreated()` publishes to Redis
|
|
6. Backend service subscribes and processes
|
|
|
|
### AI Moderation
|
|
1. `aiAnalyzer.ts` queues messages for analysis
|
|
2. `llmModerationClient.ts` calls LLM API with context
|
|
3. `indonesianTextNormalizer.ts` preprocesses text
|
|
4. Results stored in database
|
|
5. `eventBroadcaster.messageAnalyzed()` publishes results
|
|
6. Backend service receives and updates UI
|
|
|
|
### Voice Recording
|
|
1. `voiceController.connect()` joins voice channel
|
|
2. `recorder.ts` subscribes to user audio streams
|
|
3. For each speaking user:
|
|
- `audioStream.ts` subscribes to Opus packets
|
|
- `decoder.ts` decodes Opus to PCM
|
|
- `segment.ts` rotates OGG files (5s default)
|
|
- `metadata.ts` collects user info
|
|
4. On silence (3s):
|
|
- `sessionRecording.ts` finalizes segment
|
|
- `uploader.ts` uploads to storage
|
|
- `eventBroadcaster.voiceRecordingUploaded()` publishes
|
|
5. Backend service indexes recording
|
|
|
|
### Attachment Upload
|
|
1. `messageCapture.ts` detects attachments
|
|
2. `attachmentUploader.ts` downloads from Discord
|
|
3. `imageResizer.ts` resizes images if needed
|
|
4. Upload to external storage with retry logic
|
|
5. `eventBroadcaster.attachmentUploaded()` publishes
|
|
6. Backend service stores metadata
|
|
|
|
## Initialization Flow
|
|
|
|
```
|
|
1. Load environment config (Zod validation)
|
|
↓
|
|
2. Initialize PostgreSQL connection
|
|
↓
|
|
3. Run pending database migrations
|
|
↓
|
|
4. Create Discord client with optimized cache
|
|
↓
|
|
5. Initialize Redis event broadcaster
|
|
↓
|
|
6. Register Discord event listeners
|
|
- messageCapture (message events)
|
|
- aiAnalyzer (analysis worker)
|
|
↓
|
|
7. Login to Discord
|
|
↓
|
|
8. Listen for graceful shutdown signals
|
|
```
|
|
|
|
## Graceful Shutdown
|
|
|
|
On SIGINT/SIGTERM/uncaughtException/unhandledRejection:
|
|
1. Close PostgreSQL connection
|
|
2. Disconnect from voice channels
|
|
3. Close Redis connection
|
|
4. Destroy Discord client
|
|
5. Exit process (code 0 for clean, 1 for error)
|
|
|
|
## Dependencies
|
|
|
|
**Core Discord**:
|
|
- `discord.js-selfbot-v13` — Discord client (selfbot variant)
|
|
- `@discordjs/voice` — Voice connection management
|
|
- `@discordjs/opus` — Native Opus codec
|
|
|
|
**Audio Processing**:
|
|
- `prism-media` — Opus encoding/decoding
|
|
- `opusscript` — Opus fallback for Node v26+
|
|
- `sharp` — Image resizing
|
|
|
|
**Data & Config**:
|
|
- `drizzle-orm` — Type-safe ORM
|
|
- `pg` — PostgreSQL driver
|
|
- `zod` — Config validation
|
|
- `ioredis` — Redis client
|
|
|
|
**Logging & Utilities**:
|
|
- `winston` — Structured logging
|
|
- `p-retry` — Retry with backoff
|
|
- `p-limit` — Concurrency limiting
|
|
- `piscina` — Worker pool
|
|
|
|
## No Breaking Changes
|
|
|
|
- Original `src/` remains untouched
|
|
- Discord Gateway is a **new service** in `services/discord-gateway/`
|
|
- Can run alongside existing monolith during transition
|
|
- Backend service will consume Redis events
|
|
- Frontend continues to use Backend HTTP API
|
|
|
|
## Next Steps
|
|
|
|
1. **Create Backend service** (`services/backend/`)
|
|
- HTTP API endpoints
|
|
- Redis event subscribers
|
|
- Database models
|
|
- WebSocket broadcaster
|
|
|
|
2. **Update Frontend** (`frontend/`)
|
|
- Connect to Backend HTTP API
|
|
- Subscribe to WebSocket events
|
|
|
|
3. **Nix & CI/CD**
|
|
- flake.nix package for Discord Gateway
|
|
- systemd services (gmw-backend, gmw-discord-gateway)
|
|
- GitHub Actions for build/deploy (nix copy → systemctl restart)
|
|
|
|
4. **Documentation**
|
|
- API documentation
|
|
- Event schema documentation
|
|
- Deployment guide
|
|
|
|
## Files Created
|
|
|
|
**Total: 43 files**
|
|
|
|
### Shared Infrastructure (9 files)
|
|
- `src/shared/config/config.ts`
|
|
- `src/shared/database/` (5 files)
|
|
- `@bete/shared/errors` (shared package)
|
|
- `src/shared/logger/logger.ts`
|
|
- `src/shared/logger/serialization.ts`
|
|
- `src/shared/utils/retry.ts`
|
|
- `src/shared/discord/clientOptions.ts`
|
|
|
|
### Modules (28 files)
|
|
- `src/modules/message-capture/` (5 files)
|
|
- `src/modules/ai-moderation/` (6 files)
|
|
- `src/modules/voice-recording/` (9 files)
|
|
- `src/modules/attachment-upload/` (3 files)
|
|
- `src/modules/event-broadcaster/` (3 files)
|
|
|
|
### App & Entry (4 files)
|
|
- `src/app/bootstrap.ts`
|
|
- `src/app/shutdown.ts`
|
|
- `src/index.ts`
|
|
- `src/mock-crc.ts`
|
|
|
|
### Configuration (2 files)
|
|
- `package.json`
|
|
- `ARCHITECTURE.md`
|
|
|
|
## Verification Checklist
|
|
|
|
✅ Directory structure created
|
|
✅ Shared infrastructure migrated
|
|
✅ Message capture module migrated
|
|
✅ AI moderation module migrated
|
|
✅ Voice recording module migrated
|
|
✅ Attachment upload module migrated
|
|
✅ Event broadcaster module created (Redis pub/sub)
|
|
✅ Bootstrap and entry point created
|
|
✅ Package.json with dependencies
|
|
✅ No HTTP server code (Express, WebSocket removed)
|
|
✅ Event-driven architecture implemented
|
|
✅ Graceful shutdown handler
|
|
✅ Module index files for clean exports
|
|
✅ Architecture documentation
|
|
|
|
## Event Flow Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Discord Gateway Service │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────┐ │
|
|
│ │ Message Capture │ │ AI Moderation │ │ Voice Record │ │
|
|
│ │ (Controller) │ │ (Controller) │ │ (Controller) │ │
|
|
│ └────────┬─────────┘ └────────┬─────────┘ └──────┬───────┘ │
|
|
│ │ │ │ │
|
|
│ ├─────────────────────┼───────────────────┤ │
|
|
│ │ │ │ │
|
|
│ ▼ ▼ ▼ │
|
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
|
│ │ Event Broadcaster (Redis Pub/Sub) │ │
|
|
│ │ - discord:message:created │ │
|
|
│ │ - discord:message:updated │ │
|
|
│ │ - discord:message:deleted │ │
|
|
│ │ - discord:message:analyzed │ │
|
|
│ │ - discord:attachment:created │ │
|
|
│ │ - discord:attachment:uploaded │ │
|
|
│ │ - discord:voice:started │ │
|
|
│ │ - discord:voice:stopped │ │
|
|
│ │ - discord:voice:uploaded │ │
|
|
│ │ - discord:analysis:queue_status │ │
|
|
│ └─────────────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
└───────────┼────────────────────────────────────────────────────┘
|
|
│
|
|
│ Redis Pub/Sub
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Backend Service │
|
|
│ (Subscribes to events, serves HTTP API, manages WebSocket) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
│ HTTP API
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Frontend Application │
|
|
│ (React SPA, real-time updates via WebSocket) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Summary
|
|
|
|
The Discord Gateway service has been successfully extracted with:
|
|
- **Modular MVC architecture** for clean separation of concerns
|
|
- **Event-driven design** using Redis pub/sub for inter-service communication
|
|
- **Shared infrastructure layer** for reusable components
|
|
- **No HTTP server** — pure event-driven service
|
|
- **Graceful shutdown** handling
|
|
- **Type-safe configuration** with Zod validation
|
|
- **Structured logging** with Winston
|
|
- **PostgreSQL integration** with Drizzle ORM
|
|
|
|
The service is ready for integration with the Backend service, which will consume Redis events and serve the HTTP API to the Frontend.
|