MythEclipse/GMW
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: remove voice/recording/media references from AGENTS/ARCHITECTURE/README

Browse Source
This commit is contained in:
asepharyana committed 2026-09-23 21:25:03 +07:00
1 parent 2c69c51fba
commit a13884d80f
9 files changed
+39 -142
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

services/discord-gateway/AGENTS.md
+1 -25
View File
@@ -36,9 +36,6 @@ src/
├── modules/
│ ├── ai-moderation/ # LLM moderation pipeline (largest module)
│ ├── message-capture/ # Discord event listeners + DB store
│ ├── voice-recording/ # Voice connect + Opus→OGG recording
│ │ └── recorder/ # decoder, segment, session, uploader, oggCrc
│ ├── voice-pcm-ws/ # Real-time PCM → backend WebSocket
│ ├── attachment-upload/ # Download + sharp resize + upload
│ ├── event-broadcaster/ # Redis pub/sub publisher
│ ├── command-handler/ # Backend→gateway Redis commands
@@ -76,24 +73,6 @@ textBatchProcessor.ts mediaBatchProcessor.ts llmClient.ts
- Piscina: text pool (4 threads) + media pool (2 threads)
- **Each worker thread has its own pg Pool** (min 0, grows to `POSTGRES_POOL_MAX`)
## Voice recording pipeline
```
receiver.speaking "start" → speakingHandler(userId)
→ collectUserMetadata → receiver.subscribe → PacketFilter → oggPacketStream
→ SegmentManager.open → OggLogicalBitstream → .ogg file
→ data: rotateIfNeeded + decoder.write
→ end: finalizeSegment → upload + transcribe
```
Key files:
- `voiceController.ts` — connect/disconnect/list
- `recorder.ts` — orchestration
- `recorder/segment.ts` — segment rotation
- `recorder/sessionRecording.ts` — session management
- `recorder/uploader.ts` — upload to storage
- `voiceTranscriber.ts` — Whisper transcription (if enabled)
## Module: message-capture
- `messageCapture.ts` — Discord event listeners (messageCreate/Update/Delete)
@@ -107,7 +86,7 @@ Key files:
See `src/shared/redis-channels.ts` for canonical names. Examples:
```
discord:message:created, discord:voice:active_user, discord:attachment:uploaded
discord:message:created, discord:moderation:action, discord:attachment:uploaded
```
## Config (env vars)
@@ -120,7 +99,6 @@ All validated via Zod in `shared/config/index.ts`. Critical:
- `REDIS_URL` — pub/sub to backend
- `AI_ANALYSIS_ENABLED` — master toggle for AI moderation
- `AI_LLM_BASE_URL` / `AI_LLM_API_KEY` — LLM router
- `AI_VOICE_TRANSCRIPTION_ENABLED` — toggle Whisper transcription
- `PISCINA_MAX_THREADS` / `PISCINA_MEDIA_MAX_THREADS` — worker pool sizing
## Concurrency model
@@ -143,8 +121,6 @@ All validated via Zod in `shared/config/index.ts`. Critical:
- **Piscina pool isolation**: worker threads are NOT the main thread. Cannot
share state via module-level variables. Use DB or Redis for cross-thread state.
- **AfterSilence race**: `@discordjs/voice` AfterSilence can fail to emit "end"
on disconnect. Always have a watchdog/timeout.
- **Cache eviction**: LRU caches (user metadata, term glossary) evict at max size.
Don't assume cache hit after eviction.
- **Circuit breaker**: per-conversation CB opens after repeated failures.
services/discord-gateway/ARCHITECTURE.md
+5 -9
View File
@@ -1,7 +1,7 @@
# Discord Gateway — Architecture
Pure event-driven microservice (no HTTP server). Captures Discord
messages/voice/attachments/reactions/threads/presence, runs LLM-based AI
messages/attachments/reactions/threads/presence, runs LLM-based AI
moderation, and publishes everything to Redis pub/sub for the backend to
consume. The backend serves the HTTP/WS API to the frontend.
@@ -24,9 +24,9 @@ services/discord-gateway/
│ │ ├── config/ # Zod-validated env (index.ts = schema+loader)
│ │ ├── database/ # Drizzle ORM + pg Pool + migrations
│ │ │ ├── init.ts drizzle.ts pool.ts migrate.ts migrateCli.ts
│ │ │ └── schema/ # messages, cache, voice, analytics, meta
│ │ │ └── schema/ # messages, cache, meta, analytics
│ │ ├── logger/ # pino wrapper + createChildLogger()
│ │ ├── errors/ # AppError / ConfigError / AudioError ...
│ │ ├── errors/ # AppError / ConfigError ...
│ │ ├── utils/ # retry, pagination
│ │ ├── discord/clientOptions.ts # discord.js-selfbot-v13 client options
│ │ ├── uploader.ts # Shared attachment upload helper
@@ -35,9 +35,6 @@ services/discord-gateway/
│ └── modules/
│ ├── message-capture/ # Discord event listeners + DB store
│ ├── ai-moderation/ # LLM moderation pipeline (see below)
│ ├── voice-recording/ # Voice connect + Opus→OGG recording
│ │ └── recorder/ # decoder, segment, session, uploader, oggCrc
│ ├── voice-pcm-ws/ # Real-time PCM → backend WebSocket (bypasses Redis)
│ ├── attachment-upload/ # Download + (sharp) resize + upload
│ ├── event-broadcaster/ # RedisEventPublisher + EventBroadcaster
│ ├── command-handler/ # Redis-subscribed backend→gateway commands
@@ -102,7 +99,6 @@ grows on demand up to `POSTGRES_POOL_MAX`.
`discord:message:{created,updated,deleted,analyzed}`,
`discord:attachment:{created,uploaded}`,
`discord:voice:{started,stopped,uploaded,active_user,pcm,analyzed}`,
`discord:analysis:queue_status`,
`discord:reaction:{added,removed}`,
`discord:thread:{created,deleted,updated}`,
@@ -124,8 +120,8 @@ See `src/shared/redis-channels.ts` for the canonical names.
`SIGINT`/`SIGTERM` (and uncaught transient stream errors: EPIPE / ECONNRESET /
ERR_STREAM_DESTROYED / ERR_STREAM_WRITE_AFTER_END are treated as non-fatal):
stop metrics → stop muxer → disconnect voice → close PCM WS → close Redis →
close command handler → close DB → destroy client → exit.
stop metrics → close event broadcaster (Redis) → close command handler →
close DB → destroy client → exit.
## Observability
services/discord-gateway/MODULE_STRUCTURE.md
+1 -8
View File
@@ -18,8 +18,6 @@ services/discord-gateway/
│ └── modules/
│ ├── message-capture/ # Discord listeners + DB store + metadata
│ ├── ai-moderation/ # LLM moderation pipeline (largest module)
│ ├── voice-recording/ # Voice connect + Opus→OGG recording (+ recorder/)
│ ├── voice-pcm-ws/ # Real-time PCM → backend WebSocket
│ ├── attachment-upload/ # Download + sharp resize + upload
│ ├── event-broadcaster/ # RedisEventPublisher + EventBroadcaster
│ ├── command-handler/ # Backend→gateway Redis commands
@@ -51,11 +49,6 @@ semantic Qdrant → LLM), `textBatchProcessor.ts` / `mediaBatchProcessor.ts`
`embeddingClient.ts` + `qdrantClient.ts` (semantic cache), plus
`channelCultureStore.ts` / `userProfileStore.ts`.
### voice-recording
`voiceController.ts` (connect/disconnect/list) + `recorder.ts` (orchestration)
+ `recorder/` (decoder, segment, session, uploader, oggCrc). Publishes
`discord:voice:*` events. Real-time audio also streamed via `voice-pcm-ws`.
### attachment-upload
`attachmentUploader.ts` (download → upload to storage) + `imageResizer.ts`
(sharp resize). Emits `discord:attachment:*`.
@@ -72,7 +65,7 @@ per scrape; live pipeline gauges registered in `bootstrap.ts`.
- **config** — Zod schema in `shared/config/index.ts` (single source of truth).
- **database** — Drizzle ORM over `pg`; pool `min:0` (`shared/config`).
- **logger** — `pino` wrapper, `createChildLogger()` for context loggers.
- **errors** — `AppError` hierarchy (`ConfigError`, `AudioError`, …).
- **errors** — `AppError` hierarchy (`ConfigError`, …).
## Notes
- No HTTP server (other than the metrics endpoint). Pure event-driven.
services/discord-gateway/README.md
+29 -73
View File
@@ -19,7 +19,6 @@ services/discord-gateway/
│ │ │ ├── schema.ts # Drizzle ORM schema
│ │ │ ├── drizzle.ts # PostgreSQL connection
│ │ │ ├── migrate.ts # Migration runner
│ │ │ └── voiceRecordingRepo.ts
│ │ ├── errors/
│ │ │ └── errors.ts # Custom error classes
│ │ ├── logger/
@@ -43,17 +42,6 @@ services/discord-gateway/
│ │ │ ├── indonesianTextNormalizer.ts # Service: Text normalization
│ │ │ ├── moderationPrompt.ts # Service: Prompt generation
│ │ │ └── index.ts # Module exports
│ │ ├── voice-recording/ # Controller-Service-Repository
│ │ │ ├── voiceController.ts # Controller: Voice connection mgmt
│ │ │ ├── recorder.ts # Service: Recording orchestration
│ │ │ ├── recorder/ # Sub-services
│ │ │ │ ├── audioStream.ts # Audio stream subscription
│ │ │ │ ├── decoder.ts # Opus decoding
│ │ │ │ ├── segment.ts # OGG segment rotation
│ │ │ │ ├── metadata.ts # Segment metadata
│ │ │ │ ├── sessionRecording.ts # Session management
│ │ │ │ └── uploader.ts # Segment upload
│ │ │ └── index.ts # Module exports
│ │ ├── attachment-upload/ # Controller-Service-Repository
│ │ │ ├── attachmentUploader.ts # Service: Upload orchestration
│ │ │ ├── imageResizer.ts # Service: Image resizing
@@ -85,11 +73,6 @@ Each feature module follows **Controller-Service-Repository** pattern:
- **Service** (`aiAnalysisWorker.ts`): Worker pool management
- **Service** (`indonesianTextNormalizer.ts`): Text preprocessing
**Voice Recording Module**:
- **Controller** (`voiceController.ts`): Voice channel connection management
- **Service** (`recorder.ts`): Recording orchestration
- **Sub-services** (`recorder/*`): Audio stream, decoding, segmentation, upload
**Attachment Upload Module**:
- **Service** (`attachmentUploader.ts`): Upload orchestration
- **Service** (`imageResizer.ts`): Image processing
@@ -107,9 +90,6 @@ Discord Events → Discord Gateway Service → Redis Pub/Sub → Backend Service
- discord:message:analyzed
- discord:attachment:created
- discord:attachment:uploaded
- discord:voice:started
- discord:voice:stopped
- discord:voice:uploaded
- discord:analysis:queue_status
```
@@ -146,20 +126,6 @@ Centralized, reusable components:
5. `eventBroadcaster.messageAnalyzed()` publishes results
6. Backend service receives and updates UI
### Voice Recording
1. `voiceController.connect()` joins voice channel
2. `recorder.ts` subscribes to user audio streams
3. For each speaking user:
- `audioStream.ts` subscribes to Opus packets
- `decoder.ts` decodes Opus to PCM
- `segment.ts` rotates OGG files (5s default)
- `metadata.ts` collects user info
4. On silence (3s):
- `sessionRecording.ts` finalizes segment
- `uploader.ts` uploads to storage
- `eventBroadcaster.voiceRecordingUploaded()` publishes
5. Backend service indexes recording
### Attachment Upload
1. `messageCapture.ts` detects attachments
2. `attachmentUploader.ts` downloads from Discord
@@ -194,21 +160,16 @@ Centralized, reusable components:
On SIGINT/SIGTERM/uncaughtException/unhandledRejection:
1. Close PostgreSQL connection
2. Disconnect from voice channels
3. Close Redis connection
4. Destroy Discord client
5. Exit process (code 0 for clean, 1 for error)
2. Close Redis connection
3. Destroy Discord client
4. Exit process (code 0 for clean, 1 for error)
## Dependencies
**Core Discord**:
- `discord.js-selfbot-v13` — Discord client (selfbot variant)
- `@discordjs/voice` — Voice connection management
- `@discordjs/opus` — Native Opus codec
**Audio Processing**:
- `prism-media` — Opus encoding/decoding
- `opusscript` — Opus fallback for Node v26+
**Media Processing**:
- `sharp` — Image resizing
**Data & Config**:
@@ -269,7 +230,6 @@ On SIGINT/SIGTERM/uncaughtException/unhandledRejection:
### Modules (28 files)
- `src/modules/message-capture/` (5 files)
- `src/modules/ai-moderation/` (6 files)
- `src/modules/voice-recording/` (9 files)
- `src/modules/attachment-upload/` (3 files)
- `src/modules/event-broadcaster/` (3 files)
@@ -289,7 +249,6 @@ On SIGINT/SIGTERM/uncaughtException/unhandledRejection:
✅ Shared infrastructure migrated
✅ Message capture module migrated
✅ AI moderation module migrated
✅ Voice recording module migrated
✅ Attachment upload module migrated
✅ Event broadcaster module created (Redis pub/sub)
✅ Bootstrap and entry point created
@@ -304,36 +263,33 @@ On SIGINT/SIGTERM/uncaughtException/unhandledRejection:
```
┌─────────────────────────────────────────────────────────────────┐
│ Discord Gateway Service │
│ Discord Gateway Service │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────┐ │
│ │ Message Capture │ │ AI Moderation │ │ Voice Record │ │
│ │ (Controller) │ │ (Controller) │ │ (Controller) │ │
│ └────────┬─────────┘ └────────┬─────────┘ └──────┬───────┘ │
│ │ │ │ │
│ ├─────────────────────┼───────────────────┤ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Event Broadcaster (Redis Pub/Sub) │ │
│ │ - discord:message:created │ │
│ │ - discord:message:updated │ │
│ │ - discord:message:deleted │ │
│ │ - discord:message:analyzed │ │
│ │ - discord:attachment:created │ │
│ │ - discord:attachment:uploaded │ │
│ │ - discord:voice:started │ │
│ │ - discord:voice:stopped │ │
│ │ - discord:voice:uploaded │ │
│ │ - discord:analysis:queue_status │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
└───────────┼────────────────────────────────────────────────────┘
│
│ Redis Pub/Sub
│
▼
│ ┌────────────────────────────┐ ┌────────────────────────────┐ │
│ │ Message Capture │ │ AI Moderation │ │
│ │ (Controller) │ │ (Controller) │ │
│ └──────────────┬─────────────┘ └──────────────┬─────────────┘ │
│ │ │ │
│ ├───────────────────────────────┤ │
│ │ │ │
│ ▼ ▼ │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ Event Broadcaster (Redis Pub/Sub) │ │
│ │ - discord:message:created │ │
│ │ - discord:message:updated │ │
│ │ - discord:message:deleted │ │
│ │ - discord:message:analyzed │ │
│ │ - discord:attachment:created │ │
│ │ - discord:attachment:uploaded │ │
│ │ - discord:analysis:queue_status │ │
│ └───────────────────────────────────────────────────────────┘ │
│ │ │
└────────────────────────────────┼────────────────────────────────┘
│
│ Redis Pub/Sub
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Backend Service │
│ (Subscribes to events, serves HTTP API, manages WebSocket) │