feat(docs): Add comprehensive agent guides and templates for spec-driven development

2026-09-11 18:56:28 +07:00
parent b3418ad799
commit 43f2f8449d
14 changed files with 727 additions and 5 deletions
@@ -0,0 +1,117 @@
+# Backend Service — Agent Guide
+
+> **Read `../../AGENTS.md` first.** This file adds backend-specific conventions.
+
+Backend service: Express HTTP server + WebSocket, serves the GMW dashboard API.
+
+## Quick reference
+
+```bash
+pnpm typecheck              # tsc --noEmit
+pnpm lint                   # biome check --diagnostic-level=error .
+pnpm build                  # tsc
+pnpm test                   # vitest run
+pnpm format                 # biome format --write .
+```
+
+## Architecture (Modular MVC)
+
+```
+src/
+├── shared/                          # Infrastructure (no business logic)
+│   ├── config/index.ts              # Zod-validated env
+│   ├── database/                    # Drizzle ORM + pg Pool
+│   ├── errors/index.ts              # AppError hierarchy
+│   ├── logger/index.ts              # pino + createChildLogger()
+│   ├── middlewares/index.ts         # errorHandler, asyncHandler, rateLimit
+│   └── utils/                       # Pagination, messageMapper
+├── modules/                         # Feature modules
+│   └── <module>/
+│       ├── <module>.schema.ts       # Zod validation schemas
+│       ├── <module>.repository.ts   # DB operations only
+│       ├── <module>.service.ts      # Business logic
+│       ├── <module>.controller.ts   # HTTP handlers
+│       └── routes/index.ts          # Express router
+├── http/                            # app.ts (factory) + server.ts (startup)
+├── ws/                              # WebSocket server + Redis bridge
+└── index.ts                         # Entry point
+```
+
+## Dependency rules
+
+- Controller → Service → Repository → Database
+- No cross-module repo imports (each module owns its data)
+- No HTTP in Service layer (no req/res)
+- No DB in Controller layer
+- Any layer → Config, Logger, Errors
+
+## API: oRPC + Express
+
+- **oRPC** (`src/orpc/router.ts` + `src/orpc/ws.ts`): type-safe procedures for frontend
+- **Express routes** (`src/modules/*/routes/`): REST endpoints under `/api/`
+- **WebSocket** (`src/ws/`): Redis bridge → broadcast to connected browsers
+- Never invent endpoints. Match what the frontend calls (`src/lib/api/`).
+
+## Key modules
+
+| Module | Purpose | DB tables |
+|---|---|---|
+| messages | Store & query Discord messages | `messages`, `ai_moderations`, `ai_moderation_flags` |
+| moderation | Moderation actions & metrics | `ai_moderations`, `moderation_actions` |
+| media | Media file management | `media_attachments` |
+| voice | Live speakers + recordings | `voice_recordings` |
+| recordings | Recordings API | `voice_recordings` |
+| dashboard | Stats aggregation | Various (read-only) |
+| knowledge | Semantic search | Qdrant vector DB |
+| chatbot | AI chatbot with tools | `chatbot_history` |
+| health | Health checks + metrics | Various |
+| analysis | Text analysis cache | `text_analysis_cache` |
+| ui-state | Persist UI preferences | `ui_state` |
+
+## Config (env vars)
+
+All validated via Zod in `shared/config/index.ts`. Key vars:
+
+- `WEBSERVER_PORT` (default 4001)
+- `DATABASE_URL` or individual `DATABASE_HOST/PORT/NAME/USER/PASSWORD`
+- `REDIS_URL` — for pub/sub with gateway
+- `MONITOR_GUILD_ID` — primary Discord guild
+- `ADMIN_PASSWORD` — admin endpoints
+
+## Data contract with frontend
+
+The frontend fetches via `src/lib/api/server.ts` (SSR, server-side) and
+`src/lib/api/client.ts` (browser, same-origin through proxy).
+
+**Do not change response shapes without updating both sides.** Check
+`services/frontend/src/lib/types/` for frontend type definitions.
+
+## Redis channels (inbound from gateway)
+
+```
+discord:message:{created,updated,deleted,analyzed}
+discord:attachment:{created,uploaded}
+discord:voice:{started,stopped,uploaded,active_user,pcm,analyzed}
+discord:analysis:queue_status
+discord:reaction:{added,removed}
+discord:thread:{created,deleted,updated}
+discord:channel_topic:updated
+discord:presence:updated
+discord:guild_member:{added,removed}
+```
+
+Canonical names: `src/shared/redis-channels.ts`.
+
+## Testing
+
+- Vitest for unit tests
+- Mock database and external services
+- Test files: `src/modules/<module>/*.test.ts` or `tests/*.test.ts`
+- Run: `pnpm test`
+
+## Common pitfalls
+
+- **oRPC vs REST**: check both routers when adding an endpoint
+- **Redis channel mismatch**: gateway publishes → backend subscribes. Channel
+  names must match exactly (see `redis-channels.ts` in BOTH services)
+- **DB pool**: backend uses one pool (main thread). Gateway has per-piscina-thread pools.
@@ -0,0 +1,153 @@
+# Discord Gateway — Agent Guide
+
+> **Read `../../AGENTS.md` first.** This file adds gateway-specific conventions.
+
+Event-driven microservice: captures Discord events, runs AI moderation, publishes to Redis.
+
+## Quick reference
+
+```bash
+pnpm typecheck              # tsc --noEmit
+pnpm lint                   # biome check --diagnostic-level=error .
+pnpm build                  # tsc
+pnpm test                   # vitest run
+pnpm format                 # biome format --write .
+```
+
+## Architecture (Event-driven)
+
+```
+src/
+├── index.ts                     # Entry point → initializeDiscordGateway()
+├── app/
+│   ├── bootstrap.ts             # Wires client, DB, Redis, workers, schedulers
+│   ├── shutdown.ts              # Graceful shutdown
+│   └── retention.ts             # Expired-record cleanup
+├── shared/
+│   ├── config/index.ts          # Zod-validated env (SINGLE source of truth)
+│   ├── database/                # Drizzle ORM + pg Pool + migrations
+│   ├── logger/index.ts          # pino + createChildLogger()
+│   ├── errors/index.ts          # AppError hierarchy
+│   ├── utils/                   # retry, pagination
+│   ├── discord/clientOptions.ts # discord.js-selfbot-v13 options
+│   ├── uploader.ts              # Attachment upload helper
+│   ├── redis-channels.ts        # Redis channel-name constants
+│   └── moderation-types.ts      # Shared AI analysis types
+├── modules/
+│   ├── ai-moderation/           # LLM moderation pipeline (largest module)
+│   ├── message-capture/         # Discord event listeners + DB store
+│   ├── voice-recording/         # Voice connect + Opus→OGG recording
+│   │   └── recorder/            # decoder, segment, session, uploader, oggCrc
+│   ├── voice-pcm-ws/            # Real-time PCM → backend WebSocket
+│   ├── attachment-upload/       # Download + sharp resize + upload
+│   ├── event-broadcaster/       # Redis pub/sub publisher
+│   ├── command-handler/         # Backend→gateway Redis commands
+│   ├── reaction-tracking/       # Reaction events
+│   ├── thread-tracking/         # Thread events
+│   ├── user-presence/           # Presence/status events
+│   ├── channel-topic/           # Channel topic events
+│   ├── guild-member-events/     # Member join/leave
+│   └── gateway-metrics/         # Prometheus /metrics (port 4016)
+└── tests/                       # Vitest suites
+```
+
+## Key invariants (DO NOT BREAK)
+
+1. **LLM is the only judge.** Failed LLM → `status:"error"` + recovery retry.
+   **Never** reintroduce regex/heuristic content classification.
+2. **Discord tokens sanitized** before reaching LLM (`discordTokens.ts`).
+3. **Semantic cache is batched** — one embed call + one Qdrant batch search.
+4. **Streaming is mandatory** against the omniroute base URL.
+
+## AI moderation pipeline
+
+```
+aiAnalyzer.ts → batchScheduler.ts → batchProcessor.ts → individualFallbackProcessor.ts
+    ↓                  ↓                    ↓                        ↓
+moderationOrchestrator.ts → (hash cache → Qdrant → LLM)
+    ↓                          ↓                              ↓
+textBatchProcessor.ts    mediaBatchProcessor.ts         llmClient.ts
+                                                     embeddingClient.ts
+                                                     qdrantClient.ts
+```
+
+- Entry: `aiAnalyzer.ts` (`queueMessageAnalysis`, `startPendingAIAnalysisWorker`)
+- Concurrency: LLM semaphore (`AI_LLM_MAX_CONCURRENT`, default 5)
+- Piscina: text pool (4 threads) + media pool (2 threads)
+- **Each worker thread has its own pg Pool** (min 0, grows to `POSTGRES_POOL_MAX`)
+
+## Voice recording pipeline
+
+```
+receiver.speaking "start" → speakingHandler(userId)
+  → collectUserMetadata → receiver.subscribe → PacketFilter → oggPacketStream
+  → SegmentManager.open → OggLogicalBitstream → .ogg file
+  → data: rotateIfNeeded + decoder.write
+  → end: finalizeSegment → upload + transcribe
+```
+
+Key files:
+- `voiceController.ts` — connect/disconnect/list
+- `recorder.ts` — orchestration
+- `recorder/segment.ts` — segment rotation
+- `recorder/sessionRecording.ts` — session management
+- `recorder/uploader.ts` — upload to storage
+- `voiceTranscriber.ts` — Whisper transcription (if enabled)
+
+## Module: message-capture
+
+- `messageCapture.ts` — Discord event listeners (messageCreate/Update/Delete)
+- `messageStore.ts` — DB operations
+- `messageMetadata.ts` — metadata extraction
+- `messagesDb.ts` / `messagesCrud.ts` — DB schema operations
+- `archiveEmbedder.ts` — Qdrant embedding (respect age-restricted guard)
+- `retentionDb.ts` / `reviewsDb.ts` / `attachmentsDb.ts` — auxiliary tables
+
+## Redis channels (outbound to backend)
+
+See `src/shared/redis-channels.ts` for canonical names. Examples:
+```
+discord:message:created, discord:voice:active_user, discord:attachment:uploaded
+```
+
+## Config (env vars)
+
+All validated via Zod in `shared/config/index.ts`. Critical:
+
+- `DISCORD_TOKEN` — selfbot token
+- `MONITOR_GUILD_ID` — primary guild
+- `DATABASE_URL` — PostgreSQL
+- `REDIS_URL` — pub/sub to backend
+- `AI_ANALYSIS_ENABLED` — master toggle for AI moderation
+- `AI_LLM_BASE_URL` / `AI_LLM_API_KEY` — LLM router
+- `AI_VOICE_TRANSCRIPTION_ENABLED` — toggle Whisper transcription
+- `PISCINA_MAX_THREADS` / `PISCINA_MEDIA_MAX_THREADS` — worker pool sizing
+
+## Concurrency model
+
+- Main thread: event loop + LLM semaphore
+- Text Piscina pool: `PISCINA_MAX_THREADS` (default 4)
+- Media Piscina pool: `PISCINA_MEDIA_MAX_THREADS` (default 2)
+- Batch routed to media pool if ANY message has attachment/sticker/embed
+- Each worker thread initializes own pg Pool (min 0, grows to `POSTGRES_POOL_MAX`)
+- MemoryMax: 1G (service systemd limit)
+
+## Testing
+
+- Vitest. Test files: `tests/<name>.test.ts`
+- Run: `pnpm test`
+- Key test areas: batch operations, cache guards, image/video handling, context enrichment
+- Unit tests for pure helpers (classifier, normalize, parse), integration for pipeline stages
+
+## Common pitfalls
+
+- **Piscina pool isolation**: worker threads are NOT the main thread. Cannot
+  share state via module-level variables. Use DB or Redis for cross-thread state.
+- **AfterSilence race**: `@discordjs/voice` AfterSilence can fail to emit "end"
+  on disconnect. Always have a watchdog/timeout.
+- **Cache eviction**: LRU caches (user metadata, term glossary) evict at max size.
+  Don't assume cache hit after eviction.
+- **Circuit breaker**: per-conversation CB opens after repeated failures.
+  Check `circuitBreaker.ts` state when debugging "missing analysis".
+- **Gateway≠Backend schema**: both have `redis-channels.ts` and `moderation-types.ts`.
+  Keep them in sync manually.
@@ -1,3 +1,5 @@
+@../../AGENTS.md
+
 # Bete Frontend — Project Overview

 Next.js 16 (App Router), React 19, TypeScript strict, Tailwind v4, shadcn/ui, base-ui.