refactor: split monolith into 3 microservices (frontend, backend, discord-gateway)
- Extract services into services/{frontend,backend,discord-gateway}
- Create packages/shared/ for shared logger, errors, utils, types
- Setup Modular MVC pattern in backend (controller→service→repository)
- Setup event-driven architecture in discord-gateway with Redis pub/sub
- Move Docker files to infra/docker/ with per-service Dockerfiles
- Update docker-compose.yml to use Traefik-only routing (no port exposes)
- Update GitHub Actions deploy workflow for multi-service matrix build
- Fix all import paths and resolve type errors across all services
- All 3 services pass tsc --noEmit clean
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MythEclipse
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
bda8304bb9
commit
c48a0c5e3b
@@ -0,0 +1,380 @@
|
||||
# Discord Moderation Watcher Bot - Microservices Architecture
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
- Docker & Docker Compose
|
||||
- Node.js 20+
|
||||
- pnpm 11+
|
||||
- Discord bot token
|
||||
- OpenAI API key
|
||||
|
||||
### Environment Setup
|
||||
|
||||
Create `.env.local` in the root directory:
|
||||
|
||||
```bash
|
||||
# Discord Configuration
|
||||
DISCORD_TOKEN=your_discord_token_here
|
||||
MONITOR_GUILD_ID=your_guild_id_here
|
||||
|
||||
# AI Configuration
|
||||
AI_LLM_API_KEY=your_openai_api_key_here
|
||||
|
||||
# Optional: Database URL (defaults to PostgreSQL in Docker)
|
||||
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/bete
|
||||
|
||||
# Optional: Redis URL (defaults to Redis in Docker)
|
||||
REDIS_URL=redis://localhost:6379
|
||||
```
|
||||
|
||||
### Local Development with Docker Compose
|
||||
|
||||
```bash
|
||||
# Start all services
|
||||
docker-compose up -d
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Stop all services
|
||||
docker-compose down
|
||||
|
||||
# Rebuild services
|
||||
docker-compose up -d --build
|
||||
```
|
||||
|
||||
**Services will be available at:**
|
||||
- Frontend: http://localhost:5173
|
||||
- Backend API: http://localhost:3001
|
||||
- Backend WebSocket: ws://localhost:3001
|
||||
- PostgreSQL: localhost:5432
|
||||
- Redis: localhost:6379
|
||||
|
||||
### Local Development without Docker
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pnpm install
|
||||
|
||||
# Run database migrations
|
||||
pnpm run db:migrate
|
||||
|
||||
# Start all services in separate terminals
|
||||
|
||||
# Terminal 1: Backend
|
||||
cd services/backend
|
||||
pnpm run dev
|
||||
|
||||
# Terminal 2: Discord Gateway
|
||||
cd services/discord-gateway
|
||||
pnpm run dev
|
||||
|
||||
# Terminal 3: Frontend
|
||||
cd services/frontend
|
||||
pnpm run dev:web
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### 3 Independent Microservices
|
||||
|
||||
#### 1. Frontend Service (`services/frontend/`)
|
||||
- **Tech:** React 19, Vite, TanStack Query, WebSocket
|
||||
- **Port:** 5173 (dev) / served by Backend (prod)
|
||||
- **Responsibilities:**
|
||||
- Dashboard UI (analytics, messages, voice, media)
|
||||
- Real-time WebSocket connection to Backend
|
||||
- API calls to Backend REST endpoints
|
||||
- State management (React Query)
|
||||
|
||||
#### 2. Backend Service (`services/backend/`)
|
||||
- **Tech:** Express, Drizzle ORM, PostgreSQL, Redis
|
||||
- **Port:** 3001
|
||||
- **Responsibilities:**
|
||||
- REST API endpoints (`/api/*`)
|
||||
- WebSocket server for real-time updates
|
||||
- Database operations (PostgreSQL)
|
||||
- Event orchestration from Discord Gateway
|
||||
- Static file serving (built Frontend)
|
||||
- Admin authentication
|
||||
|
||||
**Modular MVC Structure:**
|
||||
```
|
||||
services/backend/src/
|
||||
├── shared/
|
||||
│ ├── database/ → Drizzle ORM setup
|
||||
│ ├── config/ → Environment config
|
||||
│ ├── errors/ → Custom error classes
|
||||
│ ├── middlewares/ → Express middlewares
|
||||
│ ├── logger/ → Logging utilities
|
||||
│ └── utils/ → Shared utilities
|
||||
├── modules/
|
||||
│ ├── messages/ → Message CRUD
|
||||
│ ├── analytics/ → Analytics queries
|
||||
│ ├── media/ → Media management
|
||||
│ ├── voice/ → Voice recordings
|
||||
│ └── health/ → Health checks
|
||||
└── index.ts
|
||||
```
|
||||
|
||||
#### 3. Discord Gateway Service (`services/discord-gateway/`)
|
||||
- **Tech:** discord.js-selfbot-v13, @discordjs/voice, OpenAI API
|
||||
- **Port:** None (internal service, no HTTP)
|
||||
- **Responsibilities:**
|
||||
- Discord client connection
|
||||
- Message capture (create/edit/delete)
|
||||
- Voice channel recording
|
||||
- AI moderation analysis
|
||||
- Attachment upload
|
||||
- Event publishing to Backend (Redis pub/sub)
|
||||
|
||||
**Modular MVC Structure:**
|
||||
```
|
||||
services/discord-gateway/src/
|
||||
├── shared/
|
||||
│ ├── database/ → Drizzle ORM setup
|
||||
│ ├── config/ → Environment config
|
||||
│ ├── errors/ → Custom error classes
|
||||
│ ├── logger/ → Logging utilities
|
||||
│ └── utils/ → Shared utilities
|
||||
├── modules/
|
||||
│ ├── message-capture/ → Message listeners
|
||||
│ ├── voice-recording/ → Voice recording
|
||||
│ ├── ai-moderation/ → AI analysis
|
||||
│ ├── attachment-upload/ → File uploads
|
||||
│ └── event-broadcaster/ → Redis pub/sub
|
||||
└── index.ts
|
||||
```
|
||||
|
||||
### Shared Package (`packages/shared/`)
|
||||
- **Types:** Common interfaces and data models
|
||||
- **Errors:** Custom error classes
|
||||
- **Logger:** Pino logger setup
|
||||
- **Utils:** Pagination, validation, helpers
|
||||
|
||||
### Communication Patterns
|
||||
|
||||
**Frontend ↔ Backend:**
|
||||
- REST API: `GET/POST /api/*` (HTTP)
|
||||
- WebSocket: Real-time updates (JSON messages)
|
||||
- Auth: Admin password header
|
||||
|
||||
**Backend ↔ Discord Gateway:**
|
||||
- Redis pub/sub (low-latency, decoupled)
|
||||
- Events: `discord:message:created`, `discord:voice:started`, etc.
|
||||
- Backend subscribes and broadcasts to Frontend via WebSocket
|
||||
|
||||
**Shared Resources:**
|
||||
- PostgreSQL: Both Backend and Discord Gateway
|
||||
- Redis: Pub/sub and caching
|
||||
|
||||
---
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Adding a New API Endpoint
|
||||
|
||||
1. **Create module structure** (if new feature):
|
||||
```bash
|
||||
mkdir -p services/backend/src/modules/feature/{routes,controllers,services,repositories,schemas}
|
||||
```
|
||||
|
||||
2. **Define schema** (`feature.schema.ts`):
|
||||
```typescript
|
||||
import { z } from 'zod';
|
||||
|
||||
export const createFeatureSchema = z.object({
|
||||
name: z.string().min(1),
|
||||
description: z.string().optional(),
|
||||
});
|
||||
```
|
||||
|
||||
3. **Create repository** (`feature.repository.ts`):
|
||||
```typescript
|
||||
export async function createFeature(data: CreateFeatureInput) {
|
||||
return db.insert(features).values(data).returning();
|
||||
}
|
||||
```
|
||||
|
||||
4. **Create service** (`feature.service.ts`):
|
||||
```typescript
|
||||
export async function createFeatureService(data: CreateFeatureInput) {
|
||||
// Business logic, validation, orchestration
|
||||
return createFeature(data);
|
||||
}
|
||||
```
|
||||
|
||||
5. **Create controller** (`feature.controller.ts`):
|
||||
```typescript
|
||||
export async function createFeatureController(req: Request, res: Response) {
|
||||
const data = createFeatureSchema.parse(req.body);
|
||||
const result = await createFeatureService(data);
|
||||
res.json(result);
|
||||
}
|
||||
```
|
||||
|
||||
6. **Create route** (`feature.route.ts`):
|
||||
```typescript
|
||||
router.post('/features', createFeatureController);
|
||||
```
|
||||
|
||||
### Adding a New Discord Event
|
||||
|
||||
1. **Create module** in `services/discord-gateway/src/modules/event-name/`
|
||||
|
||||
2. **Register listener** in `index.ts`:
|
||||
```typescript
|
||||
client.on('eventName', async (data) => {
|
||||
await handleEvent(data);
|
||||
publishEvent('discord:event:name', data);
|
||||
});
|
||||
```
|
||||
|
||||
3. **Publish to Redis**:
|
||||
```typescript
|
||||
import { redis } from '../shared/redis';
|
||||
|
||||
redis.publish('discord:event:name', JSON.stringify(data));
|
||||
```
|
||||
|
||||
4. **Subscribe in Backend** (`services/backend/src/ws/server.ts`):
|
||||
```typescript
|
||||
redis.subscribe('discord:event:name', (message) => {
|
||||
broadcastToClients({ type: 'event_name', data: JSON.parse(message) });
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Run All Tests
|
||||
```bash
|
||||
pnpm run test
|
||||
```
|
||||
|
||||
### Run Tests for Specific Service
|
||||
```bash
|
||||
cd services/backend
|
||||
pnpm run test
|
||||
|
||||
cd services/discord-gateway
|
||||
pnpm run test
|
||||
```
|
||||
|
||||
### Type Checking
|
||||
```bash
|
||||
pnpm run typecheck
|
||||
```
|
||||
|
||||
### Linting
|
||||
```bash
|
||||
pnpm run lint
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
### Build Docker Images
|
||||
```bash
|
||||
docker-compose build
|
||||
```
|
||||
|
||||
### Push to Container Registry
|
||||
```bash
|
||||
docker tag bete-backend ghcr.io/username/bete-backend:latest
|
||||
docker push ghcr.io/username/bete-backend:latest
|
||||
```
|
||||
|
||||
### Deploy to Production
|
||||
See `.github/workflows/deploy.yml` for GitHub Actions CI/CD pipeline.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Backend can't connect to PostgreSQL
|
||||
```bash
|
||||
# Check PostgreSQL is running
|
||||
docker-compose ps postgres
|
||||
|
||||
# Check connection string
|
||||
echo $DATABASE_URL
|
||||
|
||||
# Verify credentials
|
||||
psql -h localhost -U postgres -d bete
|
||||
```
|
||||
|
||||
### Discord Gateway not receiving events
|
||||
```bash
|
||||
# Check Redis connection
|
||||
redis-cli ping
|
||||
|
||||
# Check Discord token
|
||||
echo $DISCORD_TOKEN
|
||||
|
||||
# View logs
|
||||
docker-compose logs discord-gateway
|
||||
```
|
||||
|
||||
### Frontend can't connect to Backend
|
||||
```bash
|
||||
# Check Backend is running
|
||||
curl http://localhost:3001/health
|
||||
|
||||
# Check WebSocket connection
|
||||
# Open browser DevTools → Network → WS
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Documentation
|
||||
|
||||
### Health Check
|
||||
```bash
|
||||
GET /health
|
||||
```
|
||||
|
||||
### Messages
|
||||
```bash
|
||||
GET /api/messages?channel=<id>&type=text|image
|
||||
POST /api/messages (admin only)
|
||||
```
|
||||
|
||||
### Analytics
|
||||
```bash
|
||||
GET /api/analytics
|
||||
```
|
||||
|
||||
### Voice Recordings
|
||||
```bash
|
||||
GET /api/recordings
|
||||
```
|
||||
|
||||
### WebSocket Events
|
||||
```
|
||||
message_created
|
||||
message_updated
|
||||
message_deleted
|
||||
attachment_uploaded
|
||||
user_state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Create a feature branch
|
||||
2. Make changes following Modular MVC pattern
|
||||
3. Run tests and linting
|
||||
4. Submit PR with description
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Reference in New Issue
Block a user