refactor(infra): repurpose asepharyana-hub into infra-only reverse-proxy repo
- Remove app submodules (hub/scraper/tools/llm-api/plugins) — apps are now standalone repos with their own flake.nix + deploy.yml CI - Delete monorepo CI: nix-build.yml matrix, update-submodule.yml, lint, security, flakehub-publish — replaced by infra-only caddy-deploy.yml - Sync Caddyfile.prod with live /etc/caddy/Caddyfile (add wiki. + mcp. blocks) - Prune legacy Docker/Traefik/Dapr/NATS/otel + scripts/root tooling - Docs: rename ADR 0001 superseded, add ADR 0003 (repo rename + split CI), update add-new-app, infra README, troubleshooting
This commit is contained in:
Asep Haryana
@@ -1,238 +0,0 @@
|
||||
# Arsitektur asepharyana-hub
|
||||
|
||||
## Topologi Fisik
|
||||
|
||||
Dua node terhubung via **Tailscale** overlay network:
|
||||
|
||||
```
|
||||
┌──────────────────────────────┐ ┌──────────────────────────────┐
|
||||
│ orangevps (VPS) │ │ imrnes (Bare-metal) │
|
||||
│ IP: 45.127.35.244 │ │ Tailscale: 100.121.180.82 │
|
||||
│ Tailscale: 100.x.x.x │◄──────┤ │
|
||||
│ │ │ Layanan: │
|
||||
│ Layanan: │ │ ├─ PostgreSQL (port 6432) │
|
||||
│ ├─ Caddy (port 80/443) │ │ └─ Redis (port 6379) │
|
||||
│ ├─ NATS + JetStream │ │ │
|
||||
│ ├─ Dapr Placement │ └──────────────────────────────┘
|
||||
│ ├─ Redis (cache, Dapr) │
|
||||
│ ├─ Scraper API + Dapr │
|
||||
│ └─ Hub (Next.js SPA) │
|
||||
└──────────────────────────────┘
|
||||
```
|
||||
|
||||
### Konektivitas Container ke Tailscale
|
||||
|
||||
Container di `orangevps` tidak bisa langsung mencapai IP Tailscale (`100.x.x.x`). Route Tailscale harus ditambahkan ke tabel routing utama (`main`) via `tailscale-routes.service` agar traffic dari container bisa melewati host ke Tailscale.
|
||||
|
||||
## Alur Request HTTP (External)
|
||||
|
||||
```
|
||||
Internet
|
||||
│
|
||||
▼ Port 443
|
||||
Caddy 2.11.4 (auto-TLS LE, HTTP/3)
|
||||
├─ TLS termination (sertifikat dari volume mount)
|
||||
├─ Middleware chain: secure-headers → compress → retry → rate-limit → buffer
|
||||
├─ Plugin: real-ip (Cloudflare), block-sensitive-paths
|
||||
│
|
||||
▼ Router matching
|
||||
Host(`asepharyana.my.id`) || Host(`www.asepharyana.my.id`) → hub
|
||||
host(`hub.asepharyana.my.id`) → hub (SPA + dashboard)
|
||||
Host(`scraper.asepharyana.my.id`) || Host(`api.asepharyana.my.id`) → scraper-api
|
||||
│
|
||||
├─ hub (Next.js, port 4003)
|
||||
│ ├─ / — Portfolio SPA
|
||||
│ ├─ /dashboard — Ops dashboard (client-side, auto-refresh 15s)
|
||||
│ ├─ /api/dashboard — JSON: systemd services, Jaeger traces, Prometheus metrics
|
||||
│ └─ Metrics via node-exporter + app endpoints
|
||||
│
|
||||
▼ Service load balancer
|
||||
http://scraper-api:4091
|
||||
│
|
||||
▼
|
||||
Scraper API (Rust / Axum)
|
||||
├─ Health check: GET /, respon 200
|
||||
├─ REST endpoints
|
||||
├─ Database via `DATABASE_URL` (Tailscale → PostgreSQL di imrnes)
|
||||
├─ Cache via `REDIS_URL` (Redis lokal di container)
|
||||
└─ Pub/sub via Dapr sidecar (localhost:3500)
|
||||
```
|
||||
|
||||
## Infrastruktur Internal
|
||||
|
||||
### Docker Compose Project
|
||||
|
||||
Semua service berjalan dalam satu Docker Compose project bernama `compose` dan bergabung di network `app-shared-net`:
|
||||
|
||||
| File | Service | Peran |
|
||||
|------|---------|-------|
|
||||
| `traefik.yml` | `traefik` | Reverse proxy + TLS + metrics Prometheus |
|
||||
| `shared.yml` | `redis` | Cache, session store, backend Dapr pub/sub & state |
|
||||
| `nats.yml` | `nats` | Message broker + JetStream persistent streaming |
|
||||
| `dapr.yml` | `dapr-placement` | Koordinasi actor placement untuk sidecar Dapr |
|
||||
| `scraper.yml` | `scraper-api` + `scraper-api-dapr` | Aplikasi Rust + sidecar Dapr |
|
||||
| systemd hub | `hub` | Next.js SPA portfolio + dashboard |
|
||||
| `observability.yml` | `otel-collector`, `jaeger`, `prometheus`, `node-exporter` | Tracing, metrics, observability |
|
||||
|
||||
### Dapr Sidecar Pattern
|
||||
|
||||
Setiap aplikasi yang menggunakan Dapr mendapat sidecar container `daprd`:
|
||||
|
||||
```
|
||||
┌─────────────────────┐
|
||||
│ scraper-api │
|
||||
│ (app port 4091) │
|
||||
└────────┬────────────┘
|
||||
│ localhost:3500 (HTTP)
|
||||
│ localhost:50001 (gRPC)
|
||||
┌────────▼────────────┐
|
||||
│ scraper-api-dapr │
|
||||
│ (daprd sidecar) │
|
||||
│ │
|
||||
│ Dapr components: │
|
||||
│ ├─ pubsub.redis │
|
||||
│ └─ state.redis │
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
Komponen Dapr:
|
||||
|
||||
| Komponen | Tipe | Backend |
|
||||
|----------|------|---------|
|
||||
| `pubsub` | `pubsub.redis` | `redis:6379` |
|
||||
| `statestore` | `state.redis` | `redis:6379` (prefix `dapr`) |
|
||||
|
||||
### Monitoring & Auto-Discovery
|
||||
|
||||
#### Prometheus Docker Auto-Discovery
|
||||
|
||||
Prometheus menggunakan `docker_sd_configs` untuk auto-detect container yang perlu di-scrape. Cukup tambah label pada container:
|
||||
|
||||
```yaml
|
||||
labels:
|
||||
- 'prometheus.io/scrape=true'
|
||||
- 'prometheus.io/port=8080' # port metrics endpoint
|
||||
- 'prometheus.io/path=/metrics' # optional, default /metrics
|
||||
```
|
||||
|
||||
Prometheus akan auto-detect dan mulai scrape container dalam 15 detik.
|
||||
|
||||
#### Traefik Metrics
|
||||
|
||||
Traefik mengekspos metrics Prometheus di port 8080 (`--metrics.prometheus=true`). Metrics yang tersedia:
|
||||
|
||||
| Metric | Query untuk dashboard |
|
||||
|--------|----------------------|
|
||||
| Request rate | `sum(rate(traefik_service_requests_total[1m]))` |
|
||||
| Latency | `avg(traefik_service_request_duration_seconds_sum / traefik_service_request_duration_seconds_count) * 1000` |
|
||||
| Error rate | `sum(rate(traefik_service_requests_total{code=~"5.."}[1m]))` |
|
||||
|
||||
Dashboard di `/api/dashboard` returns node metrics + Traefik range data untuk 4 sparkline charts (RPS, latency, errors, trace volume).
|
||||
|
||||
#### Docker Socket Access
|
||||
|
||||
Container yang perlu akses Docker socket (`/var/run/docker.sock`) harus punya group docker (GID 988):
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
group_add:
|
||||
- '988'
|
||||
```di compose atau `--group-add 988` via CLI. Berlaku untuk `hub` (container list) dan `prometheus` (Docker SD).
|
||||
|
||||
### NATS + JetStream
|
||||
|
||||
NATS berjalan dengan flag `-js` untuk mengaktifkan JetStream. Persistent stream disimpan di volume `nats_data`. Dapr pub/sub routing:
|
||||
|
||||
```
|
||||
Service → Dapr sidecar (pubsub.redis) → Redis streams
|
||||
```
|
||||
|
||||
> **Catatan:** Saat ini Dapr pub/sub menggunakan Redis, bukan NATS. Jika ingin migrasi ke NATS untuk pub/sub, komponen Dapr perlu diganti dengan `pubsub.nats`.
|
||||
|
||||
## Arsitektur CI/CD
|
||||
|
||||
```
|
||||
Push ke main (apps/**, infra/**)
|
||||
│
|
||||
▼
|
||||
docker-build-push.yml
|
||||
├─ Phase 1: Detect changed services
|
||||
├─ Phase 2: Build & Push image ke GHCR
|
||||
└─ Phase 3: Update compose manifest + submodule pointer
|
||||
│
|
||||
▼ (workflow_run trigger)
|
||||
deploy-docker.yml
|
||||
├─ SSH ke orangevps
|
||||
├─ Git sync, pull images
|
||||
├─ Remove stale containers
|
||||
└─ Selective restart service
|
||||
```
|
||||
|
||||
Submodule update dari remote repo via `repository_dispatch`:
|
||||
|
||||
```
|
||||
Push ke asepharyana-hub-scraper
|
||||
│
|
||||
▼ (repository_dispatch)
|
||||
update-submodule.yml
|
||||
├─ Update submodule pointer
|
||||
└─ Commit & push ke hub repo
|
||||
│
|
||||
▼ (repository_dispatch trigger)
|
||||
docker-build-push.yml
|
||||
└─ Build, push, deploy
|
||||
```
|
||||
|
||||
## Image Tagging Strategy
|
||||
|
||||
| Tag | Contoh | Penggunaan |
|
||||
|-----|--------|------------|
|
||||
| `sha-<short>` | `sha-a3c5d74` | Immutable, deterministic rollback |
|
||||
| `latest` | `latest` | Mutable, convenience |
|
||||
| `buildcache` | `sha-a3c5d74-buildcache` | Registry-based build cache (internal) |
|
||||
|
||||
## Networking
|
||||
|
||||
### Port Map
|
||||
|
||||
| Port | Service | Deskripsi |
|
||||
|------|---------|-----------|
|
||||
| 443 | Traefik | HTTPS eksternal |
|
||||
| 80 | Traefik | Redirect ke HTTPS |
|
||||
| 4222 | NATS | Client connections |
|
||||
| 8222 | NATS | HTTP monitor / health |
|
||||
| 6379 | Redis | Internal container network |
|
||||
| 3500 | Dapr sidecar | Dapr HTTP API (per service) |
|
||||
| 50001 | Dapr sidecar | Dapr gRPC API (per service) |
|
||||
| 50005 | Dapr placement | Actor placement |
|
||||
| 4091 | Scraper API | Aplikasi HTTP |
|
||||
|
||||
## Event Topics Convention
|
||||
|
||||
Semua event menggunakan prefix `hub.`:
|
||||
|
||||
| Topic | Payload | Deskripsi |
|
||||
|-------|---------|-----------|
|
||||
| `hub.image.cached` | `{original_url, cdn_url, source}` | Image selesai di-cache |
|
||||
| `hub.image.repaired` | `{old_url, new_url}` | CNAME image diperbaiki |
|
||||
| `hub.scrape.anime.done` | `{source, slug, duration}` | Scrape anime selesai |
|
||||
| `hub.system.alert` | `{service, level, message}` | Error/alert dari service |
|
||||
|
||||
## Service Registry (Traefik)
|
||||
|
||||
Domain routing:
|
||||
|
||||
| Subdomain | Service | URL Backend |
|
||||
|-----------|---------|-------------|
|
||||
| `asepharyana.my.id` (root) | Hub SPA + dashboard | `http://hub:3000` |
|
||||
| `www.*` | Hub (alias) | `http://hub:3000` |
|
||||
| `hub.*` | Hub (alias) | `http://hub:3000` |
|
||||
| `scraper.*` | Scraper API | `http://scraper-api:4091` |
|
||||
| `api.*` | Scraper API (alias) | `http://scraper-api:4091` |
|
||||
| `traefik.*` | Traefik Dashboard | `api@internal` |
|
||||
| `jaeger.*` | Jaeger UI | `http://jaeger:16686` |
|
||||
|
||||
Semua domain tersedia di:
|
||||
- `<service>.asepharyana.my.id`
|
||||
- `<service>.asepharyana.web.id`
|
||||
@@ -1,681 +0,0 @@
|
||||
# Deployment Guide
|
||||
|
||||
Panduan deploy aplikasi apapun menggunakan **Docker + Docker Compose + GitHub Actions + VPS**.
|
||||
|
||||
## Arsitektur
|
||||
|
||||
```
|
||||
GitHub Repo ──► GitHub Actions ──► Registry (GHCR / Docker Hub / ECR / dll.)
|
||||
│
|
||||
▼
|
||||
VPS (<VPS_HOST>)
|
||||
docker compose pull + up
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker Engine >= 24.x
|
||||
- Docker Compose v2 (plugin)
|
||||
- Git
|
||||
- Akun GitHub dengan akses repo
|
||||
- SSH key di `~/.ssh/<KEY_NAME>` (default: `id_ed25519`)
|
||||
|
||||
## Konfigurasi VPS Target
|
||||
|
||||
Buat berkas `~/orangevps` (atau sesuaikan dengan env Anda):
|
||||
|
||||
```text
|
||||
ssh <USER>@<VPS_HOST>
|
||||
```
|
||||
|
||||
Contoh isi `~/orangevps`:
|
||||
|
||||
```text
|
||||
ssh root@45.127.35.244
|
||||
```
|
||||
|
||||
| Parameter | Nilai | Contoh |
|
||||
|-----------|-------|--------|
|
||||
| User | `<USER>` | `root` |
|
||||
| Host | `<VPS_HOST>` | `45.127.35.244` |
|
||||
| SSH Key | `~/.ssh/<KEY_NAME>` | `~/.ssh/id_ed25519` |
|
||||
| Target Dir di VPS | `<VPS_TARGET_DIR>` | `/opt/app` atau `/root/app` |
|
||||
|
||||
> Tip: Jika SSH key menggunakan nama selain default, sesuaikan path dan `ssh -i` sesuai.
|
||||
|
||||
## Registry
|
||||
|
||||
Pilih registry untuk menyimpan image Docker. Sesuaikan dengan proyek:
|
||||
|
||||
| Registry | URL | Auth |
|
||||
|----------|-----|------|
|
||||
| GitHub Container Registry | `ghcr.io` | `GITHUB_TOKEN` |
|
||||
| Docker Hub | `docker.io` | username / PAT |
|
||||
| AWS ECR | `<account>.dkr.ecr.<region>.amazonaws.com` | `aws ecr get-login-password` |
|
||||
| Google GCR | `gcr.io` | `gcloud auth print-access-token` |
|
||||
| Azure ACR | `<registry>.azurecr.io` | `az acr login` |
|
||||
|
||||
Contoh namespace untuk GHCR:
|
||||
|
||||
```text
|
||||
Registry : ghcr.io
|
||||
Namespace: <GITHUB_USERNAME_OR_ORG>
|
||||
Repo : <REPO_NAME>
|
||||
```
|
||||
|
||||
Pastikan package/visibility di registry mengizinkan akses pull dari VPS.
|
||||
|
||||
---
|
||||
|
||||
## Deploy Otomatis (Recommended)
|
||||
|
||||
Gunakan GitHub Actions untuk otomatisasi build, push, dan deploy.
|
||||
|
||||
### Workflow 1: Build dan Push Image
|
||||
|
||||
File: `.github/workflows/docker-build-push.yml`
|
||||
|
||||
```yaml
|
||||
name: Build and Push Docker Images
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- uses: docker/login-action@v4
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- uses: docker/setup-buildx-action@v4
|
||||
|
||||
- uses: docker/build-push-action@v7
|
||||
with:
|
||||
context: .
|
||||
file: Dockerfile
|
||||
push: true
|
||||
tags: |
|
||||
ghcr.io/${{ github.repository }}/<SERVICE_NAME>:latest
|
||||
ghcr.io/${{ github.repository }}/<SERVICE_NAME>:sha-${{ github.sha }}
|
||||
cache-from: type=registry,ref=ghcr.io/${{ github.repository }}/<SERVICE_NAME>:buildcache
|
||||
cache-to: type=registry,ref=ghcr.io/${{ github.repository }}/<SERVICE_NAME>:buildcache,mode=max
|
||||
```
|
||||
|
||||
Ubah `<SERVICE_NAME>` sesuai service (misal: `app`, `web`, `api`). Jika monorepo, gunakan matrix strategy untuk build beberapa service sekaligus.
|
||||
|
||||
### Workflow 2: Deploy ke VPS
|
||||
|
||||
File: `.github/workflows/deploy-docker.yml`
|
||||
|
||||
```yaml
|
||||
name: Deploy Docker to VPS
|
||||
|
||||
on:
|
||||
workflow_run:
|
||||
workflows: ['Build and Push Docker Images']
|
||||
types: [completed]
|
||||
push:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Deploy to VPS
|
||||
env:
|
||||
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
|
||||
VPS_HOST: ${{ secrets.VPS_HOST }}
|
||||
VPS_USER: ${{ secrets.VPS_USER }}
|
||||
VPS_TARGET_DIR: ${{ secrets.VPS_TARGET_DIR }}
|
||||
ENV_FILE_PRODUCTION: ${{ secrets.ENV_FILE_PRODUCTION }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
mkdir -p ~/.ssh
|
||||
echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
|
||||
chmod 600 ~/.ssh/id_rsa
|
||||
ssh-keyscan -H -t ed25519,rsa "$VPS_HOST" >> ~/.ssh/known_hosts
|
||||
|
||||
SSH_OPTS=(-o ControlMaster=auto -o ControlPath=/tmp/ssh-%r@%h:%p -o ControlPersist=600 -o StrictHostKeyChecking=yes)
|
||||
|
||||
ssh "${SSH_OPTS[@]}" "$VPS_USER@$VPS_HOST" "mkdir -p $VPS_TARGET_DIR && mkdir -p $VPS_TARGET_DIR/infra/compose"
|
||||
echo "$ENV_FILE_PRODUCTION" > .env.prod
|
||||
scp "${SSH_OPTS[@]}" .env.prod "$VPS_USER@$VPS_HOST:$VPS_TARGET_DIR/.env"
|
||||
|
||||
ssh "${SSH_OPTS[@]}" "$VPS_USER@$VPS_HOST" bash -s <<'EOF'
|
||||
set -euo pipefail
|
||||
cd "$VPS_TARGET_DIR"
|
||||
|
||||
docker network inspect app-shared-net >/dev/null 2>&1 || docker network create app-shared-net
|
||||
|
||||
if [ ! -d ".git" ]; then
|
||||
git init
|
||||
git remote add origin https://github.com/<GITHUB_USER>/<REPO_NAME>.git
|
||||
fi
|
||||
git fetch origin main --depth=1 || true
|
||||
git reset --hard FETCH_HEAD
|
||||
|
||||
docker compose --env-file .env pull
|
||||
docker compose --env-file .env up -d --remove-orphans
|
||||
EOF
|
||||
```
|
||||
|
||||
### Secrets GitHub yang Diperlukan
|
||||
|
||||
Buka **Settings > Secrets and variables > Actions**:
|
||||
|
||||
| Secret | Deskripsi |
|
||||
|--------|-----------|
|
||||
| `SSH_PRIVATE_KEY` | Isi dengan `cat ~/.ssh/<KEY_NAME>` |
|
||||
| `VPS_HOST` | IP atau domain VPS |
|
||||
| `VPS_USER` | User SSH (misal: `root`, `ubuntu`, `deploy`) |
|
||||
| `VPS_TARGET_DIR` | Direktori aplikasi di VPS |
|
||||
| `ENV_FILE_PRODUCTION` | Isi dengan environment production |
|
||||
|
||||
### Trigger Manual
|
||||
|
||||
```bash
|
||||
gh workflow run deploy-docker.yml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dockerfile Patterns
|
||||
|
||||
Pilih pattern sesuai jenis aplikasi.
|
||||
|
||||
### Pattern 1: Multi-stage Build (SPA / static assets)
|
||||
|
||||
```dockerfile
|
||||
FROM oven/bun:1 AS builder
|
||||
WORKDIR /app
|
||||
COPY package.json bun.lock ./
|
||||
RUN bun install --frozen-lockfile
|
||||
COPY . .
|
||||
RUN bun run build
|
||||
|
||||
FROM nginx:alpine
|
||||
COPY --from=builder /app/dist /usr/share/nginx/html
|
||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
||||
EXPOSE 80
|
||||
CMD ["nginx", "-g", "daemon off;"]
|
||||
```
|
||||
|
||||
### Pattern 2: Single-stage (runtime image)
|
||||
|
||||
```dockerfile
|
||||
FROM oven/bun:1
|
||||
WORKDIR /app
|
||||
COPY package.json bun.lock ./
|
||||
RUN bun install --frozen-lockfile
|
||||
COPY . .
|
||||
EXPOSE 3000
|
||||
CMD ["bun", "run", "start"]
|
||||
```
|
||||
|
||||
### Pattern 3: Compiled binary (Rust / Go / Zig)
|
||||
|
||||
```dockerfile
|
||||
FROM rust:1 AS builder
|
||||
WORKDIR /app
|
||||
COPY . .
|
||||
RUN cargo build --release
|
||||
|
||||
FROM debian:bookworm-slim
|
||||
COPY --from=builder /app/target/release/app /usr/local/bin/app
|
||||
EXPOSE 8080
|
||||
CMD ["app"]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker Compose Patterns
|
||||
|
||||
### Single service
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
container_name: app
|
||||
image: registry.example.com/org/app:latest
|
||||
restart: always
|
||||
ports:
|
||||
- "3000:3000"
|
||||
environment:
|
||||
- NODE_ENV=production
|
||||
```
|
||||
|
||||
### Multi-service dengan shared network
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
container_name: app
|
||||
image: registry.example.com/org/app:latest
|
||||
restart: always
|
||||
networks: [app-shared-net]
|
||||
|
||||
redis:
|
||||
container_name: redis
|
||||
image: redis:7-alpine
|
||||
restart: always
|
||||
networks: [app-shared-net]
|
||||
|
||||
networks:
|
||||
app-shared-net:
|
||||
name: app-shared-net
|
||||
external: true
|
||||
```
|
||||
|
||||
### Dengan reverse proxy (Traefik / Caddy / Nginx)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
container_name: app
|
||||
image: registry.example.com/org/app:latest
|
||||
restart: always
|
||||
networks: [app-shared-net]
|
||||
labels:
|
||||
- 'traefik.enable=true'
|
||||
- 'traefik.http.routers.app.rule=Host(`app.example.com`)'
|
||||
- 'traefik.http.routers.app.entrypoints=websecure'
|
||||
- 'traefik.http.routers.app.tls=true'
|
||||
- 'traefik.http.services.app.loadbalancer.server.port=3000'
|
||||
|
||||
networks:
|
||||
app-shared-net:
|
||||
name: app-shared-net
|
||||
external: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deploy Manual (Lokal)
|
||||
|
||||
### 1. Build dan Push ke Registry
|
||||
|
||||
Login ke registry:
|
||||
|
||||
```bash
|
||||
echo $GITHUB_TOKEN | docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
|
||||
```
|
||||
|
||||
Build dan push:
|
||||
|
||||
```bash
|
||||
docker build -t ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:latest -f Dockerfile .
|
||||
|
||||
docker push ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:latest
|
||||
```
|
||||
|
||||
Tag tambahan dengan SHA commit:
|
||||
|
||||
```bash
|
||||
SHORT_SHA=$(git rev-parse --short HEAD)
|
||||
docker tag ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:latest \
|
||||
ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:sha-${SHORT_SHA}
|
||||
docker push ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:sha-${SHORT_SHA}
|
||||
```
|
||||
|
||||
### 2. Pull dan Deploy di VPS
|
||||
|
||||
SSH ke VPS:
|
||||
|
||||
```bash
|
||||
ssh -i ~/.ssh/<KEY_NAME> <USER>@<VPS_HOST>
|
||||
```
|
||||
|
||||
Clone repo (jika belum):
|
||||
|
||||
```bash
|
||||
git clone https://github.com/<GITHUB_USER>/<REPO_NAME>.git <VPS_TARGET_DIR>
|
||||
cd <VPS_TARGET_DIR>
|
||||
```
|
||||
|
||||
Buat shared network (hanya sekali):
|
||||
|
||||
```bash
|
||||
docker network create app-shared-net
|
||||
```
|
||||
|
||||
Siapkan environment:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env sesuai nilai production
|
||||
nano .env
|
||||
```
|
||||
|
||||
Login ke registry di VPS:
|
||||
|
||||
```bash
|
||||
echo $GITHUB_TOKEN | docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
|
||||
```
|
||||
|
||||
Pull gambar terbaru:
|
||||
|
||||
```bash
|
||||
cd <VPS_TARGET_DIR>
|
||||
docker compose -f docker-compose.yml --env-file .env pull
|
||||
```
|
||||
|
||||
Deploy (up):
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml --env-file .env up -d --remove-orphans
|
||||
```
|
||||
|
||||
Verifikasi:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml ps
|
||||
docker compose -f docker-compose.yml logs -f <SERVICE_NAME>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deployment Order (Manual)
|
||||
|
||||
Jika deploy bertahap, gunakan urutan ini:
|
||||
|
||||
```bash
|
||||
# 1. Shared services (Redis, database, dll.)
|
||||
docker compose -f infra/compose/shared.yml up -d
|
||||
|
||||
# 2. Reverse proxy
|
||||
docker compose -f infra/compose/traefik.yml up -d
|
||||
|
||||
# 3. Aplikasi
|
||||
docker compose \
|
||||
-f infra/compose/app1.yml \
|
||||
-f infra/compose/app2.yml \
|
||||
up -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Perintah Berguna di VPS
|
||||
|
||||
```bash
|
||||
# Lihat semua container
|
||||
docker ps -a
|
||||
|
||||
# Log service
|
||||
docker logs -f <container_name>
|
||||
|
||||
# Restart satu service
|
||||
docker compose -f <compose_file> up -d --force-recreate
|
||||
|
||||
# Hapus network lama (hati-hati)
|
||||
docker network rm app-shared-net
|
||||
docker network create app-shared-net
|
||||
|
||||
# Bersihkan image unused
|
||||
docker image prune -a -f
|
||||
docker system prune -a -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Image tidak bisa di-pull
|
||||
|
||||
Pastikan sudah login ke registry di VPS:
|
||||
|
||||
```bash
|
||||
docker logout ghcr.io
|
||||
echo $GITHUB_TOKEN | docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
|
||||
```
|
||||
|
||||
Periksa visibility package di registry (harus `Public` atau akses diberikan).
|
||||
|
||||
### Port sudah dipakai
|
||||
|
||||
```bash
|
||||
docker ps | grep :80
|
||||
docker ps | grep :443
|
||||
```
|
||||
|
||||
### Reverse proxy tidak routing
|
||||
|
||||
Periksa label di compose file dan pastikan shared network ada:
|
||||
|
||||
```bash
|
||||
docker network inspect app-shared-net
|
||||
docker logs traefik
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment Variable Management
|
||||
|
||||
### Pola 1: `.env` di VPS (recommended untuk production)
|
||||
|
||||
```bash
|
||||
# Di VPS
|
||||
cd <VPS_TARGET_DIR>
|
||||
cp .env.example .env
|
||||
# Edit sesuai production
|
||||
nano .env
|
||||
```
|
||||
|
||||
CI/CD upload `.env` via secret, tidak simpan di repo.
|
||||
|
||||
### Pola 2: Docker secrets (Swarm mode)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
image: app:latest
|
||||
secrets:
|
||||
- db_password
|
||||
|
||||
secrets:
|
||||
db_password:
|
||||
file: ./secrets/db_password.txt
|
||||
```
|
||||
|
||||
### Pola 3: External secret manager
|
||||
|
||||
- **HashiCorp Vault**: inject via env atau file
|
||||
- **AWS Secrets Manager**: `aws secretsmanager get-secret-value`
|
||||
- **Doppler / Infisical**: unified secret management
|
||||
|
||||
---
|
||||
|
||||
## Tagging dan Versioning
|
||||
|
||||
### Strategy yang umum
|
||||
|
||||
| Strategy | Contoh tag | Kegunaan |
|
||||
|----------|-----------|----------|
|
||||
| Latest + SHA | `latest`, `sha-abc1234` | CI/CD cepat, traceable |
|
||||
| SemVer | `1.2.3`, `1.2`, `1` | Release publik |
|
||||
| Git tag mirror | `v1.2.3` | Sync dengan git tag |
|
||||
| Branch mirror | `main`, `develop` | Preview / staging |
|
||||
|
||||
### Contoh git tag driven deploy
|
||||
|
||||
```bash
|
||||
git tag v1.2.3
|
||||
git push origin v1.2.3
|
||||
```
|
||||
|
||||
CI/CD membaca tag, build image dengan tag yang sama, dan deploy.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
### Rollback via registry
|
||||
|
||||
```bash
|
||||
# Lihat tag yang tersedia
|
||||
docker manifest inspect ghcr.io/org/app:latest
|
||||
# atau lihat UI registry
|
||||
|
||||
# Di VPS, edit compose file ke tag sebelumnya
|
||||
# lalu:
|
||||
docker compose --env-file .env pull
|
||||
docker compose --env-file .env up -d --remove-orphans
|
||||
```
|
||||
|
||||
### Rollback via git
|
||||
|
||||
```bash
|
||||
git revert HEAD
|
||||
git push origin main
|
||||
# CI/CD otomatis build dan deploy versi sebelumnya
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Di Dockerfile
|
||||
|
||||
```dockerfile
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD curl -f http://localhost:3000/health || exit 1
|
||||
```
|
||||
|
||||
### Di Docker Compose
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
image: app:latest
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
```bash
|
||||
# Log aggregated
|
||||
docker compose logs -f --tail=100
|
||||
|
||||
# Resource usage
|
||||
docker stats
|
||||
|
||||
# Disk usage
|
||||
docker system df
|
||||
|
||||
# Cleanup
|
||||
docker system prune -a -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Catatan Keamanan
|
||||
|
||||
- Jangan commit `.env` atau SSH private key ke repo.
|
||||
- Gunakan GitHub Secrets (atau secret manager) untuk credential di CI/CD.
|
||||
- Rotate token dan key secara berkala.
|
||||
- Batasi akses SSH ke VPS (ubah port default, gunakan fail2ban).
|
||||
- Set `StrictHostKeyChecking=yes` pada SSH opsional deployment.
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## Proyek Ini: asepharyana-hub
|
||||
|
||||
> Dokumentasi spesifik untuk repo ini. Lihat juga [ADR-0002](adr/0002-env-file-via-github-secret.md).
|
||||
|
||||
### Topologi
|
||||
|
||||
| Host | IP | Peran |
|
||||
|------|----|-------|
|
||||
| `orangevps` (VPS) | `45.127.35.244` | Docker host: Traefik, scraper-api, Redis, NATS, Dapr |
|
||||
| `imrnes` (bare-metal) | `100.121.180.82` (Tailscale) | PostgreSQL (port 6432), Redis (port 6379) |
|
||||
|
||||
### Environment Variables
|
||||
|
||||
**Production `.env` tidak pernah di-commit.** File ini disimpan sebagai GitHub secret `ENV_FILE_PRODUCTION` dan di-SCP ke VPS saat deploy via `deploy-docker.yml`.
|
||||
|
||||
Cara update:
|
||||
```bash
|
||||
# Baca current .env dari VPS
|
||||
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env"
|
||||
|
||||
# Update GitHub secret (dari output di atas)
|
||||
cat > /tmp/env-updated << 'EOF'
|
||||
<paste content, edit, lalu>
|
||||
EOF
|
||||
cat /tmp/env-updated | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub
|
||||
```
|
||||
|
||||
**Jangan manual edit `.env` di VPS tanpa update GitHub secret juga** — nanti ke- overwrite pas deploy berikutnya.
|
||||
|
||||
### Database
|
||||
|
||||
| Variable | Value |
|
||||
|----------|-------|
|
||||
| `DATABASE_URL` | `postgres://asephs:hunterz@100.121.180.82:6432/hub` |
|
||||
| `REDIS_URL` | `redis://redis:6379` (Docker network) |
|
||||
|
||||
### Kompose
|
||||
|
||||
Proyek compose bernama `compose`, terdiri dari 5 file yang selalu di-include bersamaan:
|
||||
|
||||
```bash
|
||||
/root/asepharyana-hub/infra/compose/
|
||||
├── traefik.yml # Reverse proxy
|
||||
├── shared.yml # Redis
|
||||
├── nats.yml # NATS
|
||||
├── dapr.yml # Dapr placement
|
||||
├── scraper.yml # Scraper API
|
||||
└── observability.yml # OTel Collector, Jaeger, Dashboard
|
||||
|
||||
```bash
|
||||
cd /root/asepharyana-hub
|
||||
docker compose \
|
||||
-p compose \
|
||||
--env-file .env \
|
||||
-f infra/compose/traefik.yml \
|
||||
-f infra/compose/shared.yml \
|
||||
-f infra/compose/scraper.yml \
|
||||
-f infra/compose/nats.yml \
|
||||
-f infra/compose/dapr.yml \
|
||||
-f infra/compose/observability.yml \
|
||||
up -d --remove-orphans
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checklist Deploy Proyek Baru
|
||||
|
||||
1. [ ] Dockerfile ditest lokal (`docker build`, `docker run`)
|
||||
2. [ ] Docker Compose file valid (`docker compose config`)
|
||||
3. [ ] `.dockerignore` sesuai (node_modules, .git, .env)
|
||||
4. [ ] Registry dibuat (GHCR package / Docker Hub repo / ECR / dll.)
|
||||
5. [ ] GitHub Actions workflow dibuat dengan permission `packages: write`
|
||||
6. [ ] VPS siap: Docker, Docker Compose, SSH key
|
||||
7. [ ] Shared network dibuat (`docker network create`)
|
||||
8. [ ] `.env` production di-VPS atau via secret manager
|
||||
9. [ ] Reverse proxy (Traefik / Caddy / Nginx) routing ke container
|
||||
10. [ ] Health check endpoint aktif
|
||||
@@ -1,206 +0,0 @@
|
||||
# Development Guide
|
||||
|
||||
Panduan setup lingkungan development lokal untuk kontributor `asepharyana-hub`.
|
||||
|
||||
## Prasyarat
|
||||
|
||||
| Tool | Versi Minimal | Catatan |
|
||||
|------|---------------|---------|
|
||||
| Git | 2.40+ | Submodule support |
|
||||
| Docker | 24+ | Dengan Docker Compose v2 plugin |
|
||||
| Rust | 1.85+ | Hanya untuk `apps/scraper` |
|
||||
| Bun | 1.x | Root tooling (Biome) |
|
||||
| Dapr CLI | 1.14+ | Opsional, untuk development dengan Dapr |
|
||||
|
||||
## Setup Awal
|
||||
|
||||
```bash
|
||||
# 1. Clone repo
|
||||
git clone https://github.com/asepharyana/asepharyana-hub.git
|
||||
cd asepharyana-hub
|
||||
|
||||
# 2. Init submodules
|
||||
make init-submodules
|
||||
|
||||
# 3. Setup environment
|
||||
cp .env.example .env
|
||||
# Edit .env sesuai kebutuhan lokal
|
||||
|
||||
# 4. Install root dependencies
|
||||
bun install
|
||||
```
|
||||
|
||||
## Menjalankan Infrastruktur Lokal
|
||||
|
||||
Beberapa service membutuhkan Redis. Jalankan dengan:
|
||||
|
||||
```bash
|
||||
make dev
|
||||
# atau equivalen:
|
||||
docker compose -f infra/compose/shared.yml up -d
|
||||
```
|
||||
|
||||
Ini akan menjalankan Redis Alpine di `localhost:6379`.
|
||||
|
||||
### (Opsional) NATS Lokal
|
||||
|
||||
Jika service membutuhkan pub/sub:
|
||||
|
||||
```bash
|
||||
docker compose -f infra/compose/nats.yml up -d
|
||||
# NATS client: localhost:4222
|
||||
# NATS monitor: localhost:8222
|
||||
```
|
||||
|
||||
### (Opsional) Dapr Placement Lokal
|
||||
|
||||
Jika service membutuhkan sidecar Dapr:
|
||||
|
||||
```bash
|
||||
docker compose -f infra/compose/dapr.yml up -d
|
||||
# Dapr placement: localhost:50005
|
||||
```
|
||||
|
||||
## Menjalankan Service Lokal
|
||||
|
||||
### Scraper API (Rust)
|
||||
|
||||
```bash
|
||||
# Pastikan Redis sudah running (make dev)
|
||||
cd apps/scraper
|
||||
|
||||
# Cargo run
|
||||
cargo run
|
||||
|
||||
# Dengan Dapr sidecar (jika placement running)
|
||||
dapr run \
|
||||
--app-id scraper-api \
|
||||
--app-port 4091 \
|
||||
--dapr-http-port 3500 \
|
||||
--resources-path ../../infra/dapr/components \
|
||||
-- cargo run
|
||||
```
|
||||
|
||||
### Dengan Docker Compose (Full Stack)
|
||||
|
||||
Untuk menjalankan semua service sekaligus:
|
||||
|
||||
```bash
|
||||
docker compose \
|
||||
-f infra/compose/shared.yml \
|
||||
-f infra/compose/nats.yml \
|
||||
-f infra/compose/dapr.yml \
|
||||
-f infra/compose/scraper.yml \
|
||||
--env-file .env \
|
||||
up -d
|
||||
```
|
||||
|
||||
Untuk service baru, tambahkan compose file-nya ke daftar.
|
||||
|
||||
## Update Submodules
|
||||
|
||||
### Pull latest dari semua submodule
|
||||
|
||||
```bash
|
||||
make update-submodules
|
||||
# atau:
|
||||
git submodule update --remote --merge --recursive
|
||||
```
|
||||
|
||||
### Check status submodule
|
||||
|
||||
```bash
|
||||
make status
|
||||
# atau:
|
||||
git submodule status
|
||||
```
|
||||
|
||||
### Sync .env ke submodule
|
||||
|
||||
```bash
|
||||
bash scripts/2updateenv.sh
|
||||
# Copy .env root ke apps/*/
|
||||
```
|
||||
|
||||
## Linting & Formatting
|
||||
|
||||
Root repo menggunakan **Biome** untuk linting dan formatting:
|
||||
|
||||
```bash
|
||||
bun run check # Lint + format + write
|
||||
bun run ci # CI mode (no write, exit code on issues)
|
||||
bun run lint # Lint only
|
||||
bun run format # Format only
|
||||
```
|
||||
|
||||
## Build Docker Image Lokal
|
||||
|
||||
```bash
|
||||
# Scraper API
|
||||
docker build -f infra/docker/scraper.Dockerfile -t scraper-api:local .
|
||||
|
||||
# Service baru: tambahkan Dockerfile di infra/docker/
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Saat ini belum ada test runner di root level. Masing-masing submodule mengelola testing sendiri:
|
||||
|
||||
```bash
|
||||
# Scraper API (Rust)
|
||||
cd apps/scraper && cargo test
|
||||
```
|
||||
|
||||
## Validasi YAML
|
||||
|
||||
Sebelum commit perubahan infra, validasi semua file YAML:
|
||||
|
||||
```bash
|
||||
python -c "
|
||||
import pathlib, yaml
|
||||
for p in pathlib.Path('infra').rglob('*.yml'):
|
||||
with open(p) as f: yaml.safe_load(f)
|
||||
print(f'OK {p}')
|
||||
for p in pathlib.Path('infra').rglob('*.yaml'):
|
||||
with open(p) as f: yaml.safe_load(f)
|
||||
print(f'OK {p}')
|
||||
"
|
||||
|
||||
for f in infra/compose/*.yml; do
|
||||
docker compose -f "$f" config >/dev/null && echo "OK $f"
|
||||
done
|
||||
```
|
||||
|
||||
## Git Workflow
|
||||
|
||||
### Commit Convention
|
||||
|
||||
```
|
||||
<type>(<scope>): <description>
|
||||
```
|
||||
|
||||
Type: `feat`, `fix`, `chore`, `docs`, `refactor`, `test`, `ci`, `perf`, `style`
|
||||
Scope: `scraper`, `infra`, `ci`, `dapr`, `nats`, `docs`, `deps`, `scripts`, `root`
|
||||
|
||||
Contoh:
|
||||
```
|
||||
feat(scraper): add image cache endpoint
|
||||
fix(infra): correct Traefik rate-limit config
|
||||
chore(deps): bump biome to 2.5.0
|
||||
```
|
||||
|
||||
### Branch Strategy
|
||||
|
||||
- `main` — production branch, push triggers CI/CD
|
||||
- Fitur baru: branch dari `main`, PR ke `main`
|
||||
- Submodule development: dilakukan di repo masing-masing, hub hanya update pointer
|
||||
|
||||
## Deployment ke VPS
|
||||
|
||||
Push ke `main` otomatis trigger CI/CD. Untuk trigger manual:
|
||||
|
||||
```bash
|
||||
gh workflow run deploy-docker.yml
|
||||
```
|
||||
|
||||
Lihat `docs/DEPLOYMENT.md` untuk detail.
|
||||
@@ -1,162 +0,0 @@
|
||||
# Menambahkan Dapr ke Service Baru
|
||||
|
||||
Panduan integrasi Dapr runtime sidecar untuk service di `asepharyana-hub`.
|
||||
|
||||
## Prasyarat
|
||||
|
||||
- NATS server berjalan (`infra/compose/nats.yml`)
|
||||
- Dapr placement service berjalan (`infra/compose/dapr.yml`)
|
||||
|
||||
## 1. Compose File
|
||||
|
||||
Setiap service butuh sidecar container Dapr. Contoh:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
container_name: app
|
||||
image: ghcr.io/asepharyana/asepharyana-hub/app:latest
|
||||
restart: always
|
||||
depends_on:
|
||||
dapr-placement:
|
||||
condition: service_healthy
|
||||
nats:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
app-shared-net:
|
||||
aliases:
|
||||
- app
|
||||
env_file:
|
||||
- ../../.env
|
||||
|
||||
app-dapr:
|
||||
container_name: app-dapr
|
||||
image: daprio/daprd:latest
|
||||
restart: always
|
||||
depends_on:
|
||||
dapr-placement:
|
||||
condition: service_healthy
|
||||
nats:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- app-shared-net
|
||||
depends_on:
|
||||
dapr-placement:
|
||||
condition: service_healthy
|
||||
nats:
|
||||
condition: service_healthy
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
networks:
|
||||
- app-shared-net
|
||||
command:
|
||||
- './daprd'
|
||||
- '--app-id=app'
|
||||
- '--app-port=3000'
|
||||
- '--dapr-http-port=3500'
|
||||
- '--dapr-grpc-port=50001'
|
||||
- '--placement-host-address=dapr-placement:50005'
|
||||
- '--config=/dapr/config.yaml'
|
||||
- '--resources-path=/dapr/components'
|
||||
volumes:
|
||||
- ../../infra/dapr:/dapr:ro
|
||||
|
||||
networks:
|
||||
app-shared-net:
|
||||
name: app-shared-net
|
||||
external: true
|
||||
```
|
||||
|
||||
## 2. Mengakses Dapr dari Service
|
||||
|
||||
### Via HTTP API (semua bahasa)
|
||||
|
||||
Sidecar listen di `localhost:3500`:
|
||||
|
||||
```bash
|
||||
# Publish event
|
||||
curl -X POST http://localhost:3500/v1.0/publish/pubsub/hub.event.type \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"key": "value"}'
|
||||
|
||||
# Service invocation
|
||||
curl http://localhost:3500/v1.0/invoke/app/method/endpoint
|
||||
|
||||
# State store
|
||||
curl -X POST http://localhost:3500/v1.0/state/statestore \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '[{"key": "mykey", "value": "myvalue"}]'
|
||||
```
|
||||
|
||||
### Via Dapr SDK (Rust)
|
||||
|
||||
Tambah ke `Cargo.toml`:
|
||||
|
||||
```toml
|
||||
dapr-sdk = { version = "0.15", features = ["pubsub", "http"] }
|
||||
tokio-stream = "0.1"
|
||||
```
|
||||
|
||||
Contoh publish event:
|
||||
|
||||
```rust
|
||||
use dapr_sdk::client::{Client, Event};
|
||||
use dapr_sdk::DaprClient;
|
||||
|
||||
let client = DaprClient::new("127.0.0.1", 3500).await?;
|
||||
client.publish_event("pubsub", "hub.image.cached", serde_json::json!({
|
||||
"original_url": url,
|
||||
"cdn_url": cdn_url,
|
||||
})).await?;
|
||||
```
|
||||
|
||||
Contoh subscribe event:
|
||||
|
||||
```rust
|
||||
let mut stream = client.subscribe_events("pubsub", "hub.image.cached").await?;
|
||||
while let Some(event) = stream.next().await {
|
||||
let data: MyEvent = serde_json::from_slice(&event.data)?;
|
||||
// handle event
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Event Topics Convention
|
||||
|
||||
Gunakan prefix `hub.` untuk semua event:
|
||||
|
||||
| Topic | Payload | Description |
|
||||
|-------|---------|-------------|
|
||||
| `hub.image.cached` | `{original_url, cdn_url, source}` | Image selesai di-cache |
|
||||
| `hub.image.repaired` | `{old_url, new_url}` | CNAME image diperbaiki |
|
||||
| `hub.scrape.anime.done` | `{source, slug, duration}` | Scrape anime selesai |
|
||||
| `hub.system.alert` | `{service, level, message}` | Error/alert dari service |
|
||||
|
||||
## 4. Local Development
|
||||
|
||||
Untuk development tanpa Docker:
|
||||
|
||||
```bash
|
||||
# 1. Install Dapr CLI
|
||||
# 2. Init Dapr local
|
||||
dapr init
|
||||
|
||||
# 3. Run service dengan sidecar
|
||||
dapr run --app-id app --app-port 3000 --dapr-http-port 3500 \
|
||||
--resources-path ./infra/dapr/components \
|
||||
-- cargo run
|
||||
```
|
||||
|
||||
## 5. Verifikasi
|
||||
|
||||
```bash
|
||||
# Sidecar health
|
||||
curl http://localhost:3500/v1.0/healthz
|
||||
|
||||
# Publish test event
|
||||
curl -X POST http://localhost:3500/v1.0/publish/pubsub/hub.test \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"test": true}'
|
||||
|
||||
# NATS stream stats
|
||||
curl http://localhost:8222/jszetstream
|
||||
```
|
||||
+33
-158
@@ -1,167 +1,42 @@
|
||||
# Menambahkan Aplikasi Baru ke Deployment
|
||||
# Menambahkan Service Baru / Subdomain
|
||||
|
||||
Dokumen ini menjelaskan langkah menambahkan service baru ke `asepharyana-hub`. Root repo berfungsi sebagai hub: source aplikasi berada di `apps/<nama-app>` sebagai submodule, sedangkan Docker Compose, Traefik, dan workflow deploy tetap berada di root repo.
|
||||
Panduan untuk menambahkan service baru di ekosistem `asepharyana/infra` (2026-08-28+, pasca monorepo).
|
||||
|
||||
## 1. Buat repo aplikasi
|
||||
## Prinsip
|
||||
|
||||
Buat repo baru di GitHub dengan pola nama:
|
||||
- **Aplikasi hidup di repo sendiri** (`hub`, `scraper`, `tools`, `llm-api`) dengan
|
||||
`flake.nix` + `.github/workflows/deploy.yml` mandiri. Repo infra TIDAK berisi kode app.
|
||||
- Repo infra (`asepharyana/infra`) hanya mengatur **reverse proxy & config VPS**.
|
||||
|
||||
```text
|
||||
https://github.com/asepharyana/asepharyana-hub-<nama-app>.git
|
||||
## Langkah
|
||||
|
||||
1. **Buat repo aplikasi** (contoh pola: `asepharyana/scraper`).
|
||||
2. **Tambahkan `flake.nix`** di repo app — derivasi Nix (lihat template di repo app yang ada:
|
||||
Next.js/bun atau Rust/cargo). Nama paket = nama unit systemd.
|
||||
3. **Tambahkan `.github/workflows/deploy.yml`** (pola `nix build .#<pkg>` → `nix copy ssh://`
|
||||
→ `nix-env --profile /nix/var/nix/profiles/<pkg> --set` → `systemctl restart <pkg>`).
|
||||
Secrets yang dibutuhkan: `SSH_PRIVATE_KEY`, `VPS_HOST`, `VPS_USER`.
|
||||
4. **Di VPS**: buat user systemd + unit (mis. `/etc/systemd/system/<app>.service`,
|
||||
`ExecStart=/usr/local/bin/bws-exec <app> /nix/var/nix/profiles/<app>/bin/<app>`),
|
||||
pastikan app jalan di port lokal.
|
||||
5. **Tambah site block** di `infra/caddy/Caddyfile.prod` (pola `import proxy <port>`),
|
||||
push ke `main` → CI `caddy-deploy.yml` sync + reload + verifikasi rute.
|
||||
6. **(Opsional)** Tambah unit ke `MONITORED_UNITS` di hub dashboard
|
||||
(repo `asepharyana/hub`, `src/app/api/dashboard/route.ts`).
|
||||
|
||||
## Contoh site block Caddy
|
||||
|
||||
```caddyfile
|
||||
nama-app.asepharyana.my.id {
|
||||
import proxy <PORT>
|
||||
}
|
||||
```
|
||||
|
||||
Lalu tambahkan ke root hub sebagai submodule:
|
||||
## Verifikasi
|
||||
|
||||
```bash
|
||||
git submodule add https://github.com/asepharyana/asepharyana-hub-<nama-app>.git apps/<nama-app>
|
||||
git submodule update --init --recursive
|
||||
```
|
||||
# Dari local
|
||||
curl -s -o /dev/null -w '%{http_code}\n' https://nama-app.asepharyana.my.id/
|
||||
|
||||
## 2. Tambahkan Dockerfile
|
||||
|
||||
Tambahkan Dockerfile runtime di `infra/docker/<nama-app>.Dockerfile`.
|
||||
|
||||
Gunakan root repo sebagai build context agar Dockerfile bisa mengakses submodule path:
|
||||
|
||||
```bash
|
||||
docker build -f infra/docker/<nama-app>.Dockerfile -t <nama-app>:local .
|
||||
```
|
||||
|
||||
## 3. Tambahkan Compose file
|
||||
|
||||
Buat `infra/compose/<nama-app>.yml`:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
<nama-app>:
|
||||
container_name: <nama-app>
|
||||
image: ghcr.io/asepharyana/asepharyana-hub/<nama-app>:sha-<short-sha>
|
||||
restart: always
|
||||
networks:
|
||||
app-shared-net:
|
||||
aliases:
|
||||
- <nama-app>
|
||||
env_file:
|
||||
- ../../.env
|
||||
|
||||
networks:
|
||||
app-shared-net:
|
||||
name: app-shared-net
|
||||
external: true
|
||||
```
|
||||
|
||||
Gunakan `app-shared-net` agar service dapat diakses oleh Traefik dan service lain.
|
||||
|
||||
## 3.5. Tambahkan Dapr sidecar (wajib untuk pub/sub)
|
||||
|
||||
Setiap service yang ingin menggunakan Dapr pub/sub atau service invocation harus punya sidecar.
|
||||
Tambah di `infra/compose/<nama-app>.yml`:
|
||||
|
||||
```yaml
|
||||
<nama-app>-dapr:
|
||||
container_name: <nama-app>-dapr
|
||||
image: daprio/daprd:latest
|
||||
restart: always
|
||||
depends_on:
|
||||
dapr-placement:
|
||||
condition: service_healthy
|
||||
nats:
|
||||
condition: service_healthy
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
networks:
|
||||
- app-shared-net
|
||||
command:
|
||||
- './daprd'
|
||||
- '--app-id=<nama-app>'
|
||||
- '--app-port=<port>'
|
||||
- '--dapr-http-port=3500'
|
||||
- '--dapr-grpc-port=50001'
|
||||
- '--placement-host-address=dapr-placement:50005'
|
||||
- '--config=/dapr/config.yaml'
|
||||
- '--resources-path=/dapr/components'
|
||||
volumes:
|
||||
- ../../infra/dapr:/dapr:ro
|
||||
```
|
||||
|
||||
Pastikan juga app container punya `depends_on` ke dapr-placement, nats, dan otel-collector:
|
||||
```yaml
|
||||
depends_on:
|
||||
dapr-placement:
|
||||
condition: service_healthy
|
||||
nats:
|
||||
condition: service_healthy
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
```
|
||||
|
||||
## 4. Tambahkan route Traefik
|
||||
|
||||
Update `infra/traefik/dynamic/apps.yaml`:
|
||||
|
||||
```yaml
|
||||
http:
|
||||
routers:
|
||||
<nama-app>:
|
||||
rule: 'Host(`<subdomain>.asepharyana.my.id`) || Host(`<subdomain>.asepharyana.web.id`)'
|
||||
entryPoints:
|
||||
- websecure
|
||||
tls: {}
|
||||
middlewares:
|
||||
- common-chain@file
|
||||
service: <nama-app>-service
|
||||
|
||||
services:
|
||||
<nama-app>-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: 'http://<nama-app>:<port>'
|
||||
```
|
||||
|
||||
## 5. Update workflow build
|
||||
|
||||
Update `.github/workflows/docker-build-push.yml`:
|
||||
|
||||
1. Tambahkan path detection untuk `apps/<nama-app>` dan `infra/docker/<nama-app>.Dockerfile`.
|
||||
2. Tambahkan service ke matrix build.
|
||||
3. Tambahkan mapping Dockerfile di step `Docker metadata`.
|
||||
4. Tambahkan mapping compose file dan submodule path di step `Update tags and submodules`.
|
||||
|
||||
## 6. Update workflow deploy
|
||||
|
||||
Tambahkan compose file baru ke `ALL_COMPOSE_FILES` di `.github/workflows/deploy-docker.yml`:
|
||||
|
||||
```bash
|
||||
infra/compose/<nama-app>.yml
|
||||
```
|
||||
|
||||
## 7. Update dokumentasi
|
||||
|
||||
Update file berikut bila service baru mengubah arsitektur publik:
|
||||
|
||||
- `README.md`
|
||||
- `ARCHITECTURE.md`
|
||||
- `infra/README.md`
|
||||
- `.gitmodules`
|
||||
|
||||
## 8. Validasi
|
||||
|
||||
Jalankan validasi YAML dan compose rendering:
|
||||
|
||||
```bash
|
||||
python - <<'PY'
|
||||
import pathlib, yaml
|
||||
for path in pathlib.Path('infra').rglob('*.yml'):
|
||||
with path.open() as fh:
|
||||
yaml.safe_load(fh)
|
||||
print(f'OK {path}')
|
||||
for path in pathlib.Path('infra').rglob('*.yaml'):
|
||||
with path.open() as fh:
|
||||
yaml.safe_load(fh)
|
||||
print(f'OK {path}')
|
||||
PY
|
||||
|
||||
for f in infra/compose/*.yml; do
|
||||
docker compose -f "$f" config >/dev/null && echo "OK $f"
|
||||
done
|
||||
```
|
||||
# Dari VPS
|
||||
systemctl status <app>
|
||||
@@ -1,49 +1,38 @@
|
||||
# ADR 0001: Use a Hub Repository with App Submodules
|
||||
# ADR 0001: Infra Repo — Reverse Proxy Config Only (Submodules Removed)
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
**Superseded** (2026-08-28) — lihat ADR ini sebagai arsip keputusan awal.
|
||||
|
||||
## Context
|
||||
## Context (aslinya)
|
||||
|
||||
The project contains multiple independent application services that share one deployment surface: Docker Compose, Traefik routing, GitHub Actions workflows, and operational documentation.
|
||||
Proyek awal memakai `asepharyana-hub` sebagai monorepo: aplikasi di `apps/<service>` sebagai
|
||||
git submodule, infra (compose/traefik/dokumen/CI) terpusat di root. Keputusan itu masuk akal
|
||||
saat semua service berbagi satu deployment surface.
|
||||
|
||||
The services should be developed and versioned independently, while deployment infrastructure should remain centralized so production routing and compose manifests stay consistent.
|
||||
## Decision (aslinya)
|
||||
|
||||
## Decision
|
||||
Gunakan `asepharyana-hub` sebagai root hub: app code submodule, infra + CI di root.
|
||||
|
||||
Use `asepharyana-hub` as the root hub repository.
|
||||
## Superseded By
|
||||
|
||||
- Application code lives under `apps/<service>` as Git submodules.
|
||||
- Infrastructure lives in the root repo under `infra/`.
|
||||
- Documentation lives in the root repo under `docs/`.
|
||||
- CI/CD workflows live in the root repo under `.github/workflows/`.
|
||||
- Root tooling stays minimal: `package.json`, Prettier, ESLint, Makefile helpers, and deployment scripts.
|
||||
Mulai **2026-08-28** repo dirombak:
|
||||
|
||||
Current app submodules:
|
||||
|
||||
| Service | Path | Remote |
|
||||
| ----------- | -------------- | --------------------------------------- |
|
||||
| Scraper API | `apps/scraper` | `asepharyana/asepharyana-hub-scraper` |
|
||||
- **Parent `asepharyana-hub` → `asepharyana/infra`** — murni config reverse proxy (Caddy),
|
||||
firewall, systemd drop-ins, docs. CI hanya untuk deploy Caddy.
|
||||
- **App repos di-rename & self-contained**: `hub`, `scraper`, `tools`, `llm-api`.
|
||||
Masing-masing punya `flake.nix` + `.github/workflows/deploy.yml` sendiri
|
||||
(`nix build → nix copy → nix-env --profile → systemctl restart`).
|
||||
- **Submodule dihapus** — tidak ada lagi pointer submodule / repository_dispatch chain.
|
||||
- `update-submodule.yml`, `notify-parent.yml`, matrix `nix-build.yml` dihapus.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Each app can evolve in its own repository.
|
||||
- The hub pins exact submodule revisions for reproducible deployments.
|
||||
- Deployment infrastructure remains centralized and easier to audit.
|
||||
- Root tooling stays lightweight and does not impose one build system on every service.
|
||||
- CI tiap app independen: push ke repo app langsung build+deploy, tak perlu 2 hop.
|
||||
- Parent kecil & fokus: diff Caddyfile mudah di-audit.
|
||||
- Tanpa submodule = tanpa `dubious ownership` / pointer drift / fetchGit pin.
|
||||
|
||||
### Negative
|
||||
|
||||
- Developers must understand Git submodule workflows.
|
||||
- Updating a service requires updating the submodule pointer in the hub repo.
|
||||
- Cross-service changes require coordinating commits across multiple repositories.
|
||||
|
||||
### Mitigations
|
||||
|
||||
- Keep `.gitmodules` accurate and minimal.
|
||||
- Use `scripts/sync-submodules.sh` for local checkout consistency.
|
||||
- Document service-addition steps in `docs/add-new-app.md`.
|
||||
- Keep GitHub Actions responsible for Docker image builds, compose tag updates, and deployments.
|
||||
- Koordinasi cross-repo manual (app + Caddy bila perlu port baru).
|
||||
- Repo lama `asepharyana-hub-*` redirect ke nama baru (GitHub auto).
|
||||
@@ -1,8 +1,12 @@
|
||||
# ADR 0002: Production `.env` via GitHub Encrypted Secret
|
||||
|
||||
> **LEGACY (2026-08-28):** Repo `asepharyana-hub` sudah dirombak → `asepharyana/infra`.
|
||||
> Workflow lama yang SCP `.env` ke VPS tidak dipakai lagi (deploy app pindah ke repo masing-masing,
|
||||
> secrets via Bitwarden `bws-exec`). ADR ini dipertahankan sebagai arsip.
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
Accepted (archived)
|
||||
|
||||
## Context
|
||||
|
||||
@@ -33,7 +37,7 @@ Container reads $DATABASE_URL, $JWT_SECRET, etc.
|
||||
|
||||
```bash
|
||||
# 1. Read current content from the VPS
|
||||
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env"
|
||||
ssh root@45.127.35.244 "cat <VPS app dir, e.g. /home/code/hub>/.env"
|
||||
|
||||
# 2. Pipe updated content to the GitHub secret
|
||||
# (requires gh CLI with repo access)
|
||||
@@ -43,7 +47,7 @@ cat /path/to/updated-env | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/
|
||||
gh workflow run deploy-docker.yml
|
||||
|
||||
# OR apply immediately on the VPS (for hotfix):
|
||||
ssh root@45.127.35.244 "sed -i 's|OLD_VALUE|NEW_VALUE|' /root/asepharyana-hub/.env"
|
||||
ssh root@45.127.35.244 "sed -i 's|OLD_VALUE|NEW_VALUE|' <VPS app dir, e.g. /home/code/hub>/.env"
|
||||
# Then restart affected containers
|
||||
```
|
||||
|
||||
@@ -69,7 +73,7 @@ ssh root@45.127.35.244 "sed -i 's|OLD_VALUE|NEW_VALUE|' /root/asepharyana-hub/.e
|
||||
The VPS runs a single Docker Compose project named `compose` composed of multiple files:
|
||||
|
||||
```bash
|
||||
/root/asepharyana-hub/infra/compose/
|
||||
<VPS app dir, e.g. /home/code/hub>/infra/compose/
|
||||
├── traefik.yml # Reverse proxy (TLS termination, routing)
|
||||
├── shared.yml # Redis
|
||||
├── nats.yml # NATS message broker + JetStream
|
||||
@@ -99,7 +103,7 @@ docker compose \
|
||||
| `SSH_PRIVATE_KEY` | SSH key for VPS access |
|
||||
| `VPS_HOST` | `45.127.35.244` |
|
||||
| `VPS_USER` | `root` |
|
||||
| `VPS_TARGET_DIR` | `/root/asepharyana-hub` |
|
||||
| `VPS_TARGET_DIR` | `<VPS app dir, e.g. /home/code/hub>` |
|
||||
| `ENV_FILE_PRODUCTION` | Full `.env` content for production |
|
||||
|
||||
## Consequences
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# ADR 0003: Rename Repositori & Pisahkan CI per Aplikasi
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (2026-08-28)
|
||||
|
||||
## Context
|
||||
|
||||
Monorepo `asepharyana-hub` (app submodule + infra + CI terpusat) punya kelemahan:
|
||||
- CI build+deploy semua app menyatu di parent (`nix-build.yml` matrix) — setiap push app
|
||||
butuh 2 hop (notify-parent → update-submodule → nix-build), rawan drift pointer.
|
||||
- Nama `asepharyana-hub` ambigu (parent & app prefix sama), dan submodule menambah kompleksitas.
|
||||
|
||||
## Decision
|
||||
|
||||
Rombak total:
|
||||
|
||||
| Lama | Baru | Peran |
|
||||
|------|------|-------|
|
||||
| `asepharyana-hub` | `asepharyana/infra` | Reverse proxy (Caddy) + firewall + systemd + docs. CI: caddy-deploy saja. |
|
||||
| `asepharyana-hub-hub` | `asepharyana/hub` | Portfolio SPA. CI mandiri (deploy.yml). |
|
||||
| `asepharyana-hub-scraper` | `asepharyana/scraper` | Rust scraper API. CI mandiri. |
|
||||
| `asepharyana-hub-tools` | `asepharyana/tools` | Tools stack (gateway/workers/frontend). CI mandiri. |
|
||||
| `asepharyana-hub-llm-api` | `asepharyana/llm-api` | Rust LLM API. CI mandiri. |
|
||||
| `asepharyana-hub-guide` | `asepharyana/hub-guide` | (tidak di-root; plugin guide — diarsipkan) |
|
||||
|
||||
Setiap app repo mendapat:
|
||||
- `flake.nix` (derivasi build sendiri, tanpa fetchGit submodule)
|
||||
- `.github/workflows/deploy.yml` (nix build → nix copy → nix-env --profile → systemctl restart)
|
||||
- Secret `SSH_PRIVATE_KEY`, `VPS_HOST`, `VPS_USER`
|
||||
|
||||
Parent `infra` mendapat:
|
||||
- Hapus semua submodule + `update-submodule.yml` + matrix `nix-build.yml`
|
||||
- `.github/workflows/caddy-deploy.yml` (sync Caddyfile → reload → verify)
|
||||
- Docs diarahkan ulang.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positif**: CI per-app independen & cepat; parent kecil; tanpa submodule = tanpa fetchGit pin
|
||||
/ dubious-ownership / pointer churn. Rename GitHub auto-redirect URL lama.
|
||||
- **Negatif**: koordinasi manual bila app butuh port baru di Caddy; workflow lama di
|
||||
downstream (skill/cron) perlu update referensi.
|
||||
|
||||
## Referensi
|
||||
|
||||
- `docs/add-new-app.md` — proses menambah service baru
|
||||
- `infra/caddy/Caddyfile.prod` — pola site block
|
||||
@@ -1,234 +0,0 @@
|
||||
# Backup & Disaster Recovery
|
||||
|
||||
## Aset yang Perlu di-Backup
|
||||
|
||||
| Aset | Lokasi | Frekuensi | Metode |
|
||||
|------|--------|-----------|--------|
|
||||
| Database PostgreSQL | `imrnes` (100.121.180.82:6432) | Harian | `pg_dump` |
|
||||
| Volume Redis | `orangevps` (Docker volume) | Opsional | Redis RDB / AOF |
|
||||
| Volume NATS JetStream | `orangevps` (Docker volume) | Opsional | File copy |
|
||||
| Docker Compose manifests | GitHub (hub repo) | Real-time | Git |
|
||||
| Environment variables | GitHub secret `ENV_FILE_PRODUCTION` | Manual | `gh secret set` |
|
||||
| TLS certificates | `orangevps` (`/root/*.pem`, `*.key`) | Saat renew | SCP |
|
||||
| Tailscale auth | Tailscale admin console | - | Cloud-managed |
|
||||
| GitHub Actions secrets | GitHub UI | Manual | Backup list |
|
||||
|
||||
## Database PostgreSQL (Prioritas Tertinggi)
|
||||
|
||||
### Backup Manual
|
||||
|
||||
```bash
|
||||
# Dari orangevps (via Tailscale)
|
||||
pg_dump -h 100.121.180.82 -p 6432 -U asephs -d hub \
|
||||
--no-owner --no-acl \
|
||||
-F c -f /root/db-backups/hub-$(date +%Y%m%d-%H%M%S).dump
|
||||
|
||||
# Atau dari imrnes langsung
|
||||
pg_dump -U asephs -d hub \
|
||||
-F c -f /backup/hub/hub-$(date +%Y%m%d-%H%M%S).dump
|
||||
```
|
||||
|
||||
### Restore
|
||||
|
||||
```bash
|
||||
# Drop dan recreate database
|
||||
dropdb -h 100.121.180.82 -p 6432 -U asephs hub
|
||||
createdb -h 100.121.180.82 -p 6432 -U asephs hub
|
||||
|
||||
# Restore dari dump
|
||||
pg_restore -h 100.121.180.82 -p 6432 -U asephs -d hub \
|
||||
--no-owner --no-acl \
|
||||
/path/to/backup/hub-20260101-120000.dump
|
||||
```
|
||||
|
||||
### Backup Otomatis (via Cron di imrnes)
|
||||
|
||||
```bash
|
||||
# /etc/cron.d/hub-db-backup
|
||||
0 2 * * * root pg_dump -U asephs -d hub -F c -f /backup/hub/hub-$(date +\%Y\%m\%d).dump && find /backup/hub -name "hub-*.dump" -mtime +30 -delete
|
||||
```
|
||||
|
||||
## Volume Docker
|
||||
|
||||
### Redis
|
||||
|
||||
Redis data bisa di-recover dari NATS events (event sourcing). Jika tidak ada persistence requirement, cukup restart:
|
||||
|
||||
```bash
|
||||
docker volume rm redis_data
|
||||
docker compose -f infra/compose/shared.yml up -d
|
||||
```
|
||||
|
||||
Jika perlu backup:
|
||||
|
||||
```bash
|
||||
# Save RDB snapshot
|
||||
docker exec redis redis-cli SAVE
|
||||
|
||||
# Copy dari volume
|
||||
docker run --rm -v redis_data:/data -v /backup:/backup alpine cp /data/dump.rdb /backup/redis-$(date +%Y%m%d).rdb
|
||||
```
|
||||
|
||||
### NATS JetStream
|
||||
|
||||
```bash
|
||||
# Backup volume
|
||||
docker run --rm -v nats_data:/data -v /backup:/backup alpine \
|
||||
tar czf /backup/nats-$(date +%Y%m%d).tar.gz -C /data .
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Backup `.env` dari VPS
|
||||
|
||||
```bash
|
||||
# Simpan current .env dari VPS
|
||||
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env" > .env.backup.$(date +%Y%m%d)
|
||||
|
||||
# Update GitHub secret
|
||||
cat .env.backup.$(date +%Y%m%d) | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub
|
||||
```
|
||||
|
||||
### Restore `.env` jika hilang
|
||||
|
||||
```bash
|
||||
# Buat .env baru dari template
|
||||
cp .env.example .env
|
||||
|
||||
# Edit secrets (manual dari password manager atau GitHub secret)
|
||||
# Atau download dari GitHub secret
|
||||
gh secret list --repo asepharyana/asepharyana-hub
|
||||
```
|
||||
|
||||
## TLS Certificates
|
||||
|
||||
### Backup
|
||||
|
||||
```bash
|
||||
# Di orangevps
|
||||
tar czf /root/cert-backup-$(date +%Y%m%d).tar.gz \
|
||||
/root/asepharyana.my.id.pem \
|
||||
/root/asepharyana.my.id.key \
|
||||
/root/asepharyana.web.id.pem \
|
||||
/root/asepharyana.web.id.key \
|
||||
/root/asepharyana-hub/infra/traefik/dynamic/ssl.yaml
|
||||
|
||||
# SCP ke local
|
||||
scp root@45.127.35.244:/root/cert-backup-*.tar.gz .
|
||||
```
|
||||
|
||||
### Restore
|
||||
|
||||
```bash
|
||||
# SCP ke VPS
|
||||
scp cert-backup-20260101.tar.gz root@45.127.35.244:/root/
|
||||
|
||||
# Extract
|
||||
ssh root@45.127.35.244 "tar xzf /root/cert-backup-20260101.tar.gz -C / && docker restart traefik"
|
||||
```
|
||||
|
||||
## Disaster Recovery Scenarios
|
||||
|
||||
### Skenario 1: VPS (orangevps) mati total
|
||||
|
||||
**Dampak:** Semua service down.
|
||||
|
||||
**Recovery:**
|
||||
|
||||
```bash
|
||||
# 1. Provision VPS baru (atau restore dari snapshot)
|
||||
# 2. Install Docker + Tailscale
|
||||
# 3. Clone repo
|
||||
git clone https://github.com/asepharyana/asepharyana-hub.git /root/asepharyana-hub
|
||||
|
||||
# 4. Setup Tailscale, route service
|
||||
# 5. Restore .env
|
||||
echo "<ENV_FILE_PRODUCTION>" > /root/asepharyana-hub/.env
|
||||
|
||||
# 6. Restore TLS certs
|
||||
# 7. Create network
|
||||
docker network create app-shared-net
|
||||
|
||||
# 8. Start services sesuai urutan
|
||||
cd /root/asepharyana-hub
|
||||
for f in shared.yml nats.yml dapr.yml traefik.yml scraper.yml; do
|
||||
docker compose -f infra/compose/$f --env-file .env up -d
|
||||
done
|
||||
|
||||
# 9. Update DNS jika IP baru
|
||||
```
|
||||
|
||||
### Skenario 2: Database (imrnes) mati total
|
||||
|
||||
**Dampak:** Semua service yang butuh database error.
|
||||
|
||||
**Recovery:**
|
||||
|
||||
```bash
|
||||
# 1. Fix imrnes atau provision server baru
|
||||
# 2. Setup PostgreSQL
|
||||
# 3. Restore dari backup terakhir
|
||||
# 4. Update Tailscale IP jika perlu
|
||||
# 5. Update .env dan GitHub secret
|
||||
# 6. Redeploy
|
||||
```
|
||||
|
||||
### Skenario 3: GitHub repository hilang
|
||||
|
||||
**Dampak:** Kehilangan CI/CD, tapi Docker images masih ada di GHCR.
|
||||
|
||||
**Recovery:**
|
||||
|
||||
```bash
|
||||
# 1. Create repo baru di GitHub
|
||||
# 2. Push dari local clone
|
||||
git remote add origin-new https://github.com/asepharyana/asepharyana-hub-new.git
|
||||
git push origin-new main
|
||||
|
||||
# 3. Re-create GitHub secrets
|
||||
# 4. Re-create workflows
|
||||
# 5. Update VPS remote
|
||||
ssh root@45.127.35.244 "cd /root/asepharyana-hub && git remote set-url origin https://github.com/asepharyana/asepharyana-hub-new.git"
|
||||
```
|
||||
|
||||
### Skenario 4: GHCR registry tidak bisa diakses
|
||||
|
||||
**Dampak:** Tidak bisa pull image.
|
||||
|
||||
**Recovery:**
|
||||
|
||||
```bash
|
||||
# 1. Build image langsung di VPS
|
||||
docker build -f infra/docker/scraper.Dockerfile -t ghcr.io/asepharyana/asepharyana-hub/scraper-api:local .
|
||||
|
||||
# 2. Update compose file untuk sementara
|
||||
sed -i 's|image: ghcr.io/.*|image: ghcr.io/asepharyana/asepharyana-hub/scraper-api:local|' infra/compose/scraper.yml
|
||||
|
||||
# 3. Start
|
||||
docker compose -f infra/compose/scraper.yml up -d
|
||||
```
|
||||
|
||||
### Skenario 5: Semua server mati (total loss)
|
||||
|
||||
**Recovery:**
|
||||
|
||||
```bash
|
||||
# 1. Provision VPS baru
|
||||
# 2. Provision server database baru
|
||||
# 3. Setup Tailscale
|
||||
# 4. Clone repo, restore .env, certs
|
||||
# 5. Restore database dari backup (jika ada)
|
||||
# 6. Jika tidak ada backup database:
|
||||
# - Build image dari GHCR
|
||||
# - Start service dengan database kosong
|
||||
# - Data akan terisi ulang dari scraping
|
||||
```
|
||||
|
||||
## Checklist Pencegahan
|
||||
|
||||
- [ ] Cron job backup database berjalan
|
||||
- [ ] Backup `.env` disimpan di luar VPS (password manager)
|
||||
- [ ] TLS certificates backup disimpan di luar VPS
|
||||
- [ ] GitHub secrets terdaftar (tidak hanya diingat)
|
||||
- [ ] Docker images bisa di-rebuild dari CI (GHCR sebagai source of truth)
|
||||
- [ ] Tailscale admin access via multiple accounts
|
||||
@@ -1,248 +0,0 @@
|
||||
# CI/CD Pipeline
|
||||
|
||||
Dokumentasi pipeline CI/CD untuk `asepharyana-hub`. Terdiri dari 5 GitHub Actions workflow yang saling terhubung.
|
||||
|
||||
## Workflow Overview
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ Lint │ (PR/push → Biome)
|
||||
└──────┬──────┘
|
||||
│
|
||||
Push ke main ─────┼────── repository_dispatch
|
||||
│
|
||||
┌──────▼──────────────────┐
|
||||
│ docker-build-push.yml │
|
||||
│ │
|
||||
│ Phase 1: Detect │
|
||||
│ Phase 2: Build & Push │
|
||||
│ Phase 3: Update │
|
||||
│ manifests │
|
||||
└──────┬──────────────────┘
|
||||
│ workflow_run
|
||||
┌──────▼──────────────┐
|
||||
│ deploy-docker.yml │
|
||||
│ SSH → VPS │
|
||||
│ Pull → Restart │
|
||||
└─────────────────────┘
|
||||
|
||||
repository_dispatch ──► update-submodule.yml
|
||||
(dari submodule) (update pointer → commit)
|
||||
│
|
||||
▼
|
||||
docker-build-push.yml
|
||||
(triggered by push)
|
||||
```
|
||||
|
||||
## Workflow Detail
|
||||
|
||||
### 1. Lint (`lint.yml`)
|
||||
|
||||
**Trigger:** PR/push ke `main` yang mengubah `*.json`, `*.js`, `biome.json`
|
||||
|
||||
**Aksi:**
|
||||
- Checkout repo dengan submodules
|
||||
- Setup Bun
|
||||
- `bun install --frozen-lockfile`
|
||||
- `bun run ci` (Biome CI mode)
|
||||
|
||||
**Permissions:** read-only
|
||||
|
||||
### 2. Build and Push Docker Images (`docker-build-push.yml`)
|
||||
|
||||
**Trigger:**
|
||||
- Push ke `main` yang mengubah `apps/**`, `infra/**`, atau file workflow
|
||||
- `repository_dispatch` tipe `submodule-updated`
|
||||
- `workflow_dispatch` (manual)
|
||||
|
||||
**Concurrency:** Satu workflow per branch (cancel-in-progress=false)
|
||||
|
||||
#### Phase 1: Detect Changes
|
||||
|
||||
Job `changes` mendeteksi service mana yang perlu di-build:
|
||||
|
||||
- **Push event:** `git diff --name-only` antara `before` dan `after` SHA
|
||||
- **repository_dispatch:** Parse payload `{service, sha}` dan validasi
|
||||
- **workflow_dispatch:** Build semua service
|
||||
|
||||
Output format matrix:
|
||||
```json
|
||||
[{"id":"scraper-api","target":"docker-scraper","path":"apps/scraper"}]
|
||||
```
|
||||
|
||||
#### Phase 2: Build & Push (Matrix)
|
||||
|
||||
Job `build` berjalan paralel per service (matrix strategy):
|
||||
|
||||
1. Checkout repo + sync submodule
|
||||
2. Jika `repository_dispatch`, checkout submodule ke SHA tertentu
|
||||
3. Login ke GHCR
|
||||
4. Setup Docker Buildx
|
||||
5. Build & push dengan tag:
|
||||
- `ghcr.io/asepharyana/asepharyana-hub/<service>:latest`
|
||||
- `ghcr.io/asepharyana/asepharyana-hub/<service>:sha-<shortsha>`
|
||||
6. Build cache: registry-based (`:<service>:buildcache`)
|
||||
|
||||
#### Phase 3: Update Manifests
|
||||
|
||||
Job `update-manifest`:
|
||||
|
||||
1. Update image tag di compose file (`infra/compose/<service>.yml`)
|
||||
2. Jika `repository_dispatch`, update submodule pointer
|
||||
3. Commit dengan message `chore: update manifests and submodules [skip ci]`
|
||||
4. Push dengan retry (3 attempts, rebase jika conflict)
|
||||
|
||||
### 3. Deploy Docker to VPS (`deploy-docker.yml`)
|
||||
|
||||
**Trigger:**
|
||||
- `workflow_run` setelah `docker-build-push.yml` selesai
|
||||
- Push ke `main` yang mengubah `infra/**`
|
||||
- `workflow_dispatch` (manual)
|
||||
|
||||
**Concurrency:** Satu deployment dalam satu waktu (`group: deploy-vps`)
|
||||
|
||||
**Aksi di VPS (via SSH):**
|
||||
|
||||
```
|
||||
1. Setup SSH multiplexing
|
||||
2. SCP .env dari GitHub secret ke VPS
|
||||
3. Docker login ke GHCR
|
||||
4. Git sync (fetch + reset --hard)
|
||||
5. Detect changed files:
|
||||
├─ Compose stack changes → selective container update
|
||||
├─ Traefik dynamic config → SIGHUP
|
||||
└─ Other infra → full deploy
|
||||
6. Pull images (retry 3x)
|
||||
7. Remove stale containers
|
||||
8. Up services
|
||||
9. SIGHUP Traefik jika perlu
|
||||
```
|
||||
|
||||
### 4. Security Scan (`security.yml`)
|
||||
|
||||
**Trigger:**
|
||||
- PR ke `main`
|
||||
- Jadwal: Setiap Senin (`0 6 * * 1`)
|
||||
|
||||
**Aksi:**
|
||||
- Checkout dengan fetch-depth 2
|
||||
- CodeQL init untuk Rust
|
||||
- `cargo build` di `apps/scraper`
|
||||
- CodeQL analyze
|
||||
|
||||
### 5. Update Submodule Pointer (`update-submodule.yml`)
|
||||
|
||||
**Trigger:** `repository_dispatch` tipe `submodule-updated`
|
||||
|
||||
**Aksi:**
|
||||
1. Validasi payload (`service`, `sha`)
|
||||
2. Map service ke submodule path (e.g., `scraper-api` → `apps/scraper`)
|
||||
3. Update submodule ke SHA yang diberikan
|
||||
4. Commit sebagai `monrepo-bot` dengan message:
|
||||
`chore: update <service> to <shortsha>`
|
||||
5. Push dengan retry (3 attempts)
|
||||
|
||||
## Flow Submodule Update
|
||||
|
||||
Flow lengkap ketika code berubah di submodule repo:
|
||||
|
||||
```
|
||||
1. Developer push ke asepharyana-hub-scraper
|
||||
2. GitHub Action di scraper repo kirim repository_dispatch
|
||||
ke asepharyana-hub
|
||||
3. update-submodule.yml terima dispatch, update pointer
|
||||
4. Commit masuk ke hub repo main
|
||||
5. Commit ini trigger docker-build-push.yml
|
||||
(push ke main dengan path apps/scraper/**)
|
||||
6. Build image baru, update compose file
|
||||
7. Deploy ke VPS
|
||||
```
|
||||
|
||||
## Secrets yang Diperlukan
|
||||
|
||||
| Secret | Workflow | Deskripsi |
|
||||
|--------|----------|-----------|
|
||||
| `SSH_PRIVATE_KEY` | deploy-docker | SSH key untuk akses VPS |
|
||||
| `VPS_HOST` | deploy-docker | IP VPS (`45.127.35.244`) |
|
||||
| `VPS_USER` | deploy-docker | User SSH (`root`) |
|
||||
| `VPS_TARGET_DIR` | deploy-docker | Dir di VPS (`/root/asepharyana-hub`) |
|
||||
| `ENV_FILE_PRODUCTION` | deploy-docker | Full `.env` production |
|
||||
|
||||
## Menambahkan Service Baru ke Pipeline
|
||||
|
||||
Untuk menambahkan service baru, update:
|
||||
|
||||
### `docker-build-push.yml`
|
||||
|
||||
1. **Phase 1 — `changes` job:** Tambah detection logic untuk service baru:
|
||||
|
||||
```yaml
|
||||
echo "new-service=$(changed '^(apps/new-service(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/new-service\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
|
||||
```
|
||||
|
||||
2. **Phase 1 — `repository_dispatch`:** Tambah case:
|
||||
|
||||
```yaml
|
||||
case "$SERVICE" in
|
||||
scraper-api|new-service) ;;
|
||||
```
|
||||
|
||||
3. **Phase 1 — `set-matrix`:** Tambah service:
|
||||
|
||||
```bash
|
||||
if [ "${{ ...['new-service'] == 'true' ... }}" == "true" ]; then add_service "new-service" "docker-new-service" "apps/new-service"; fi
|
||||
```
|
||||
|
||||
4. **Phase 2 — `meta` step:** Tambah mapping Dockerfile:
|
||||
|
||||
```bash
|
||||
"new-service") echo "dockerfile=infra/docker/new-service.Dockerfile" >> $GITHUB_OUTPUT ;;
|
||||
```
|
||||
|
||||
5. **Phase 3 — `update-manifest`:** Tambah mapping:
|
||||
|
||||
```bash
|
||||
SERVICES["new-service"]="new-service.yml"
|
||||
PATHS["new-service"]="apps/new-service"
|
||||
```
|
||||
|
||||
### `deploy-docker.yml`
|
||||
|
||||
Tambah compose file ke `ALL_COMPOSE_FILES`:
|
||||
|
||||
```bash
|
||||
ALL_COMPOSE_FILES="infra/compose/traefik.yml infra/compose/shared.yml infra/compose/scraper.yml infra/compose/nats.yml infra/compose/dapr.yml infra/compose/new-service.yml"
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
### Rollback Image
|
||||
|
||||
```bash
|
||||
# Cari SHA tag sebelumnya di GHCR packages
|
||||
# Update compose file ke tag tersebut
|
||||
sed -i 's|sha-badcommit|sha-goodcommit|g' infra/compose/scraper.yml
|
||||
git commit -am "fix: rollback scraper-api to sha-goodcommit"
|
||||
git push
|
||||
```
|
||||
|
||||
### Rollback via Git Revert
|
||||
|
||||
```bash
|
||||
git revert HEAD
|
||||
git push origin main
|
||||
# Pipeline otomatis build dan deploy
|
||||
```
|
||||
|
||||
## Monitoring Pipeline
|
||||
|
||||
```bash
|
||||
# Cek status workflow terbaru
|
||||
gh run list --limit 5
|
||||
|
||||
# Lihat log workflow tertentu
|
||||
gh run view <run-id> --log
|
||||
|
||||
# Trigger workflow manual
|
||||
gh workflow run deploy-docker.yml
|
||||
```
|
||||
@@ -1,286 +0,0 @@
|
||||
# NATS + JetStream Guide
|
||||
|
||||
Dokumentasi konfigurasi, penggunaan, dan troubleshooting NATS di infrastruktur `asepharyana-hub`.
|
||||
|
||||
## Arsitektur
|
||||
|
||||
NATS berjalan di container `nats` dengan JetStream diaktifkan (`-js`). Data persistent disimpan di volume Docker `nats_data`.
|
||||
|
||||
```
|
||||
Service ──► NATS (port 4222) ──► JetStream (disk)
|
||||
│
|
||||
├─ Monitoring HTTP: port 8222
|
||||
└─ Client connections: port 4222
|
||||
```
|
||||
|
||||
### Hubungan dengan Dapr
|
||||
|
||||
Saat ini Dapr pub/sub menggunakan **Redis** (`pubsub.redis`), bukan NATS. NATS berfungsi sebagai message broker independen untuk:
|
||||
|
||||
- Event streaming antar service
|
||||
- Persistent job queues
|
||||
- Pub/sub untuk service yang tidak menggunakan Dapr
|
||||
|
||||
Jika ingin Dapr menggunakan NATS sebagai backend pub/sub, ganti komponen `pubsub.yaml`:
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Component
|
||||
metadata:
|
||||
name: pubsub
|
||||
spec:
|
||||
type: pubsub.nats
|
||||
version: v1
|
||||
metadata:
|
||||
- name: natsURL
|
||||
value: nats://nats:4222
|
||||
```
|
||||
|
||||
## Konfigurasi Compose
|
||||
|
||||
File: `infra/compose/nats.yml`
|
||||
|
||||
```yaml
|
||||
services:
|
||||
nats:
|
||||
container_name: nats
|
||||
image: nats:latest
|
||||
restart: always
|
||||
networks:
|
||||
app-shared-net:
|
||||
aliases:
|
||||
- nats
|
||||
ports:
|
||||
- '4222:4222' # client connections
|
||||
- '8222:8222' # HTTP monitor
|
||||
command:
|
||||
- '-js' # enable JetStream
|
||||
- '-sd'
|
||||
- '/data' # storage directory
|
||||
volumes:
|
||||
- nats_data:/data
|
||||
```
|
||||
|
||||
## CLI Tools
|
||||
|
||||
### Install NATS CLI
|
||||
|
||||
```bash
|
||||
# Linux
|
||||
curl -sf https://bin.nats.dev/nats | sh
|
||||
sudo mv nats /usr/local/bin/
|
||||
|
||||
# Atau via package manager
|
||||
# brew install nats-io/nats-tools/nats (macOS)
|
||||
```
|
||||
|
||||
### Koneksi ke NATS
|
||||
|
||||
```bash
|
||||
# Dari host (port 4222 ter-expose)
|
||||
nats context save hub --server nats://localhost:4222 --description "Hub Production"
|
||||
nats context select hub
|
||||
|
||||
# Test koneksi
|
||||
nats server check
|
||||
nats server info
|
||||
```
|
||||
|
||||
### Manage Streams (JetStream)
|
||||
|
||||
```bash
|
||||
# List semua stream
|
||||
nats stream list
|
||||
|
||||
# Lihat detail stream
|
||||
nats stream info <stream-name>
|
||||
|
||||
# Buat stream
|
||||
nats stream add <stream-name> \
|
||||
--subjects "hub.>" \
|
||||
--storage file \
|
||||
--max-msgs 1000000 \
|
||||
--max-bytes 1G \
|
||||
--retention limits
|
||||
|
||||
# Hapus stream
|
||||
nats stream rm <stream-name>
|
||||
|
||||
# Purge (hapus semua message, retain stream)
|
||||
nats stream purge <stream-name>
|
||||
```
|
||||
|
||||
### Pub/Sub
|
||||
|
||||
```bash
|
||||
# Subscribe ke subject
|
||||
nats sub "hub.>"
|
||||
nats sub "hub.image.cached"
|
||||
|
||||
# Publish message
|
||||
nats pub "hub.test" '{"message": "hello"}'
|
||||
nats pub "hub.image.cached" '{"original_url": "https://example.com/img.jpg", "cdn_url": "https://cdn.example.com/img.jpg"}'
|
||||
|
||||
# Request-reply
|
||||
nats request "hub.service.do" '{"task": "process"}'
|
||||
```
|
||||
|
||||
### Monitoring via HTTP API
|
||||
|
||||
```bash
|
||||
# Server info
|
||||
curl http://localhost:8222/
|
||||
|
||||
# JetStream info
|
||||
curl http://localhost:8222/jszetstream
|
||||
|
||||
# Stream detail
|
||||
curl http://localhost:8222/jszetstream?stream=<stream-name>
|
||||
|
||||
# Consumer info
|
||||
curl http://localhost:8222/jszetstream?stream=<stream-name>&consumer=<consumer-name>
|
||||
|
||||
# Server stats
|
||||
curl http://localhost:8222/varz
|
||||
|
||||
# Connections
|
||||
curl http://localhost:8222/connz
|
||||
```
|
||||
|
||||
## Event Topics Convention
|
||||
|
||||
Semua topik menggunakan prefix `hub.`:
|
||||
|
||||
| Subject | Payload | Deskripsi |
|
||||
|---------|---------|-----------|
|
||||
| `hub.image.cached` | `{original_url, cdn_url, source}` | Image selesai di-cache |
|
||||
| `hub.image.repaired` | `{old_url, new_url}` | CNAME image diperbaiki |
|
||||
| `hub.scrape.anime.done` | `{source, slug, duration}` | Scrape anime selesai |
|
||||
| `hub.system.alert` | `{service, level, message}` | Error/alert dari service |
|
||||
| `hub.test` | Any | Testing |
|
||||
|
||||
### Wildcard Subjects
|
||||
|
||||
NATS mendukung wildcard:
|
||||
|
||||
- `hub.>` — semua event hub (multi-level)
|
||||
- `hub.image.*` — semua event image (single-level)
|
||||
- `hub.*.done` — semua event yang selesai (single-level)
|
||||
|
||||
## JetStream Configuration
|
||||
|
||||
### Storage
|
||||
|
||||
Data JetStream disimpan di volume Docker `nats_data`.
|
||||
|
||||
Lokasi di VPS:
|
||||
```bash
|
||||
docker volume inspect nats_data
|
||||
# atau
|
||||
ls -la /var/lib/docker/volumes/nats_data/_data/
|
||||
```
|
||||
|
||||
### Memory & Limits
|
||||
|
||||
NATS tidak memiliki konfigurasi limit memori default. Untuk production, pertimbangkan:
|
||||
|
||||
```yaml
|
||||
command:
|
||||
- '-js'
|
||||
- '-sd'
|
||||
- '/data'
|
||||
- '--max_pending_size=64MB'
|
||||
- '--max_payload=1MB'
|
||||
```
|
||||
|
||||
Atau gunakan NATS configuration file:
|
||||
|
||||
```yaml
|
||||
# nats-server.conf
|
||||
jetstream:
|
||||
max_memory_store: 256MB
|
||||
max_file_store: 10GB
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Stream data tidak muncul
|
||||
|
||||
```bash
|
||||
# 1. Cek koneksi NATS
|
||||
nats server check
|
||||
|
||||
# 2. Cek apakah JetStream aktif
|
||||
curl http://localhost:8222/jszetstream
|
||||
|
||||
# 3. Cek stream dan message count
|
||||
nats stream list
|
||||
|
||||
# 4. Subscribe langsung untuk test
|
||||
nats sub ">"
|
||||
```
|
||||
|
||||
### NATS tidak bisa start
|
||||
|
||||
```bash
|
||||
# Cek log
|
||||
docker logs nats
|
||||
|
||||
# Cek apakah port 4222 sudah dipakai
|
||||
ss -tlnp | grep 4222
|
||||
|
||||
# Cek volume data korup
|
||||
docker run --rm -v nats_data:/data alpine ls -la /data
|
||||
|
||||
# Restart
|
||||
docker compose -f infra/compose/nats.yml up -d --force-recreate
|
||||
```
|
||||
|
||||
### Disk JetStream penuh
|
||||
|
||||
```bash
|
||||
# Cek ukuran volume
|
||||
docker system df -v | grep nats_data
|
||||
|
||||
# Purge stream jika perlu
|
||||
nats stream purge <stream-name>
|
||||
|
||||
# Atau hapus volume (data hilang!)
|
||||
docker compose -f infra/compose/nats.yml down
|
||||
docker volume rm nats_data
|
||||
docker compose -f infra/compose/nats.yml up -d
|
||||
```
|
||||
|
||||
### Slow consumer
|
||||
|
||||
```bash
|
||||
# Cek consumer lag
|
||||
nats stream info <stream-name>
|
||||
# Lihat fields: "Pending" dan "Acknowledgment"
|
||||
|
||||
# Lihat stats server
|
||||
curl http://localhost:8222/varz | jq '.slow_consumers'
|
||||
```
|
||||
|
||||
## Migration: Redis Pub/Sub ke NATS
|
||||
|
||||
Jika ingin migrasi dari Dapr pub/sub Redis ke NATS:
|
||||
|
||||
1. Buat stream NATS untuk topik `hub.>`
|
||||
2. Update `infra/dapr/components/pubsub.yaml` dari `pubsub.redis` ke `pubsub.nats`
|
||||
3. Deploy ulang semua service (Dapr sidecar akan reconnect)
|
||||
4. Verifikasi event flow
|
||||
|
||||
```yaml
|
||||
# infra/dapr/components/pubsub.yaml (setelah migrasi)
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Component
|
||||
metadata:
|
||||
name: pubsub
|
||||
spec:
|
||||
type: pubsub.nats
|
||||
version: v1
|
||||
metadata:
|
||||
- name: natsURL
|
||||
value: nats://nats:4222
|
||||
```
|
||||
@@ -1,60 +0,0 @@
|
||||
# Tools — Document Scanner & Media Processing Hub
|
||||
|
||||
Self-hosted, no-install document scanner dan media processing tools yang jalan di browser. Alternatif dari CamScanner, ilovepdf, compressjpeg — tanpa upload ke pihak ketiga.
|
||||
|
||||
## Visi
|
||||
|
||||
Satu platform dengan tools manipulasi file yang **beneran dipake orang setiap hari**. Semua proses di backend Rust — cepat, hemat memory, ga perlu install software.
|
||||
|
||||
## Fitur Utama
|
||||
|
||||
### Phase 1 — Document Scanner (Prioritas)
|
||||
- Foto dokumen pake HP → auto-detect tepi → lurusin (perspective correction)
|
||||
- Enhance: iluminasi merata, contrast, sharpen, B&W
|
||||
- OCR → searchable PDF (teks bisa di-copy, dicari)
|
||||
- Batch: multi-page → satu PDF
|
||||
- Fallback crop manual (kalau auto-detect gagal)
|
||||
|
||||
### Phase 2 — Image Tools
|
||||
- Compress JPEG/PNG/WebP (lossy + lossless, atur kualitas %)
|
||||
- Resize batch (atur dimensi, semua foto disamain)
|
||||
- Convert format (HEIC→JPEG, PNG→WebP, SVG→PNG)
|
||||
- Remove background (ONNX model, Rust runtime)
|
||||
|
||||
### Phase 3 — PDF Tools
|
||||
- Merge PDF (gabung file)
|
||||
- Split PDF (ekstrak halaman tertentu)
|
||||
- Images→PDF (kumpulan foto jadi 1 file)
|
||||
- PDF→Images (tiap halaman jadi gambar)
|
||||
- PDF compress (turunkin kualitas embedded images)
|
||||
|
||||
### Phase 4 — Video/Audio Tools
|
||||
- Compress video (bitrate + resolusi)
|
||||
- Extract audio (MP4→MP3)
|
||||
- Trim/crop
|
||||
- GIF maker
|
||||
- Audio convert + trim
|
||||
|
||||
## Target User
|
||||
|
||||
Orang yang:
|
||||
- Punya HP/PC, paham teknologi dasar (buka browser, upload file)
|
||||
- Butuh scan dokumen tanpa install aplikasi
|
||||
- Butuh kompres file buat kirim WA/email
|
||||
- Butuh manipulasi PDF sesekali
|
||||
- Peduli privasi — ga mau upload file ke server pihak ketiga
|
||||
|
||||
## Prinsip Desain
|
||||
|
||||
1. **Satu task selesai dalam <5 detik** — ga ada loading lama
|
||||
2. **Drag & drop + preview** — liat hasil sebelum download
|
||||
3. **Progress realtime** via WebSocket — tau lagi di tahap mana
|
||||
4. **Batch processing** — banyak file, satu klik
|
||||
5. **Privasi first** — file otomatis dihapus setelah 1 jam
|
||||
6. **WASM fallback** — tools ringan jalan di client (tanpa upload)
|
||||
|
||||
## Domain & Branding
|
||||
|
||||
- **Domain**: `tools.asepharyana.my.id` | `tools.asepharyana.web.id`
|
||||
- **Design**: Twilight Terminal theme (sama kaya portfolio), konsisten visual
|
||||
- **Dashboard**: Link dari hub dashboard → tools stats (total files processed, storage used)
|
||||
@@ -1,453 +0,0 @@
|
||||
# Architecture
|
||||
|
||||
> **LEGACY (2026-08-02):** Dokumen plan ini ditulis saat infra masih Docker/Traefik. Produksi sekarang Caddy + Nix/systemd dengan port 4000-an. Gunakan hanya sebagai referensi historis.
|
||||
|
||||
## System Overview
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ BROWSER │
|
||||
│ ┌────────────┐ ┌────────────┐ ┌────────────────────────┐ │
|
||||
│ │ Upload │ │ Camera │ │ Preview + Download │ │
|
||||
│ │ (drag/drop)│ │ (PWA) │ │ (streaming) │ │
|
||||
│ └─────┬──────┘ └─────┬──────┘ └───────────┬────────────┘ │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌──────────────────────────────────────────────────────┐ │
|
||||
│ │ WebSocket (progress: processing/step/percentage) │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
└──────────────────────────┬───────────────────────────────────┘
|
||||
│ HTTPS / WSS
|
||||
▼
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ TRAEFIK (tools.asepharyana.my.id) │
|
||||
│ Middleware chain: secure-headers → compress → rate-limit │
|
||||
└──────────────────────────┬────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ tools-app (Next.js 16 / TypeScript) │
|
||||
│ │
|
||||
│ ┌──────────────────┐ ┌─────────────────┐ │
|
||||
│ │ Pages/Routes │ │ API Routes │ │
|
||||
│ │ / → home │ │ POST /api/upload ──▶ file │
|
||||
│ │ /scan → scanner │ │ GET /api/job/:id ─▶ status │
|
||||
│ │ /image → image │ │ WS /api/job/:id/ws ─▶ progress │
|
||||
│ │ /pdf → pdf tools │ │ GET /api/download/:id ─▶ file │
|
||||
│ └──────────────────┘ └─────────────────┘ │
|
||||
│ │
|
||||
│ Upload validation: MIME type, size limit (50MB), virus scan │
|
||||
│ Temp storage bridge ke worker via HTTP/NATS │
|
||||
└──────────────────┬────────────────────────────────────────────┘
|
||||
│ HTTP (internal)
|
||||
▼
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ API GATEWAY (Rust / Axum) │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────┐ │
|
||||
│ │ Upload │ │ Job Manager │ │ Download │ │
|
||||
│ │ (streaming │ │ (CRUD job │ │ (stream file, │ │
|
||||
│ │ chunked) │ │ status) │ │ auto-delete) │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └────────────────────┘ │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ │
|
||||
│ ┌────────────────────────────────────────────────────┐ │
|
||||
│ │ NATS JetStream │ │
|
||||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
|
||||
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
|
||||
│ │ │ jobs │ │ jobs │ │ jobs │ │ │
|
||||
│ │ └────┬─────┘ └────┬─────┘ └──────┬───────┘ │ │
|
||||
│ │ │ │ │ │ │
|
||||
│ │ ▼ ▼ ▼ │ │
|
||||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
|
||||
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
|
||||
│ │ │ progress │ │ progress │ │ progress │ │ │
|
||||
│ │ └──────────┘ └──────────┘ └──────────────┘ │ │
|
||||
│ └────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────────────────┐ │
|
||||
│ │ Cache (Redis) │ │
|
||||
│ │ - Job metadata (status, progress, timestamps) │ │
|
||||
│ │ - Rate limiting (sliding window per IP/tool) │ │
|
||||
│ │ - Result metadata (file path, size, type) │ │
|
||||
│ └────────────────────────────────────────────────────┘ │
|
||||
└──────────────────┬────────────────────────────────────────────┘
|
||||
│ consume NATS queue
|
||||
▼
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ WORKER POOL (Rust / Tokio + Rayon) │
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
|
||||
│ │ Scan Worker │ │ Image Worker │ │ PDF Worker │ │
|
||||
│ │ ×4 instances │ │ ×2 instances │ │ ×2 instances│ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ │ 1. Load image │ │ 1. Load image │ │ 1. Load PDF │ │
|
||||
│ │ 2. Edge detect │ │ 2. Compress │ │ 2. Merge/ │ │
|
||||
│ │ 3. Warp │ │ /resize/ │ │ split │ │
|
||||
│ │ 4. Enhance │ │ convert │ │ 3. Save │ │
|
||||
│ │ 5. OCR │ │ 3. Save │ │ 4. Update │ │
|
||||
│ │ 6. Gen PDF │ │ 4. Update │ │ job │ │
|
||||
│ │ 7. Update job │ │ job status │ │ status │ │
|
||||
│ │ └───────────────┘ └─────────────────┘ └──────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────────────────────────────┐ │
|
||||
│ │ Temp Storage (filesystem volume / S3-compatible) │ │
|
||||
│ │ Auto-cleanup: job TTL 1 jam, NATS cron tiap 10m │ │
|
||||
│ └────────────────────────────────────────────────────┘ │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Component Diagram
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────┐
|
||||
│ apps/tools │
|
||||
│ │
|
||||
│ ├── frontend/ │
|
||||
│ │ ├── pages/ ← Next.js pages │
|
||||
│ │ ├── components/ ← React components │
|
||||
│ │ ├── lib/ ← utilities │
|
||||
│ │ └── public/ ← static assets │
|
||||
│ │ │
|
||||
│ ├── backend/ ← Rust workspace │
|
||||
│ │ ├── gateway/ ← Axum API server │
|
||||
│ │ ├── workers/ ← Processing workers │
|
||||
│ │ │ ├── scanner/ ← Document scanner │
|
||||
│ │ │ ├── image/ ← Image tools │
|
||||
│ │ │ └── pdf/ ← PDF tools │
|
||||
│ │ └── common/ ← Shared libs │
|
||||
│ │ │
|
||||
│ └── Dockerfile │
|
||||
└────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow (Document Scanner — Flow Lengkap)
|
||||
|
||||
```
|
||||
1. User buka tools.asepharyana.my.id/scan
|
||||
2. Upload foto via drag-drop atau kamera HP (PWA)
|
||||
3. Next.js route handler menerima file
|
||||
├─ Validasi: MIME type (image/*), max 50MB, virus header scan
|
||||
└─ Upload chunked ke Gateway internal (HTTP POST)
|
||||
|
||||
4. Gateway menerima stream:
|
||||
├─ Simpan ke temp storage
|
||||
├─ Buat job record di Redis: {id, tool: "scan", status: "queued", progress: 0}
|
||||
└─ Publish ke NATS: tools.scan.jobs {job_id, file_path, options}
|
||||
|
||||
5. Scan Worker consume dari NATS:
|
||||
├─ Update Redis: status = "processing", progress = 10
|
||||
├─ Load image (image-rs)
|
||||
├─ Pipeline (detail di pipeline.md):
|
||||
│ 1. Edge detection ──▶ progress 25
|
||||
│ 2. Perspective warp ──▶ progress 40
|
||||
│ 3. Shadow removal ──▶ progress 55
|
||||
│ 4. Binarization ──▶ progress 70
|
||||
│ 5. Contrast/sharpen ──▶ progress 80
|
||||
│ 6. OCR ──▶ progress 90
|
||||
│ 7. Generate PDF ──▶ progress 95
|
||||
├─ Simpan file hasil ke temp storage
|
||||
├─ Update Redis: status = "completed", progress = 100, result_path, ocr_text
|
||||
└─ Publish ke NATS: tools.scan.progress {job_id, status, progress}
|
||||
|
||||
6. WebSocket handler di Gateway:
|
||||
├─ Subscribe NATS topics tools.scan.progress
|
||||
├─ Forward ke browser user (per-job-id filter)
|
||||
└─ Browser update progress bar + preview
|
||||
|
||||
7. User download PDF:
|
||||
├─ GET /api/download/:job_id
|
||||
├─ Gateway stream file dari temp storage
|
||||
└─ Browser save file
|
||||
```
|
||||
|
||||
## Tech Stack
|
||||
|
||||
### Frontend (Next.js + TypeScript)
|
||||
|
||||
| Library | Fungsi |
|
||||
|---------|--------|
|
||||
| Next.js 16 | App router, API routes |
|
||||
| shadcn/ui + Tailwind v4 | UI components |
|
||||
| Framer Motion | Animasi progress, transisi |
|
||||
| Canvas API | Preview crop manual, image manipulation client-side |
|
||||
| WebSocket API | Real-time progress |
|
||||
|
||||
### Backend (Rust)
|
||||
|
||||
| Crate | Fungsi |
|
||||
|-------|--------|
|
||||
| `axum` | HTTP server (Gateway) |
|
||||
| `tokio` | Async runtime |
|
||||
| `image` | Image I/O, resize, convert, compress |
|
||||
| `imageproc` | Edge detection, contour, thresholding |
|
||||
| `lopdf` | PDF generation, merge, split, compress |
|
||||
| `leptess` | Tesseract OCR binding |
|
||||
| `ort` | ONNX Runtime (background removal) |
|
||||
| `async-nats` | NATS JetStream client |
|
||||
| `deadpool-redis` | Redis connection pool |
|
||||
| `redis` | Redis async client |
|
||||
| `rayon` | Parallel processing (batch, pixel ops) |
|
||||
| `serde` | Serialization |
|
||||
| `tracing` + `opentelemetry` | Observability |
|
||||
| `uuid` | Job ID generation |
|
||||
|
||||
### Infrastructure
|
||||
|
||||
| Komponen | Fungsi |
|
||||
|----------|--------|
|
||||
| NATS JetStream | Job queue, progress pub/sub, scheduler |
|
||||
| Redis | Job metadata, rate limiting, cache |
|
||||
| PostgreSQL | Opsional — audit log, usage statistics |
|
||||
| Tesseract | OCR engine (data files di Docker image) |
|
||||
| Prometheus | Metrics (jobs/min, queue depth, latency per stage) |
|
||||
|
||||
## Job Queue (NATS Streams & Consumers)
|
||||
|
||||
### Streams
|
||||
|
||||
```
|
||||
tools-scan-jobs → 1 stream, mirror to all scan workers
|
||||
tools-image-jobs → 1 stream, mirror to all image workers
|
||||
tools-pdf-jobs → 1 stream, mirror to all pdf workers
|
||||
tools-progress → 1 stream, all progress events (key-value by job_id)
|
||||
tools-scheduler → 1 stream, cron events
|
||||
```
|
||||
|
||||
### Subjects
|
||||
|
||||
```
|
||||
tools.scan.jobs.{job_id} → job submission
|
||||
tools.scan.progress.{job_id} → progress update (fan-out ke Gateway)
|
||||
tools.image.jobs.{job_id} → job submission
|
||||
tools.image.progress.{job_id} → progress update
|
||||
tools.pdf.jobs.{job_id} → job submission
|
||||
tools.pdf.progress.{job_id} → progress update
|
||||
tools.scheduler.cleanup → cleanup expired files (every 10 min)
|
||||
```
|
||||
|
||||
## Redis Schema
|
||||
|
||||
```
|
||||
job:{id} → Hash {status, tool, progress, file_path, result_path, ocr_text, created_at, ttl}
|
||||
rate_limit:{ip}:{tool} → Sorted Set (sliding window)
|
||||
file_meta:{hash} → String {original_name, size, mime}
|
||||
```
|
||||
|
||||
## Metrics (Prometheus)
|
||||
|
||||
| Metric | Type | Labels | Description |
|
||||
|--------|------|--------|-------------|
|
||||
| `tools_jobs_total` | Counter | `tool`, `status` | Total jobs processed |
|
||||
| `tools_jobs_in_flight` | Gauge | `tool` | Currently processing jobs |
|
||||
| `tools_queue_depth` | Gauge | `tool` | NATS queue depth |
|
||||
| `tools_processing_duration` | Histogram | `tool`, `stage` | Duration per stage |
|
||||
| `tools_file_size_bytes` | Histogram | `tool` | Upload file size distribution |
|
||||
| `tools_rate_limit_hits` | Counter | `tool` | Rate limit violations |
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
apps/tools/
|
||||
├── frontend/
|
||||
│ ├── src/
|
||||
│ │ ├── app/
|
||||
│ │ │ ├── page.tsx # Landing page
|
||||
│ │ │ ├── scan/
|
||||
│ │ │ │ ├── page.tsx # Scanner page
|
||||
│ │ │ │ └── result/[id]/
|
||||
│ │ │ │ └── page.tsx # Result page
|
||||
│ │ │ ├── image/
|
||||
│ │ │ │ ├── compress/page.tsx
|
||||
│ │ │ │ ├── resize/page.tsx
|
||||
│ │ │ │ ├── convert/page.tsx
|
||||
│ │ │ │ └── remove-bg/page.tsx
|
||||
│ │ │ ├── pdf/
|
||||
│ │ │ │ ├── merge/page.tsx
|
||||
│ │ │ │ ├── split/page.tsx
|
||||
│ │ │ │ ├── images-to-pdf/page.tsx
|
||||
│ │ │ │ └── compress/page.tsx
|
||||
│ │ │ ├── api/
|
||||
│ │ │ │ ├── upload/route.ts
|
||||
│ │ │ │ ├── job/[id]/route.ts
|
||||
│ │ │ │ │ └── ws/route.ts
|
||||
│ │ │ │ └── download/[id]/route.ts
|
||||
│ │ │ ├── layout.tsx
|
||||
│ │ │ └── globals.css
|
||||
│ │ ├── components/
|
||||
│ │ │ ├── upload-zone.tsx # Drag & drop area
|
||||
│ │ │ ├── progress-bar.tsx # WebSocket-connected progress
|
||||
│ │ │ ├── preview.tsx # Before/after preview
|
||||
│ │ │ ├── crop-editor.tsx # Manual corner adjustment
|
||||
│ │ │ ├── tool-layout.tsx # Consistent tool page layout
|
||||
│ │ │ └── camera-capture.tsx # PWA camera interface
|
||||
│ │ ├── hooks/
|
||||
│ │ │ ├── use-job-status.ts # WebSocket connection
|
||||
│ │ │ ├── use-upload.ts # Upload with progress
|
||||
│ │ │ └── use-camera.ts # Camera access
|
||||
│ │ └── lib/
|
||||
│ │ ├── utils.ts
|
||||
│ │ └── types.ts
|
||||
│ ├── next.config.ts
|
||||
│ ├── package.json
|
||||
│ └── tsconfig.json
|
||||
│
|
||||
├── backend/
|
||||
│ ├── Cargo.toml
|
||||
│ ├── gateway/
|
||||
│ │ ├── Cargo.toml
|
||||
│ │ └── src/
|
||||
│ │ ├── main.rs
|
||||
│ │ ├── routes/
|
||||
│ │ │ ├── mod.rs
|
||||
│ │ │ ├── upload.rs
|
||||
│ │ │ ├── job.rs
|
||||
│ │ │ ├── download.rs
|
||||
│ │ │ └── ws.rs
|
||||
│ │ ├── nats/
|
||||
│ │ │ ├── mod.rs
|
||||
│ │ │ └── publisher.rs
|
||||
│ │ ├── redis/
|
||||
│ │ │ ├── mod.rs
|
||||
│ │ │ ├── job.rs
|
||||
│ │ │ └── ratelimit.rs
|
||||
│ │ ├── metrics.rs
|
||||
│ │ └── config.rs
|
||||
│ │
|
||||
│ ├── workers/
|
||||
│ │ ├── Cargo.toml
|
||||
│ │ └── src/
|
||||
│ │ ├── main.rs
|
||||
│ │ ├── scanner/
|
||||
│ │ │ ├── mod.rs
|
||||
│ │ │ ├── pipeline.rs
|
||||
│ │ │ ├── edge.rs # Edge detection
|
||||
│ │ │ ├── warp.rs # Perspective correction
|
||||
│ │ │ ├── enhance.rs # Shadow removal, B&W, contrast
|
||||
│ │ │ ├── ocr.rs # Tesseract wrapper
|
||||
│ │ │ └── pdf.rs # Generate searchable PDF
|
||||
│ │ ├── image/
|
||||
│ │ │ ├── mod.rs
|
||||
│ │ │ ├── compress.rs
|
||||
│ │ │ ├── resize.rs
|
||||
│ │ │ ├── convert.rs
|
||||
│ │ │ └── remove_bg.rs
|
||||
│ │ ├── pdf/
|
||||
│ │ │ ├── mod.rs
|
||||
│ │ │ ├── merge.rs
|
||||
│ │ │ ├── split.rs
|
||||
│ │ │ ├── extract.rs
|
||||
│ │ │ └── compress.rs
|
||||
│ │ ├── nats/
|
||||
│ │ │ ├── mod.rs
|
||||
│ │ │ └── consumer.rs
|
||||
│ │ └── config.rs
|
||||
│ │
|
||||
│ └── common/
|
||||
│ ├── Cargo.toml
|
||||
│ └── src/
|
||||
│ ├── lib.rs
|
||||
│ ├── types.rs # Shared types (JobStatus, Job, etc.)
|
||||
│ ├── error.rs # Error types
|
||||
│ └── nats.rs # NATS subject constants
|
||||
│
|
||||
├── Dockerfile
|
||||
├── compose.yml # Local dev compose
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## API Design
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `POST` | `/api/upload` | Upload file, create job |
|
||||
| `GET` | `/api/job/:id` | Get job status + result metadata |
|
||||
| `WS` | `/api/job/:id/ws` | WebSocket — realtime progress |
|
||||
| `GET` | `/api/download/:id` | Download result file |
|
||||
| `DELETE` | `/api/job/:id` | Cancel job, delete files |
|
||||
| `GET` | `/health` | Health check |
|
||||
|
||||
### Upload Request
|
||||
|
||||
```
|
||||
POST /api/upload
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
{
|
||||
file: <binary>,
|
||||
tool: "scan" | "image-compress" | "image-resize" | "image-convert" | "remove-bg" |
|
||||
"pdf-merge" | "pdf-split" | "images-to-pdf" | "pdf-compress",
|
||||
options?: { // tool-specific options
|
||||
quality?: 80, // compress quality
|
||||
width?: 1920, // resize width
|
||||
format?: "webp", // convert format
|
||||
pages?: "1,3-5", // PDF split pages
|
||||
dpi?: 300, // scan DPI
|
||||
enhance?: true, // scan auto-enhance
|
||||
ocr?: true // scan OCR
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Response (202 Accepted)
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "uuid",
|
||||
"status": "queued",
|
||||
"tool": "scan",
|
||||
"ws_url": "/api/job/uuid/ws",
|
||||
"created_at": "2026-07-24T10:00:00Z",
|
||||
"estimated_seconds": 5
|
||||
}
|
||||
```
|
||||
|
||||
### WebSocket Messages
|
||||
|
||||
```json
|
||||
// Server → Client
|
||||
{
|
||||
"type": "progress",
|
||||
"job_id": "uuid",
|
||||
"status": "processing",
|
||||
"progress": 45,
|
||||
"stage": "warp",
|
||||
"message": "Meluruskan perspektif dokumen..."
|
||||
}
|
||||
|
||||
{
|
||||
"type": "complete",
|
||||
"job_id": "uuid",
|
||||
"status": "completed",
|
||||
"progress": 100,
|
||||
"result": {
|
||||
"download_url": "/api/download/uuid",
|
||||
"file_name": "scan_20260724.pdf",
|
||||
"file_size": 1245678,
|
||||
"pages": 1,
|
||||
"ocr_text": "Nama: Asep...",
|
||||
"preview_url": "/api/job/uuid/preview"
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
"type": "error",
|
||||
"job_id": "uuid",
|
||||
"status": "failed",
|
||||
"error": "Edge detection failed: cannot find document boundary"
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with Existing Portfolio
|
||||
|
||||
| Area | Detail |
|
||||
|------|--------|
|
||||
| **Domain** | `tools.asepharyana.my.id` — tambah entry di `infra/traefik/dynamic/apps.yaml` |
|
||||
| **Dashboard** | Link ke tools stats di dashboard hub yang sudah ada |
|
||||
| **Docker Compose** | `infra/compose/tools.yml` — pola sama kaya `hub.yml` |
|
||||
| **CI/CD** | Tambah service `tools` di `docker-build-push.yml` |
|
||||
| **Style** | Ulang Twilight Terminal theme dari hub, konsisten visual branding |
|
||||
| **Monitoring** | Reuse existing Prometheus + Grafana, tambah metrics tools |
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,429 +0,0 @@
|
||||
# Infrastructure & Deployment
|
||||
|
||||
> **LEGACY (2026-08-02):** Dokumen plan ini ditulis saat infra masih Docker/Traefik. Produksi sekarang Caddy + Nix/systemd dengan port 4000-an. Gunakan hanya sebagai referensi historis.
|
||||
|
||||
## Docker Image Architecture
|
||||
|
||||
Project ini punya **satu Docker image** dengan multi-stage build. Backend Rust + Tesseract + ONNX model plus frontend Next.js.
|
||||
|
||||
### Dockerfile Structure
|
||||
|
||||
```dockerfile
|
||||
# ============================================================
|
||||
# Stage 1: Build Rust Backend
|
||||
# ============================================================
|
||||
FROM rust:1.85-slim-bookworm AS chef
|
||||
RUN cargo install cargo-chef
|
||||
WORKDIR /app
|
||||
|
||||
FROM chef AS planner
|
||||
COPY backend/ .
|
||||
RUN cargo chef prepare --recipe-path recipe.json
|
||||
|
||||
FROM chef AS builder
|
||||
COPY --from=planner /app/recipe.json recipe.json
|
||||
RUN cargo chef cook --release --recipe-path recipe.json
|
||||
|
||||
COPY backend/ .
|
||||
RUN cargo build --release --bin gateway --bin workers
|
||||
|
||||
# ============================================================
|
||||
# Stage 2: Build Next.js Frontend
|
||||
# ============================================================
|
||||
FROM oven/bun:1.3 AS frontend-builder
|
||||
WORKDIR /app
|
||||
COPY frontend/package.json frontend/bun.lock ./
|
||||
RUN bun install --frozen-lockfile
|
||||
COPY frontend/ .
|
||||
RUN bun run build
|
||||
|
||||
# ============================================================
|
||||
# Stage 3: Production Runtime
|
||||
# ============================================================
|
||||
FROM debian:bookworm-slim AS runtime
|
||||
|
||||
# Install runtime dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
tesseract-ocr \
|
||||
tesseract-ocr-eng \
|
||||
tesseract-ocr-ind \
|
||||
ca-certificates \
|
||||
fonts-dejavu-core \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy Rust binaries
|
||||
COPY --from=builder /app/target/release/gateway /app/gateway
|
||||
COPY --from=builder /app/target/release/workers /app/workers
|
||||
|
||||
# Copy Next.js build
|
||||
COPY --from=frontend-builder /app/.next /app/.next
|
||||
COPY --from=frontend-builder /app/public /app/public
|
||||
COPY --from=frontend-builder /app/package.json /app/package.json
|
||||
COPY --from=frontend-builder /app/node_modules /app/node_modules
|
||||
|
||||
# Copy ONNX model (for background removal)
|
||||
COPY models/ /app/models/
|
||||
|
||||
# Create temp storage directory
|
||||
RUN mkdir -p /data/tools && chmod 1777 /data/tools
|
||||
|
||||
# Environment
|
||||
ENV TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata
|
||||
ENV TOOLS_STORAGE_PATH=/data/tools
|
||||
ENV TOOLS_GATEWAY_PORT=3001
|
||||
ENV TOOLS_WORKER_CONCURRENCY=4
|
||||
ENV RUST_LOG=info
|
||||
|
||||
# Expose port
|
||||
EXPOSE 3001
|
||||
|
||||
# Run both gateway and workers via supervisor script
|
||||
COPY scripts/entrypoint.sh /app/entrypoint.sh
|
||||
RUN chmod +x /app/entrypoint.sh
|
||||
|
||||
CMD ["/app/entrypoint.sh"]
|
||||
```
|
||||
|
||||
### Entrypoint Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Start Gateway (Axum HTTP server)
|
||||
/app/gateway &
|
||||
GATEWAY_PID=$!
|
||||
|
||||
# Start Worker(s)
|
||||
/app/workers &
|
||||
WORKER_PID=$!
|
||||
|
||||
# Handle graceful shutdown
|
||||
trap "kill $GATEWAY_PID $WORKER_PID; exit 0" SIGINT SIGTERM
|
||||
|
||||
# Wait for either process to exit
|
||||
wait -n $GATEWAY_PID $WORKER_PID
|
||||
|
||||
# If one exits, kill the other
|
||||
kill $GATEWAY_PID $WORKER_PID 2>/dev/null
|
||||
exit 1
|
||||
```
|
||||
|
||||
### Image Size Estimates
|
||||
|
||||
| Component | Size |
|
||||
|-----------|------|
|
||||
| Rust binary (gateway) | ~8 MB |
|
||||
| Rust binary (workers) | ~15 MB |
|
||||
| Next.js build | ~10 MB |
|
||||
| Tesseract + data | ~25 MB |
|
||||
| ONNX model | ~50 MB |
|
||||
| Base (Debian slim) | ~80 MB |
|
||||
| **Total** | **~188 MB** |
|
||||
|
||||
> ONNX model opsional — bisa di-download runtime daripada di-include di image.
|
||||
|
||||
---
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
# infra/compose/tools.yml
|
||||
services:
|
||||
tools:
|
||||
container_name: tools
|
||||
image: ghcr.io/asepharyana/asepharyana-hub/tools:sha-xxxxxxx
|
||||
restart: always
|
||||
networks:
|
||||
app-shared-net:
|
||||
aliases:
|
||||
- tools
|
||||
env_file:
|
||||
- ../../.env
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- NATS_URL=nats://nats:4222
|
||||
- TOOLS_STORAGE_PATH=/data/tools
|
||||
- TOOLS_GATEWAY_PORT=3001
|
||||
- TOOLS_WORKER_CONCURRENCY=4
|
||||
- RUST_LOG=info
|
||||
volumes:
|
||||
- tools_data:/data/tools
|
||||
ports:
|
||||
- "3001:3001"
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_started
|
||||
nats:
|
||||
condition: service_started
|
||||
|
||||
volumes:
|
||||
tools_data:
|
||||
|
||||
networks:
|
||||
app-shared-net:
|
||||
name: app-shared-net
|
||||
external: true
|
||||
```
|
||||
|
||||
### Environment Variables (`../../.env`)
|
||||
|
||||
```bash
|
||||
# Tools
|
||||
TOOLS_GATEWAY_PORT=3001
|
||||
TOOLS_WORKER_CONCURRENCY=4
|
||||
TOOLS_STORAGE_PATH=/data/tools
|
||||
TOOLS_JOB_TTL_SECONDS=3600
|
||||
TOOLS_RATE_LIMIT_PER_MINUTE=30
|
||||
TOOLS_MAX_FILE_SIZE_MB=50
|
||||
TOOLS_OCR_LANG=eng+ind
|
||||
|
||||
# Infra (reuse existing)
|
||||
REDIS_URL=redis://redis:6379
|
||||
NATS_URL=nats://nats:4222
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### Docker Build Workflow
|
||||
|
||||
Tambah service `tools` di `.github/workflows/docker-build-push.yml`:
|
||||
|
||||
```yaml
|
||||
# Di job "changes" step "Detect changed services"
|
||||
changed() {
|
||||
printf '%s\n' "$CHANGED_FILES" | grep -Eq "$1" && echo true || echo false
|
||||
}
|
||||
echo "tools=$(changed '^(apps/tools(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/tools\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Di job "build" step "Set matrix"
|
||||
if [ "${{ steps.filter.outputs['tools'] == 'true' || steps.dispatch.outputs['tools'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then
|
||||
add_service "tools" "docker-tools" "apps/tools"
|
||||
fi
|
||||
|
||||
# Di job "build" step "Docker metadata"
|
||||
case "$SVC_NAME" in
|
||||
"tools") echo "dockerfile=infra/docker/tools.Dockerfile" >> $GITHUB_OUTPUT ;;
|
||||
esac
|
||||
|
||||
# Di job "update-manifest"
|
||||
SERVICES["tools"]="tools.yml"
|
||||
PATHS["tools"]="apps/tools"
|
||||
```
|
||||
|
||||
### Deploy Workflow
|
||||
|
||||
Tambah di `.github/workflows/deploy-docker.yml`:
|
||||
```yaml
|
||||
# Tidak perlu perubahan — deploy-docker.yml auto-detect compose file changes.
|
||||
# Kalau compose/tools.yml berubah, service tools akan di-restart.
|
||||
```
|
||||
|
||||
### Service Registration (update infra/traefik/dynamic/apps.yaml)
|
||||
|
||||
```yaml
|
||||
tools:
|
||||
rule: 'Host(`tools.asepharyana.my.id`) || Host(`tools.asepharyana.web.id`)'
|
||||
entryPoints:
|
||||
- websecure
|
||||
tls: {}
|
||||
middlewares:
|
||||
- common-chain@file
|
||||
service: tools-service
|
||||
|
||||
# ...di bagian services:
|
||||
tools-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: 'http://tools:3001'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
Tambahkan label Prometheus ke container tools:
|
||||
|
||||
```yaml
|
||||
# Di compose tools.yml
|
||||
labels:
|
||||
- 'prometheus.io/scrape=true'
|
||||
- 'prometheus.io/port=3001'
|
||||
- 'prometheus.io/path=/metrics'
|
||||
```
|
||||
|
||||
### Dashboard Integration
|
||||
|
||||
Tambah card di dashboard hub yang sudah ada:
|
||||
|
||||
```tsx
|
||||
// Di dashboard hub — tambah section "Tools Usage"
|
||||
// Data dari /api/dashboard → Prometheus query:
|
||||
// rate(tools_jobs_total[24h]) — jobs per tool per hari
|
||||
// sum(increase(tools_jobs_total[7d])) — total jobs minggu ini
|
||||
// tools_jobs_in_flight — current processing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
### Temp Storage
|
||||
|
||||
```
|
||||
/data/tools/
|
||||
├── upload/ # Uploaded files
|
||||
│ └── {job_id}.{ext}
|
||||
├── processing/ # Intermediate files (stage-by-stage)
|
||||
│ └── {job_id}/
|
||||
│ ├── 00_original.png
|
||||
│ ├── 01_grayscale.png
|
||||
│ ├── 02_edges.png
|
||||
│ ├── 03_warped.png
|
||||
│ └── ...
|
||||
└── output/ # Final output
|
||||
└── {job_id}.pdf
|
||||
```
|
||||
|
||||
### Cleanup Strategy
|
||||
|
||||
| Mekanisme | Timing |
|
||||
|-----------|--------|
|
||||
| NATS cron job | Setiap 10 menit |
|
||||
| Scan files >1 jam | `find /data/tools -mmin +60 -delete` |
|
||||
| Redis job keys >1 jam | `SCAN 0 MATCH job:*` → TTL check → DEL |
|
||||
| Storage low warning | Alert via Notification Hub (future) |
|
||||
|
||||
---
|
||||
|
||||
## Resource Estimation (VPS orangevps)
|
||||
|
||||
### Current Usage
|
||||
|
||||
| Service | CPU | RAM | Disk |
|
||||
|---------|-----|-----|------|
|
||||
| Traefik | 0.1 | 50 MB | 10 MB |
|
||||
| NATS | 0.05 | 30 MB | 10 MB |
|
||||
| Redis | 0.05 | 10 MB | 5 MB |
|
||||
| Dapr Placement | 0.02 | 20 MB | 5 MB |
|
||||
| Scraper API | 0.1 | 30 MB | 50 MB |
|
||||
| Hub | 0.05 | 120 MB | 200 MB |
|
||||
| Jaeger | 0.1 | 200 MB | 500 MB |
|
||||
| Prometheus | 0.1 | 150 MB | 1 GB |
|
||||
| Node Exporter | 0.02 | 10 MB | 5 MB |
|
||||
| OTel Collector | 0.05 | 50 MB | 10 MB |
|
||||
| **Total Current** | **~0.64** | **~670 MB** | **~1.8 GB** |
|
||||
|
||||
### Tools Addition
|
||||
|
||||
| Resources | Estimate | Notes |
|
||||
|-----------|----------|-------|
|
||||
| CPU | +1.0 core (burst) | Pipeline processing berat di CPU. Scoring, warp, OCR semua CPU-bound. |
|
||||
| RAM | +300 MB | Rust binary + image processing buffers + Tesseract + ONNX |
|
||||
| Disk | +5 GB | Temp files, bisa lebih untuk batch processing. Butuh auto-cleanup ketat. |
|
||||
| **Total After** | **~1.64 cores** | **~970 MB RAM** | **~6.8 GB disk** |
|
||||
|
||||
> **Catatan**: Kalau VPS cuma punya 1-2 cores, processing akan antri. NATS queue handle ini. Untuk production, pastikan CPU ada >2 cores.
|
||||
|
||||
### Scalability
|
||||
|
||||
```
|
||||
VPS 1 core:
|
||||
- Scanner: ~5-8 detik per page
|
||||
- Concurrent: 1 job at a time
|
||||
- Antrian: NATS queue buffer unlimited
|
||||
|
||||
VPS 4+ core:
|
||||
- Scanner: ~2-3 detik per page
|
||||
- Concurrent: 4 jobs parallel (1 per worker)
|
||||
- Rayon: parallel per-page dalam batch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
| Area | Mitigation |
|
||||
|------|-----------|
|
||||
| **Upload validation** | MIME type check (whitelist), magic bytes verification, max size 50MB |
|
||||
| **Path traversal** | Job ID = UUID v4, no user-controlled filenames in storage |
|
||||
| **Command injection** | No shell commands — semua processing via Rust crates, FFmpeg via crate binding |
|
||||
| **Temporary files** | Auto-cleanup, random filenames, restricted permissions (0600) |
|
||||
| **Rate limiting** | Redis sliding window: 30 requests/min/IP per tool, 429 response |
|
||||
| **CORS** | Origin terbatas ke domain portfolio |
|
||||
| **Resource exhaustion** | Max image dimension 8000px, max file count per batch 50, worker concurrency limit |
|
||||
| **OCR data** | Tesseract data dari package manager, no user-trained models |
|
||||
| **ONNX model** | Model dari source terpercaya, verify checksum |
|
||||
|
||||
---
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
1. **Image tag**: `tools:sha-<short>` immutable — tinggal update compose file ke tag sebelumnya
|
||||
2. **Data**: Files auto-expire dalam 1 jam — no persistent data migration needed
|
||||
3. **Traefik**: Cukup restart, TLS certs ga berubah
|
||||
4. **Monitor**: Prometheus metrics akan langsung show error rate spike
|
||||
|
||||
---
|
||||
|
||||
## Development Setup (Local)
|
||||
|
||||
Untuk development tanpa Docker:
|
||||
|
||||
```bash
|
||||
# Terminal 1: Redis + NATS
|
||||
docker compose -f infra/compose/shared.yml -f infra/compose/nats.yml up -d
|
||||
|
||||
# Terminal 2: Rust workers
|
||||
cd apps/tools/backend
|
||||
REDIS_URL=redis://localhost:6379 NATS_URL=nats://localhost:4222 \
|
||||
cargo run --bin workers
|
||||
|
||||
# Terminal 3: Rust gateway
|
||||
REDIS_URL=redis://localhost:6379 NATS_URL=nats://localhost:4222 \
|
||||
TOOLS_STORAGE_PATH=/tmp/tools \
|
||||
cargo run --bin gateway
|
||||
|
||||
# Terminal 4: Next.js
|
||||
cd apps/tools/frontend
|
||||
bun dev --port 3002
|
||||
```
|
||||
|
||||
### Test Pipeline Locally (tanpa NATS/Redis)
|
||||
|
||||
Untuk development pipeline image processing doang:
|
||||
|
||||
```rust
|
||||
// Di workers/src/scanner/pipeline.rs — test function
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_full_pipeline() {
|
||||
let pipeline = ScanPipeline::default();
|
||||
let result = pipeline.process_sync(
|
||||
"test_images/scan_miring.jpg",
|
||||
ScanOptions { ocr: false, enhance: true }
|
||||
);
|
||||
assert!(result.is_ok());
|
||||
assert!(result.unwrap().output_path.exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_edge_detection_variations() {
|
||||
// Test dengan berbagai kondisi: kertas putih, background ramai, sudut ekstrim
|
||||
for case in &["normal.jpg", "dark.jpg", "angle45.jpg", "shadow.jpg"] {
|
||||
let img = image::open(format!("test_images/{}", case)).unwrap();
|
||||
let corners = detect_corners_with_fallback(&img.grayscale().into_luma8());
|
||||
assert!(corners.is_ok(), "Failed on: {}", case);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Test images kumpulin dari foto dokumen real di berbagai kondisi — ini penting buat tuning parameter.
|
||||
@@ -1,800 +0,0 @@
|
||||
# Document Scanner — Processing Pipeline
|
||||
|
||||
> **LEGACY (2026-08-02):** Dokumen plan ini ditulis saat infra masih Docker/Traefik. Produksi sekarang Caddy + Nix/systemd dengan port 4000-an. Gunakan hanya sebagai referensi historis.
|
||||
|
||||
Ini adalah inti dari project. Pipeline mengubah foto dokumen HP jadi dokumen scan yang proper. Setiap tahap dibahas detail teknisnya.
|
||||
|
||||
## Pipeline Overview
|
||||
|
||||
```
|
||||
Input: Foto HP (JPEG/PNG/HEIC, 2-12MP)
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 1. Preprocess ──▶ resize + │
|
||||
│ konversi grayscale │
|
||||
└────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 2. Edge Detection ──▶ cari │
|
||||
│ kontur dokumen │
|
||||
└────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 3. Corner Detection ──▶ 4 titik │
|
||||
│ sudut dokumen │
|
||||
└────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 4. Perspective Warp ──▶ lurusin│
|
||||
│ (homography) │
|
||||
└────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 5. Shadow Removal ──▶ iluminasi │
|
||||
│ merata │
|
||||
└────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 6. Binarization ──▶ hitam-putih │
|
||||
│ bersih │
|
||||
└────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 7. Deskew ──▶ lurusin teks │
|
||||
│ (kalau masih miring) │
|
||||
└────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 8. OCR ──▶ extract teks │
|
||||
└────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 9. Generate PDF ──▶ output │
|
||||
│ PDF + hidden text layer │
|
||||
└────────────────┬─────────────────┘
|
||||
│
|
||||
▼
|
||||
Output: searchable PDF + teks OCR
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stage 1: Preprocess
|
||||
|
||||
### Input
|
||||
- Raw image dari HP (bisa 4000×3000 = 12MP, ~3-5MB JPEG)
|
||||
- Format: JPEG, PNG, HEIC (via `image` crate, HEIC butuh feature)
|
||||
|
||||
### Proses
|
||||
```rust
|
||||
use image::{DynamicImage, imageops};
|
||||
|
||||
fn preprocess(img: &DynamicImage) -> DynamicImage {
|
||||
// 1. Resize kalau terlalu besar → max 2000px di sisi terpanjang
|
||||
// Ini penting: edge detection di resolusi tinggi lambat
|
||||
// dan ga nambah akurasi secara signifikan
|
||||
let max_dim = 2000.0;
|
||||
let (w, h) = (img.width() as f64, img.height() as f64);
|
||||
let img = if w.max(h) > max_dim {
|
||||
let scale = max_dim / w.max(h);
|
||||
let new_w = (w * scale) as u32;
|
||||
let new_h = (h * scale) as u32;
|
||||
img.resize_exact(new_w, new_h, imageops::FilterType::Lanczos3)
|
||||
} else {
|
||||
img.clone()
|
||||
};
|
||||
|
||||
// 2. Grayscale → untuk edge detection
|
||||
img.grayscale()
|
||||
}
|
||||
```
|
||||
|
||||
### Edge Cases
|
||||
| Kasus | Penanganan |
|
||||
|-------|-----------|
|
||||
| Foto resolusi rendah (<800px) | Skip resize, langsung proses |
|
||||
| HEIC format | Butuh feature `heic` di `image` crate |
|
||||
| Grayscale input | `img.grayscale()` no-op |
|
||||
| Foto malam/noise tinggi | Gaussian blur sebelum edge detection |
|
||||
|
||||
---
|
||||
|
||||
## Stage 2: Edge Detection
|
||||
|
||||
### Tujuan
|
||||
Cari tepi dokumen dalam foto. Ini hardest part karena background bisa kacau.
|
||||
|
||||
### Algoritma: Canny Edge Detection + Adaptive Threshold
|
||||
|
||||
```rust
|
||||
use image::GrayImage;
|
||||
use imageproc::edges::canny;
|
||||
|
||||
fn detect_edges(img: &GrayImage) -> GrayImage {
|
||||
// Canny dengan dual threshold
|
||||
// low: 50, high: 150 — parameter ini harus di-tune
|
||||
// buat kondisi pencahayaan yang berbeda
|
||||
canny(img, 50.0, 150.0)
|
||||
}
|
||||
```
|
||||
|
||||
### Masalah & Solusi
|
||||
|
||||
| Masalah | Penyebab | Solusi |
|
||||
|---------|----------|--------|
|
||||
| **Tepi dokumen putus** | Kontras rendah, bayangan | Morphological close (dilate → erode) untuk sambungin tepi |
|
||||
| **Tepi palsu** | Background ramai (meja motif, lantai) | Cari contour terbesar + area terluas = dokumen |
|
||||
| **Tidak ada tepi** | Background putih, dokumen putih (kertas di meja putih) | Adaptive threshold dulu sebelum Canny, atau fallback ke manual crop |
|
||||
| **Noise garis** | Texture background | Gaussian blur (kernel 5x5) sebelum Canny |
|
||||
|
||||
### Implementation Detail
|
||||
|
||||
```rust
|
||||
/// Edge detection yang robust terhadap berbagai kondisi
|
||||
fn robust_edge_detection(img: &GrayImage) -> GrayImage {
|
||||
// 1. Gaussian blur untuk noise reduction
|
||||
let blurred = imageproc::filter::gaussian_blur_f32(img, 3.0);
|
||||
|
||||
// 2. Coba Canny standard
|
||||
let edges = canny(&blurred, 50.0, 150.0);
|
||||
|
||||
// 3. Morphological close untuk sambung tepi yang putus
|
||||
let kernel = imageproc::morphology::dilate_square(5);
|
||||
let closed = imageproc::morphology::close(&edges, &kernel);
|
||||
|
||||
// 4. Kalau jumlah tepi terlalu sedikit (<1% pixels),
|
||||
// ulang dengan threshold lebih rendah
|
||||
let edge_count = count_non_zero(&closed);
|
||||
let total_pixels = (closed.width() * closed.height()) as u32;
|
||||
if edge_count < total_pixels / 100 {
|
||||
let edges2 = canny(&blurred, 20.0, 80.0);
|
||||
return imageproc::morphology::close(&edges2, &kernel);
|
||||
}
|
||||
|
||||
closed
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stage 3: Corner Detection
|
||||
|
||||
### Tujuan
|
||||
Dari edge image, cari 4 sudut dokumen.
|
||||
|
||||
### Algoritma: Contour Detection → Largest Rectangle
|
||||
|
||||
```rust
|
||||
use imageproc::contours::{find_contours, Contour};
|
||||
|
||||
fn find_document_corners(edges: &GrayImage) -> Option<[(f64, f64); 4]> {
|
||||
// 1. Cari semua contours
|
||||
let contours = find_contours(edges);
|
||||
|
||||
// 2. Filter: cuma contour dengan area > 20% dari total image
|
||||
// (dokumen biasanya mengisi sebagian besar frame)
|
||||
let total_area = edges.width() as f64 * edges.height() as f64;
|
||||
let docs: Vec<&Contour> = contours
|
||||
.iter()
|
||||
.filter(|c| area_perimeter_ratio(c) > 0.3)
|
||||
.collect();
|
||||
|
||||
// 3. Approximate polygon → cari yang 4 sisi
|
||||
for contour in docs {
|
||||
// Approximate contour ke polygon
|
||||
let polygon = approximate_polygon(&contour.points, 4);
|
||||
if let Some(vertices) = polygon {
|
||||
// Urutkan: top-left, top-right, bottom-right, bottom-left
|
||||
let corners = order_corners(vertices);
|
||||
return Some(corners);
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Fallback: contour terbesar → bounding rect
|
||||
contours.iter()
|
||||
.max_by_key(|c| c.points.len())
|
||||
.map(|c| {
|
||||
let rect = bounding_rect(&c.points);
|
||||
order_corners(vec![
|
||||
(rect.left as f64, rect.top as f64),
|
||||
(rect.right as f64, rect.top as f64),
|
||||
(rect.right as f64, rect.bottom as f64),
|
||||
(rect.left as f64, rect.bottom as f64),
|
||||
])
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
### Corner Ordering Convention
|
||||
|
||||
```
|
||||
(0,0) top-left ────────── top-right (w,0)
|
||||
│ │
|
||||
│ DOKUMEN │
|
||||
│ │
|
||||
(0,h) bottom-left ────── bottom-right (w,h)
|
||||
```
|
||||
|
||||
### Fallback Strategy
|
||||
|
||||
Kalau auto-detect gagal total (contour tidak ketemu, confidence rendah):
|
||||
1. **Fallback 1**: Coba di resolusi lebih rendah (noise berkurang)
|
||||
2. **Fallback 2**: Coba adaptive threshold + Canny ulang
|
||||
3. **Fallback 3**: Minta user crop manual — 4 draggable corners di canvas
|
||||
|
||||
```rust
|
||||
fn detect_corners_with_fallback(img: &GrayImage) -> Result<[(f64, f64); 4], CropMode> {
|
||||
// Attempt 1: Resolusi penuh
|
||||
if let Some(corners) = find_document_corners(img) {
|
||||
return Ok(corners);
|
||||
}
|
||||
|
||||
// Attempt 2: Half resolution (noise reduction)
|
||||
let half = image::imageops::resize(img, img.width() / 2, img.height() / 2,
|
||||
imageops::FilterType::Lanczos3);
|
||||
if let Some(corners) = find_document_corners(&half) {
|
||||
return Ok(corners.map(|(x, y)| (x * 2.0, y * 2.0)));
|
||||
}
|
||||
|
||||
// Fallback: user manual
|
||||
Err(CropMode::Manual)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stage 4: Perspective Warp
|
||||
|
||||
### Tujuan
|
||||
Transform 4 titik sudut ke persegi panjang (rectangular). Koreksi perspektif dari foto miring.
|
||||
|
||||
### Algoritma: Homography
|
||||
|
||||
```rust
|
||||
use image::{DynamicImage, GrayImage};
|
||||
use std::f64::consts::PI;
|
||||
|
||||
fn perspective_warp(img: &DynamicImage, corners: [(f64, f64); 4]) -> DynamicImage {
|
||||
// Target: persegi panjang dengan aspect ratio dokumen
|
||||
// Hitung lebar dan tinggi target dari 4 corner
|
||||
let [tl, tr, br, bl] = corners;
|
||||
|
||||
let width_top = distance(tl, tr);
|
||||
let width_bot = distance(bl, br);
|
||||
let width = width_top.max(width_bot).ceil() as u32;
|
||||
|
||||
let height_left = distance(tl, bl);
|
||||
let height_right = distance(tr, br);
|
||||
let height = height_left.max(height_right).ceil() as u32;
|
||||
|
||||
// Source points (4 corners dari detection)
|
||||
let src = [
|
||||
tl, // top-left
|
||||
tr, // top-right
|
||||
br, // bottom-right
|
||||
bl, // bottom-left
|
||||
];
|
||||
|
||||
// Destination points (rectangle)
|
||||
let dst = [
|
||||
(0.0, 0.0), // top-left
|
||||
(width as f64, 0.0), // top-right
|
||||
(width as f64, height as f64), // bottom-right
|
||||
(0.0, height as f64), // bottom-left
|
||||
];
|
||||
|
||||
// Hitung homography matrix
|
||||
let h = compute_homography(&src, &dst);
|
||||
|
||||
// Apply warp (backward mapping + bilinear interpolation)
|
||||
warp_image(img, &h, width, height)
|
||||
}
|
||||
```
|
||||
|
||||
### Homography Matrix
|
||||
|
||||
```
|
||||
H = [h11 h12 h13] x' = (h11*x + h12*y + h13) / (h31*x + h32*y + 1)
|
||||
[h21 h22 h23] y' = (h21*x + h22*y + h23) / (h31*x + h32*y + 1)
|
||||
[h31 h32 1 ]
|
||||
```
|
||||
|
||||
Komputasi manual (tanpa OpenCV):
|
||||
```rust
|
||||
/// Compute homography from 4 point correspondences using DLT algorithm
|
||||
fn compute_homography(src: &[(f64, f64); 4], dst: &[(f64, f64); 4]) -> [[f64; 3]; 3] {
|
||||
// Direct Linear Transform
|
||||
// Bangun matrix A (8x9) dari 4 titik
|
||||
// Solve Ah = 0 via SVD → h = last column of V
|
||||
// Reshape ke 3x3
|
||||
//
|
||||
// Detail implementasi:
|
||||
// Setiap titik correspondence (x,y) → (x',y') menghasilkan 2 baris:
|
||||
// [-x, -y, -1, 0, 0, 0, x*x', y*x', x'] = 0
|
||||
// [ 0, 0, 0, -x, -y, -1, x*y', y*y', y'] = 0
|
||||
//
|
||||
// 4 titik → 8 baris → SVD → H matrix
|
||||
|
||||
// Implementasi SVD atau pakai crate `nalgebra` atau `splines`
|
||||
todo!("Implement DLT + SVD")
|
||||
}
|
||||
```
|
||||
|
||||
### Image Warp (Backward Mapping)
|
||||
|
||||
```rust
|
||||
fn warp_image(img: &DynamicImage, h: &[[f64; 3]; 3], width: u32, height: u32) -> DynamicImage {
|
||||
let gray = img.grayscale().into_luma8();
|
||||
let mut output = GrayImage::new(width, height);
|
||||
|
||||
// Inverse homography (backward mapping)
|
||||
// tiap pixel output = sample dari input
|
||||
let h_inv = invert_homography(h);
|
||||
|
||||
for y in 0..height {
|
||||
for x in 0..width {
|
||||
// Map (x,y) → source image coordinates
|
||||
let (sx, sy) = apply_homography(&h_inv, x as f64, y as f64);
|
||||
|
||||
// Bilinear interpolation
|
||||
let pixel = bilinear_interpolate(&gray, sx, sy);
|
||||
output.put_pixel(x, y, pixel);
|
||||
}
|
||||
}
|
||||
|
||||
DynamicImage::ImageLuma8(output)
|
||||
}
|
||||
```
|
||||
|
||||
### Edge Cases
|
||||
|
||||
| Masalah | Solusi |
|
||||
|---------|--------|
|
||||
| Dokuen sangat miring (>60°) | Warping mungkin hasilnya gepeng. Deteksi dan skip kalau sudut terlalu ekstrim |
|
||||
| Output sangat besar | Clamp width/height ke max 3000px |
|
||||
| Pixel jaggy (aliasing) | Bilinear interpolation (bukan nearest neighbor) |
|
||||
| Koordinat negative | Clamp ke 0 |
|
||||
| Warp membuat rasio aneh | Lock aspect ratio ke common (A4=1.414, Letter=1.294) |
|
||||
|
||||
---
|
||||
|
||||
## Stage 5: Shadow Removal
|
||||
|
||||
### Tujuan
|
||||
Hilangkan bayangan (dari lampu, jari, atau sudut ruangan).
|
||||
|
||||
### Algoritma: Adaptive Illumination Correction
|
||||
|
||||
Shadow adalah low-frequency variation. Teks adalah high-frequency. Pisahkan pake low-pass filter.
|
||||
|
||||
```rust
|
||||
fn remove_shadow(img: &GrayImage) -> GrayImage {
|
||||
let (w, h) = (img.width(), img.height());
|
||||
|
||||
// 1. Large Gaussian blur untuk estimasi iluminasi background
|
||||
// Kernel besar (≥sx/50) → cuma dapet variasi iluminasi, bukan teks
|
||||
let blur_radius = (w.min(h) as f64 / 50.0).max(15.0);
|
||||
let background = imageproc::filter::gaussian_blur_f32(img, blur_radius);
|
||||
|
||||
// 2. Subtract background dari original
|
||||
// pixel = max(0, original - background + mean(background))
|
||||
let bg_mean = mean_pixel(&background);
|
||||
let mut corrected = GrayImage::new(w, h);
|
||||
|
||||
for y in 0..h {
|
||||
for x in 0..w {
|
||||
let orig = img.get_pixel(x, y)[0] as f32;
|
||||
let bg = background.get_pixel(x, y)[0] as f32;
|
||||
let corrected_val = (orig - bg + bg_mean) as u8;
|
||||
corrected.put_pixel(x, y, Luma([corrected_val]));
|
||||
}
|
||||
}
|
||||
|
||||
// 3. CLAHE (Contrast Limited Adaptive Histogram Equalization)
|
||||
// untuk normalisasi kontras lokal
|
||||
apply_clahe(&corrected, 8, 4) // 8x8 tiles, clip limit 4
|
||||
}
|
||||
```
|
||||
|
||||
### Alternatif: Retinex Theory
|
||||
|
||||
```rust
|
||||
/// Retinex-based illumination correction
|
||||
/// I(x,y) = R(x,y) × L(x,y)
|
||||
/// I = observed image, R = reflectance (teks), L = illumination (shadow)
|
||||
fn retinex_shadow_removal(img: &GrayImage) -> GrayImage {
|
||||
// Single-scale Retinex
|
||||
// log(R) = log(I) - log(G * I)
|
||||
// dimana G = Gaussian kernel
|
||||
|
||||
let float_img = convert_to_float(img);
|
||||
let blurred = gaussian_blur_float(&float_img, 30.0);
|
||||
let retinex = element_wise(|p| (p.0.ln() - p.1.ln()), &float_img, &blurred);
|
||||
|
||||
// Normalize ke [0, 255]
|
||||
normalize_to_u8(&retinex)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stage 6: Binarization
|
||||
|
||||
### Tujuan
|
||||
Ubah ke hitam-putih bersih — teks hitam, background putih.
|
||||
|
||||
### Algoritma: Sauvola Local Threshold
|
||||
|
||||
Global threshold (Otsu) gagal kalau iluminasi ga merata. Sauvola adaptif per region.
|
||||
|
||||
```rust
|
||||
fn sauvola_threshold(img: &GrayImage, window_size: u32, k: f32) -> GrayImage {
|
||||
// Sauvola: T(x,y) = m(x,y) * [1 + k * (s(x,y)/R - 1)]
|
||||
// m = local mean, s = local std dev, R = max std dev (128), k = parameter (~0.2)
|
||||
|
||||
let (w, h) = (img.width(), img.height());
|
||||
let half_win = (window_size / 2) as i32;
|
||||
let mut output = GrayImage::new(w, h);
|
||||
|
||||
// Integral image for O(1) mean and variance computation
|
||||
let integral = compute_integral_image(img);
|
||||
let integral_sq = compute_integral_image_sq(img);
|
||||
|
||||
for y in 0..h {
|
||||
for x in 0..w {
|
||||
let (mean, variance) = local_stats(&integral, &integral_sq,
|
||||
x as i32, y as i32,
|
||||
half_win, w as i32, h as i32);
|
||||
let std_dev = variance.sqrt();
|
||||
let threshold = mean * (1.0 + k * (std_dev / 128.0 - 1.0));
|
||||
|
||||
let pixel = img.get_pixel(x, y)[0] as f32;
|
||||
output.put_pixel(x, y, Luma([if pixel > threshold { 255 } else { 0 }]));
|
||||
}
|
||||
}
|
||||
|
||||
output
|
||||
}
|
||||
```
|
||||
|
||||
### Parameter Default
|
||||
|
||||
| Parameter | Value | Notes |
|
||||
|-----------|-------|-------|
|
||||
| Window size | max(w,h)/30 | Minimum 15, maksimum 100 |
|
||||
| k | 0.2 | Lower → lebih sensitif, higher → lebih toleran |
|
||||
|
||||
### Edge Cases
|
||||
|
||||
| Masalah | Solusi |
|
||||
|---------|--------|
|
||||
| Dokumen berwarna (bukan putih) | Deteksi warna dominan background, invert logic |
|
||||
| Background gradasi | Sauvola handle ini lebih baik dari Otsu |
|
||||
| Foto terlalu gelap | CLAHE dulu sebelum binarization |
|
||||
| Text tipis/kabur | Morphological erode tipis sesudah binarization |
|
||||
|
||||
---
|
||||
|
||||
## Stage 7: Deskew
|
||||
|
||||
### Tujuan
|
||||
Koreksi rotasi sisa (kalau dokumen masih miring sedikit — biasanya <5°).
|
||||
|
||||
### Algoritma: Hough Transform
|
||||
|
||||
```rust
|
||||
fn deskew(img: &GrayImage) -> GrayImage {
|
||||
// 1. Cari garis teks via Hough transform
|
||||
// Probabilistic Hough lebih cepat
|
||||
let lines = probabilistic_hough_lines(img, 10, PI / 180.0, 50, 50.0, 10.0);
|
||||
|
||||
if lines.is_empty() {
|
||||
return img.clone();
|
||||
}
|
||||
|
||||
// 2. Hitung sudut rata-rata semua garis
|
||||
let angles: Vec<f64> = lines.iter()
|
||||
.map(|line| line.angle().to_degrees())
|
||||
.filter(|a| a.abs() < 45.0) // skip garis vertikal
|
||||
.collect();
|
||||
|
||||
if angles.is_empty() {
|
||||
return img.clone();
|
||||
}
|
||||
|
||||
let median_angle = median(&angles);
|
||||
|
||||
// Skip kalau sudutnya <0.5 derajat (ga perlu koreksi)
|
||||
if median_angle.abs() < 0.5 {
|
||||
return img.clone();
|
||||
}
|
||||
|
||||
// 3. Rotate image
|
||||
rotate(img, median_angle, imageops::FilterType::Lanczos3)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stage 8: OCR
|
||||
|
||||
### Tujuan
|
||||
Extract teks dari gambar biar PDF-nya searchable dan teks bisa di-copy.
|
||||
|
||||
### Implementation
|
||||
|
||||
```rust
|
||||
use leptess::LepTess;
|
||||
|
||||
fn ocr(img: &GrayImage, lang: &str) -> Result<String, OcrError> {
|
||||
// 1. Init Tesseract
|
||||
let mut tess = LepTess::new(Some("/usr/share/tesseract/tessdata"), lang)?;
|
||||
|
||||
// 2. Set image
|
||||
tess.set_image_from_mem(&img.to_bytes())?;
|
||||
// 3. Set PSM (Page Segmentation Mode)
|
||||
// PSM 3 = Fully automatic, default
|
||||
// PSM 6 = Assume single uniform block of text
|
||||
// PSM 4 = Assume single column of text
|
||||
tess.set_source_resolution(300);
|
||||
|
||||
// 4. Recognize
|
||||
let text = tess.get_utf8_text()?;
|
||||
|
||||
Ok(text)
|
||||
}
|
||||
|
||||
/// Dapatkan word-level bounding boxes untuk positioning di PDF
|
||||
fn ocr_words(img: &GrayImage, lang: &str) -> Result<Vec<Word>, OcrError> {
|
||||
let mut tess = LepTess::new(Some("/usr/share/tesseract/tessdata"), lang)?;
|
||||
tess.set_image_from_mem(&img.to_bytes())?;
|
||||
|
||||
let words = tess.get_words()
|
||||
.iter()
|
||||
.map(|w| Word {
|
||||
text: w.text.clone(),
|
||||
bbox: Bbox {
|
||||
x: w.x,
|
||||
y: w.y,
|
||||
width: w.w,
|
||||
height: w.h,
|
||||
},
|
||||
confidence: w.confidence,
|
||||
})
|
||||
.collect();
|
||||
|
||||
Ok(words)
|
||||
}
|
||||
```
|
||||
|
||||
### Output Format
|
||||
|
||||
```rust
|
||||
struct Word {
|
||||
text: String,
|
||||
bbox: Bbox,
|
||||
confidence: i32, // 0-100
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stage 9: PDF Generation
|
||||
|
||||
### Tujuan
|
||||
Generate PDF yang:
|
||||
1. Berisi gambar hasil scan (JPEG compressed)
|
||||
2. Hidden text layer dari OCR (biar searchable, selectable)
|
||||
|
||||
### Implementation
|
||||
|
||||
```rust
|
||||
use lopdf::{Document, Object, Stream};
|
||||
use std::io::Write;
|
||||
|
||||
fn generate_searchable_pdf(
|
||||
image_data: &[u8], // JPEG-compressed scan image
|
||||
ocr_text: &str, // Full OCR text
|
||||
words: &[Word], // Word positions
|
||||
page_width: f64, // PDF page width in points
|
||||
page_height: f64, // PDF page height in points
|
||||
) -> Result<Vec<u8>, PdfError> {
|
||||
let mut doc = Document::new();
|
||||
|
||||
// 1. Create image XObject
|
||||
let image_stream = Stream::new(
|
||||
dictionary! {
|
||||
"Type" => "XObject",
|
||||
"Subtype" => "Image",
|
||||
"Width" => page_width as u32,
|
||||
"Height" => page_height as u32,
|
||||
"ColorSpace" => "DeviceGray",
|
||||
"BitsPerComponent" => 8,
|
||||
"Filter" => "DCTDecode", // JPEG compression
|
||||
},
|
||||
image_data,
|
||||
);
|
||||
let image_id = doc.add_object(image_stream);
|
||||
|
||||
// 2. Create content stream: place image, then invisible text
|
||||
// Text layer is invisible (rendering mode 3 = neither fill nor stroke)
|
||||
let mut content = Vec::new();
|
||||
writeln!(content, "q")?; // save state
|
||||
writeln!(content, "{} 0 0 {} 0 0 cm", page_width, page_height)?; // scale to page
|
||||
writeln!(content, "/Im0 Do")?; // place image
|
||||
writeln!(content, "Q")?; // restore state
|
||||
|
||||
// 3. Add invisible text layer (searchable)
|
||||
for word in words {
|
||||
let x = word.bbox.x as f64 / DPI * 72.0; // convert pixels → points
|
||||
let y = (page_height - word.bbox.y as f64 / DPI * 72.0);
|
||||
writeln!(content, "BT")?;
|
||||
writeln!(content, "3 Tr")?; // rendering mode: invisible
|
||||
writeln!(content, "1 Tw")?; // word spacing
|
||||
writeln!(content, "{} {} Td", x, y)?; // position
|
||||
writeln!(content, "({}) Tj", escape_pdf_string(&word.text))?;
|
||||
writeln!(content, "ET")?;
|
||||
}
|
||||
|
||||
let content_stream = Stream::new(
|
||||
dictionary! {},
|
||||
content,
|
||||
);
|
||||
let content_id = doc.add_object(content_stream);
|
||||
|
||||
// 4. Create page
|
||||
let page_id = doc.new_object_id();
|
||||
let pages_id = doc.new_object_id();
|
||||
|
||||
doc.objects.insert(page_id, Object::Dictionary(dictionary! {
|
||||
"Type" => "Page",
|
||||
"Parent" => pages_id,
|
||||
"MediaBox" => vec![0.0, 0.0, page_width, page_height],
|
||||
"Contents" => content_id,
|
||||
"Resources" => dictionary! {
|
||||
"XObject" => dictionary! {
|
||||
"Im0" => image_id,
|
||||
},
|
||||
},
|
||||
}));
|
||||
|
||||
// 5. Close and return bytes
|
||||
let bytes = doc.save_to_bytes()?;
|
||||
Ok(bytes)
|
||||
}
|
||||
```
|
||||
|
||||
### PDF Coordinate System
|
||||
|
||||
```
|
||||
PDF origin = bottom-left
|
||||
Image origin = top-left
|
||||
|
||||
Perlu flip Y coordinate untuk text layer:
|
||||
y_pdf = page_height - (y_image / dpi * 72)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Complete Pipeline Assembly
|
||||
|
||||
```rust
|
||||
pub struct ScanPipeline {
|
||||
config: PipelineConfig,
|
||||
metrics: MetricsRecorder,
|
||||
}
|
||||
|
||||
impl ScanPipeline {
|
||||
pub async fn process(&self, input_path: &Path, options: ScanOptions)
|
||||
-> Result<ScanResult, PipelineError>
|
||||
{
|
||||
let timer = self.metrics.start_timer("scan.full");
|
||||
|
||||
// 1. Load
|
||||
let img = image::open(input_path)
|
||||
.map_err(PipelineError::ImageLoad)?;
|
||||
self.metrics.stage_duration("load", timer.split());
|
||||
|
||||
// 2. Preprocess
|
||||
let gray = preprocess(&img);
|
||||
self.metrics.stage_duration("preprocess", timer.split());
|
||||
|
||||
// 3. Edge detection + corners (fallback chain)
|
||||
let corners = detect_corners_with_fallback(&gray)
|
||||
.map_err(PipelineError::CornerDetection)?;
|
||||
self.metrics.stage_duration("corner_detection", timer.split());
|
||||
|
||||
// 4. Perspective warp
|
||||
let warped = perspective_warp(&img, corners); // warp from COLOR original, not gray
|
||||
self.metrics.stage_duration("warp", timer.split());
|
||||
|
||||
let warped_gray = warped.grayscale().into_luma8();
|
||||
|
||||
// 5. Shadow removal
|
||||
let clean = remove_shadow(&warped_gray);
|
||||
self.metrics.stage_duration("shadow_removal", timer.split());
|
||||
|
||||
// 6. Binarization
|
||||
let binary = sauvola_threshold(&clean, 50, 0.2);
|
||||
self.metrics.stage_duration("binarization", timer.split());
|
||||
|
||||
// 7. Deskew
|
||||
let final_image = deskew(&binary);
|
||||
self.metrics.stage_duration("deskew", timer.split());
|
||||
|
||||
// 8. Enhance final (sharpening)
|
||||
let final_image = sharpen(&final_image, 1.0);
|
||||
self.metrics.stage_duration("sharpen", timer.split());
|
||||
|
||||
// 9. OCR
|
||||
let ocr_text = if options.ocr {
|
||||
Some(ocr(&final_image, "eng")?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
self.metrics.stage_duration("ocr", timer.split());
|
||||
|
||||
// 10. Generate PDF
|
||||
let pdf_bytes = generate_searchable_pdf(
|
||||
&compress_jpeg(&final_image, 90)?,
|
||||
&ocr_text.unwrap_or_default(),
|
||||
&[], // word positions (simplified)
|
||||
A4_WIDTH_PT,
|
||||
A4_HEIGHT_PT,
|
||||
)?;
|
||||
self.metrics.stage_duration("pdf_generation", timer.split());
|
||||
|
||||
// 11. Save
|
||||
let output_path = PathBuf::from("/tmp/tools").join(format!("{}.pdf", uuid::Uuid::new_v4()));
|
||||
std::fs::write(&output_path, &pdf_bytes)?;
|
||||
|
||||
timer.finish();
|
||||
|
||||
Ok(ScanResult {
|
||||
output_path,
|
||||
page_count: 1,
|
||||
file_size: pdf_bytes.len() as u64,
|
||||
ocr_text,
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Budget
|
||||
|
||||
| Stage | Target | Notes |
|
||||
|-------|--------|-------|
|
||||
| Load + Preprocess | <200ms | File I/O + resize |
|
||||
| Edge + Corner Detection | <500ms | Canny + contour |
|
||||
| Perspective Warp | <800ms | Per-pixel backward mapping |
|
||||
| Shadow Removal | <300ms | FFT convolution atau integral image |
|
||||
| Binarization | <200ms | Integral image |
|
||||
| Deskew | <300ms | Hough transform |
|
||||
| OCR | <1.5s | Tesseract, 300dpi |
|
||||
| PDF Generation | <200ms | lopdf |
|
||||
| **Total** | **<4s** | Per page |
|
||||
|
||||
> **Catatan**: Target di atas untuk image 12MP (4000×3000). Parallel via Rayon untuk batch processing.
|
||||
|
||||
## Edge Cases Matrix
|
||||
|
||||
| Skenario | Pipeline Behavior |
|
||||
|----------|------------------|
|
||||
| Kertas putih di meja putih | Edge detection gagal → fallback ke manual crop |
|
||||
| Foto dari sudut 45° | Warp koreksi perspektif, output presisi |
|
||||
| Dokumen terlipat | Edge detection dapet bentuk aneh → fallback manual |
|
||||
| Bayangan jari | Shadow removal hilangkan |
|
||||
| Teks pudar/pensil | Sauvola threshold adaptif, contrast enhance dulu |
|
||||
| Tanda tangan & stempel | OCR bisa gagal di handwriting, tetap di-image |
|
||||
| Multi-page (buku/kontrak) | Batch upload, masing-masing diproses, digabung 1 PDF |
|
||||
| Foto malam | CLAHE + strong denoise sebelum edge detection |
|
||||
| Latar belakang gradasi | Sauvola handle lebih baik dari Otsu |
|
||||
@@ -1,211 +0,0 @@
|
||||
# Security Guide
|
||||
|
||||
Praktik keamanan untuk infrastruktur `asepharyana-hub`.
|
||||
|
||||
## Ringkasan
|
||||
|
||||
| Area | Status | Prioritas |
|
||||
|------|--------|-----------|
|
||||
| Secrets management | GitHub encrypted secrets | Tinggi |
|
||||
| TLS termination | Traefik + cert volume mounts | Tinggi |
|
||||
| Container security | Non-root user (scraper-api) | Sedang |
|
||||
| Network security | Tailscale overlay, app-shared-net | Sedang |
|
||||
| Access control | SSH key, GitHub permissions | Sedang |
|
||||
| Monitoring | Belum ada alert system | Rendah |
|
||||
| Firewall | UFW/iptables (manual) | Sedang |
|
||||
| Backup | lihat `docs/backup-recovery.md` | Sedang |
|
||||
|
||||
## Secrets Management
|
||||
|
||||
### Yang Tidak Boleh di-Commit
|
||||
|
||||
- [ ] `.env` production (disimpan sebagai GitHub secret `ENV_FILE_PRODUCTION`)
|
||||
- [ ] SSH private keys
|
||||
- [ ] API tokens, JWT secret
|
||||
- [ ] Docker registry tokens
|
||||
- [ ] Database passwords
|
||||
- [ ] TLS certificate private keys
|
||||
|
||||
### GitHub Secrets
|
||||
|
||||
Setting di Settings > Secrets and variables > Actions:
|
||||
|
||||
| Secret | Tujuan | Rotasi |
|
||||
|--------|--------|--------|
|
||||
| `SSH_PRIVATE_KEY` | Akses SSH ke VPS | 6 bulan |
|
||||
| `VPS_HOST` | IP VPS | Tidak berubah |
|
||||
| `VPS_USER` | User SSH | Tidak berubah |
|
||||
| `VPS_TARGET_DIR` | Directory di VPS | Tidak berubah |
|
||||
| `ENV_FILE_PRODUCTION` | Full `.env` production | Saat ada perubahan |
|
||||
|
||||
### Update Secrets dengan aman
|
||||
|
||||
```bash
|
||||
# Baca current .env dari VPS via SSH
|
||||
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env" | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub --repos
|
||||
```
|
||||
|
||||
### Production `.env` tidak boleh di-commit
|
||||
|
||||
`.env` di root repo adalah untuk development lokal. Production `.env` hanya ada di:
|
||||
1. GitHub secret `ENV_FILE_PRODUCTION`
|
||||
2. File `/root/asepharyana-hub/.env` di VPS (hasil SCP dari CI/CD)
|
||||
|
||||
## TLS / SSL
|
||||
|
||||
### Konfigurasi
|
||||
|
||||
```yaml
|
||||
# Traefik TLS certs dari file mount (bukan auto-ACME)
|
||||
volumes:
|
||||
- ${TRAEFIK_CERT_MY_ID_PEM:-/root/asepharyana.my.id.pem}:/etc/traefik/certs/asepharyana.my.id.pem:ro
|
||||
- ${TRAEFIK_CERT_MY_ID_KEY:-/root/asepharyana.my.id.key}:/etc/traefik/certs/asepharyana.my.id.key:ro
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
- Certificates disimpan di host (`/root/`), bukan di repo
|
||||
- Volume mount read-only (`:ro`)
|
||||
- Private key hanya bisa dibaca oleh root (chmod 600)
|
||||
- Renew certificates sebelum expired (monitor expiry)
|
||||
- Dua domain: `asepharyana.my.id` + `asepharyana.web.id`
|
||||
|
||||
## Container Security
|
||||
|
||||
### Non-Root User
|
||||
|
||||
Scraper API berjalan sebagai `appuser` (UID 1001):
|
||||
|
||||
```dockerfile
|
||||
RUN groupadd -g 1001 appgroup && \
|
||||
useradd -u 1001 -g appgroup -s /bin/sh appuser
|
||||
USER appuser
|
||||
```
|
||||
|
||||
Service baru harus mengikuti pattern yang sama.
|
||||
|
||||
### Read-Only Filesystem
|
||||
|
||||
Untuk container yang tidak perlu write ke filesystem:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
image: app:latest
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp
|
||||
```
|
||||
|
||||
### Docker Socket
|
||||
|
||||
Hanya Traefik yang perlu akses ke Docker socket (read-only):
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
```
|
||||
|
||||
Service lain tidak boleh mount Docker socket.
|
||||
|
||||
### Image Security
|
||||
|
||||
- Build dari base image resmi dan minimal (`debian:bookworm-slim`, `redis:alpine`, `nats:latest`)
|
||||
- Multi-stage build untuk production image (tidak include build tools)
|
||||
- Update base image secara berkala
|
||||
|
||||
## Network Security
|
||||
|
||||
### Firewall (UFW/iptables)
|
||||
|
||||
Di VPS (`orangevps`):
|
||||
|
||||
```bash
|
||||
# Hanya buka port yang diperlukan
|
||||
sudo ufw default deny incoming
|
||||
sudo ufw default allow outgoing
|
||||
sudo ufw allow 22/tcp # SSH
|
||||
sudo ufw allow 80/tcp # HTTP redirect
|
||||
sudo ufw allow 443/tcp # HTTPS
|
||||
sudo ufw allow 4222/tcp # NATS (jika perlu external akses)
|
||||
sudo ufw enable
|
||||
```
|
||||
|
||||
Di `imrnes`:
|
||||
|
||||
```bash
|
||||
# Hanya dari Tailscale interface
|
||||
sudo ufw allow in on tailscale0 to any port 6432 proto tcp # PostgreSQL
|
||||
sudo ufw allow in on tailscale0 to any port 6379 proto tcp # Redis
|
||||
sudo ufw enable
|
||||
```
|
||||
|
||||
### Network Segmentation
|
||||
|
||||
- Semua container di network `app-shared-net` (internal bridge)
|
||||
- Tidak ada port yang di-expose ke host kecuali Traefik (80,443)
|
||||
- Redis hanya accessible via Docker DNS (`redis:6379`) — tidak di-expose
|
||||
- Database hanya via Tailscale — tidak accessible dari public internet
|
||||
|
||||
### SSH Hardening
|
||||
|
||||
Konfigurasi di `/etc/ssh/sshd_config`:
|
||||
|
||||
```
|
||||
Port 22
|
||||
PermitRootLogin prohibit-password
|
||||
PasswordAuthentication no
|
||||
PubkeyAuthentication yes
|
||||
AllowUsers root
|
||||
MaxAuthTries 3
|
||||
ClientAliveInterval 300
|
||||
ClientAliveCountMax 2
|
||||
```
|
||||
|
||||
## Access Control
|
||||
|
||||
### GitHub Repository
|
||||
|
||||
- `contents: write` hanya untuk workflow `update-manifest` dan `update-submodule`
|
||||
- `packages: write` hanya untuk workflow `build`
|
||||
- `security-events: write` hanya untuk workflow `security`
|
||||
- Branch protection di `main`: require PR review, status checks
|
||||
|
||||
### VPS
|
||||
|
||||
- SSH hanya dengan key-based authentication
|
||||
- Key disimpan di GitHub secret, bukan di repo
|
||||
- Rotate SSH key secara berkala (minimal 6 bulan)
|
||||
- Jangan gunakan password login
|
||||
|
||||
## Monitoring Keamanan
|
||||
|
||||
### Saat Ini
|
||||
|
||||
- Traefik access logs (format JSON, buffer size 100)
|
||||
- Docker logs via `docker logs`
|
||||
- CodeQL analysis untuk Rust code (setiap PR + weekly)
|
||||
|
||||
### Rekomendasi
|
||||
|
||||
- [ ] Alert untuk SSH failed login (fail2ban)
|
||||
- [ ] Log monitoring (Loki / Promtail)
|
||||
- [ ] Container vulnerability scanning (Trivy / Snyk)
|
||||
- [ ] Certificate expiry monitoring
|
||||
- [ ] Disk usage alert
|
||||
- [ ] Unauthorized access detection
|
||||
|
||||
## Checklist Security
|
||||
|
||||
- [ ] SSH password authentication disabled
|
||||
- [ ] Root login via SSH key only
|
||||
- [ ] UFW/iptables configured
|
||||
- [ ] Docker socket only mounted where necessary (read-only)
|
||||
- [ ] Container berjalan sebagai non-root user
|
||||
- [ ] `.env` tidak di-commit
|
||||
- [ ] GitHub secrets ter-encrypt
|
||||
- [ ] TLS certificates valid dan belum expired
|
||||
- [ ] CodeQL analysis berjalan
|
||||
- [ ] Backup database berjalan
|
||||
- [ ] SSH key di-rotate
|
||||
- [ ] Docker image di-scan untuk vulnerability
|
||||
@@ -1,6 +1,6 @@
|
||||
# Troubleshooting
|
||||
|
||||
Kumpulan solusi untuk masalah umum yang spesifik di infrastruktur `asepharyana-hub`.
|
||||
Kumpulan solusi untuk masalah umum di infrastruktur `asepharyana/infra` (orangevps).
|
||||
|
||||
> **Catatan (2026-08-02):** Produksi sekarang Caddy + Nix/systemd. Section Traefik/Docker di bawah adalah LEGACY — Docker dan Traefik dihapus dari produksi; gunakan hanya sebagai referensi historis.
|
||||
|
||||
@@ -29,7 +29,7 @@ Kumpulan solusi untuk masalah umum yang spesifik di infrastruktur `asepharyana-h
|
||||
| `SSH_PRIVATE_KEY` | Wajib |
|
||||
| `VPS_HOST` | Wajib (`45.127.35.244`) |
|
||||
| `VPS_USER` | Wajib (`root`) |
|
||||
| `VPS_TARGET_DIR` | Wajib (`/root/asepharyana-hub`) |
|
||||
| ~~`VPS_TARGET_DIR`~~ | Tidak dipakai lagi (CI baru tidak checkout VPS) |
|
||||
| `ENV_FILE_PRODUCTION` | Wajib |
|
||||
|
||||
### Workflow build gagal: "Submodule commit not fetchable"
|
||||
@@ -40,7 +40,7 @@ Kumpulan solusi untuk masalah umum yang spesifik di infrastruktur `asepharyana-h
|
||||
|
||||
```bash
|
||||
# Cek apakah commit ada di remote
|
||||
git ls-remote https://github.com/asepharyana/asepharyana-hub-scraper.git <SHA>
|
||||
git ls-remote https://github.com/asepharyana/scraper.git <SHA>
|
||||
|
||||
# Trigger ulang dispatch dari submodule repo, atau push langsung ke hub
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user