Author SHA1 Message Date
asepharyana e1464fdc20 test: pr-agent auto-review v2 2026-07-17 08:02:06 +07:00
106 changed files with 4842 additions and 8371 deletions
-29
View File
@@ -1,29 +0,0 @@
{
"skills": [
{
"name": "clean-code",
"filePattern": ".claude/skills/clean-code/SKILL.md",
"description": "Clean Code, Clean Architecture, SOLID, TDD — dari kana-best-practice-engineering"
},
{
"name": "hub-rules",
"filePattern": ".claude/skills/hub-rules.md",
"description": "Aturan repository hub, submodule, infra patterns, dan arsitektur"
},
{
"name": "commit-convention",
"filePattern": ".claude/skills/commit-convention.md",
"description": "Commit message convention — type(scope): description"
},
{
"name": "event-driven",
"filePattern": ".claude/skills/event-driven.md",
"description": "Event-driven patterns with Dapr + NATS untuk hub services"
},
{
"name": "deploy-workflow",
"filePattern": ".claude/skills/deploy-workflow.md",
"description": "CI/CD pipeline, Docker patterns, deployment guide"
}
]
}
-1
View File
@@ -1 +0,0 @@
/home/asephs/kana-best-practice-engineering/skills/clean-code
-65
View File
@@ -1,65 +0,0 @@
---
name: commit-convention
description: Enforce commit message convention untuk Asepharyana Hub
---
# Commit Convention — Asepharyana Hub
## Format
```
<type>(<scope>): <description>
[optional body]
[optional footer]
```
## Types
| Type | Usage |
| ---------- | ------------------------------------ |
| `feat` | Fitur baru |
| `fix` | Bug fix |
| `chore` | Maintenance, config, tooling |
| `docs` | Dokumentasi |
| `refactor` | Perubahan kode tanpa fungsional baru |
| `test` | Nambah/update test |
| `ci` | CI/CD workflows |
| `perf` | Optimasi performa |
| `style` | Formatting (tanda kutip, dll) |
## Scopes
| Scope | Area |
| ------------- | --------------------------------- |
| `scraper` | apps/scraper submodule |
| `infra` | infra/ (compose, traefik, docker) |
| `ci` | .github/workflows/ |
| `dapr` | Dapr config & sidecar |
| `nats` | NATS message bus |
| `docs` | Dokumentasi |
| `deps` | Dependencies |
| `scripts` | Utility scripts |
| `root` | Root config files |
## Contoh
```
feat(scraper): add anime detail caching via Dapr pubsub
fix(infra): correct NATS CLI flags for JetStream
chore(deps): update biome to v2.5.3
docs(infra): add deployment order for Dapr services
ci(deploy): add nats.yml to ALL_COMPOSE_FILES
refactor(scraper): migrate EventBus from tokio broadcast to Dapr pubsub
```
## Aturan
1. **Wajib** menyertakan scope dalam tanda kurung
2. **Wajib** `Co-Authored-By` untuk commit yang digenerate AI
3. **Gunakan imperative mood**: "add" bukan "added" / "adds"
4. **Jangan capitalize** type: `feat:` bukan `Feat:`
5. **No period** di akhir subject baris
6. Body explain **why** dan **what**, bukan **how**
7. Refer issue dengan `Closes #123` atau `Fixes #123` di footer
-103
View File
@@ -1,103 +0,0 @@
---
name: deploy-workflow
description: Panduan deploy, CI/CD, dan Docker compose patterns untuk Asepharyana Hub
---
# Deploy & Workflow — Asepharyana Hub
## CI/CD Pipeline
### Build Pipeline (`docker-build-push.yml`)
Trigger: push ke `main` yang touch `apps/**`, `infra/**`, `infra/docker/**`
1. **changes** — detect service mana yg berubah via git diff
2. **wait-submodule-ref** — (repository_dispatch only) tunggu SHA commit fetchable
3. **build** — matrix build per service, push ke GHCR (`sha-<short>` + `latest`)
4. **update-manifest** — update image tag di compose file, commit + push
### Deploy Pipeline (`deploy-docker.yml`)
Trigger: build selesai, atau push ke `main` touch `infra/**`
1. SSH ke `orangevps` (via `secrets.VPS_HOST`)
2. Sync repo (`git fetch --depth=1 + reset`)
3. Login ke GHCR
4. Deteksi compose file yg berubah
5. Pull images + restart container selektif
### Secrets Required
| Secret | Untuk |
|--------|-------|
| `SSH_PRIVATE_KEY` | SSH ke VPS |
| `VPS_HOST` | IP/host VPS (tailscale IP) |
| `VPS_USER` | SSH user, biasanya `root` |
| `VPS_TARGET_DIR` | Lokasi repo di VPS |
| `ENV_FILE_PRODUCTION` | .env content untuk production |
### Selective Deployment
- Hanya compose file yg berubah yang di-redeploy
- Selective: `UP_FLAGS="-d"` (tanpa `--remove-orphans`)
- Full deploy: `UP_FLAGS="-d --remove-orphans"`
## Docker Patterns
### Build dengan cargo-chef (Rust)
```dockerfile
FROM lukemathwalker/cargo-chef:latest-rust-1.89.0 AS chef
WORKDIR /app
FROM chef AS planner
COPY apps/scraper .
RUN cargo chef prepare --recipe-path recipe.json
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json
COPY apps/scraper .
RUN cargo build --release
```
### Runtime minimal untuk Rust binary
```dockerfile
FROM debian:bookworm-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl libssl3 && rm -rf /var/lib/apt/lists/*
```
## Image Tagging
- `sha-<short-sha>` — immutable, untuk rollback
- `latest` — mutable, untuk convenience
- Build cache: `sha-<short>-buildcache`
- Registry: `ghcr.io/asepharyana/asepharyana-hub/<service>`
## Manual Deploy Steps
```bash
# 1. Login GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u asepharyana --password-stdin
# 2. Full stack
docker compose -f infra/compose/traefik.yml \
-f infra/compose/shared.yml \
-f infra/compose/nats.yml \
-f infra/compose/dapr.yml \
-f infra/compose/scraper.yml \
--env-file .env up -d --remove-orphans
# 3. Selective (hanya satu service)
docker compose -f infra/compose/scraper.yml --env-file .env up -d
```
## Troubleshooting
### Container reach Tailscale
Pastikan route ke Tailscale di main table:
```bash
ip route add 100.64.0.0/10 dev tailscale0 table main
systemctl restart tailscale-routes
```
### Healthcheck gagal di scratch images
NATS dan Dapr placement pake scratch — tidak bisa healthcheck. Cukup `service_started` di depends_on.
### Dapr sidecar crash
```bash
docker logs scraper-api-dapr | grep -iE "fatal|error"
```
Penyebab umum: komponen config salah, NATS/Dapr placement belum siap.
-133
View File
@@ -1,133 +0,0 @@
---
name: event-driven
description: Event-driven patterns dengan Dapr + NATS untuk Asepharyana Hub
---
# Event-Driven Architecture — Asepharyana Hub
## Stack
- **Message Backbone**: NATS + JetStream (untuk streaming & job queue)
- **Pub/Sub Runtime**: Dapr sidecar per service (pubsub via Redis built-in)
- **State Store**: Dapr → Redis
## Event Topics Convention
```
hub.<domain>.<action>
Contoh:
hub.image.cached → Image selesai di-cache ke CDN
hub.image.repaired → Image diperbaiki (CNAME change)
hub.scrape.anime.done → Scrape anime selesai
hub.system.alert → Error/alert dari service
```
## CloudEvents Format
```json
{
"specversion": "1.0",
"type": "hub.image.cached",
"source": "scraper-api",
"subject": "anime-poster",
"id": "uuid-v4",
"time": "2026-07-21T10:00:00Z",
"datacontenttype": "application/json",
"data": { ... }
}
```
## Publish Event (Rust via HTTP API)
Gunakan `reqwest` langsung ke Dapr sidecar (SDK Rust masih experimental):
```rust
let event = serde_json::json!({
"specversion": "1.0",
"type": "hub.image.cached",
"source": "scraper-api",
"id": Uuid::new_v4().to_string(),
"time": chrono::Utc::now().to_rfc3339(),
"datacontenttype": "application/json",
"data": { "original_url": url, "cdn_url": cdn_url }
});
reqwest::Client::new()
.post("http://localhost:3500/v1.0/publish/pubsub/hub.image.cached")
.json(&event)
.send()
.await?;
```
## Service Invocation
```bash
curl http://localhost:3500/v1.0/invoke/<app-id>/method/<path>
```
## State Store
```bash
# Set
curl -X POST http://localhost:3500/v1.0/state/statestore \
-H "Content-Type: application/json" \
-d '[{"key": "mykey", "value": "myvalue"}]'
# Get
curl http://localhost:3500/v1.0/state/statestore/mykey
# Delete
curl -X DELETE http://localhost:3500/v1.0/state/statestore/mykey
```
## Scraper Event Integration
File yang perlu dimodifikasi untuk event-driven:
| File | Perubahan |
|------|-----------|
| `src/events/bus.rs` | Ganti backend dari tokio broadcast ke Dapr pub/sub |
| `src/bootstrap/mod.rs` | Init DaprClient, inject ke AppState |
| `src/presentation/state.rs` | Tambah `dapr_client` field |
| `src/proxy/use_cases.rs` | Publish `ImageRepaired` & `ImageCached` events |
| `src/infrastructure/services/images/cache.rs` | Emit event tiap cache selesai |
| `Cargo.toml` | Tambah `reqwest`, `uuid`, `chrono` (jika belum ada) |
## Event Handlers (Subscribe)
Buat `src/subscribers/` untuk handler:
```rust
// src/subscribers/image_handler.rs
pub async fn handle_image_cached(event: CloudEvent) -> Result<()> {
// Log, notifikasi, update status
}
```
Daftarkan subscribers di `bootstrap/mod.rs` dengan spawn task:
```rust
tokio::spawn(async move {
let mut stream = dapr_client.subscribe("pubsub", "hub.image.cached");
while let Some(event) = stream.next().await {
handle_image_cached(event).await;
}
});
```
## Testing Event-Driven Code
```rust
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_publish_event() {
let client = MockDaprClient::new();
client.expect_publish()
.with(...)
.returning(|_| Ok(()));
// ... test
}
}
```
-96
View File
@@ -1,96 +0,0 @@
---
name: hub-rules
description: Aturan repository, arsitektur hub, submodule, dan workflow Asepharyana Hub
---
# Asepharyana Hub — Repository Rules
## Struktur Repository
```
asepharyana-hub/
├── apps/ # Git submodules — source code aplikasi
├── docs/ # Dokumentasi, ADR, deployment guide
├── infra/ # Infrastructure as code
│ ├── compose/ # Satu compose file per service
│ ├── dapr/ # Dapr component configs
│ ├── docker/ # Dockerfiles per service
│ └── traefik/ # Static & dynamic Traefik config
├── scripts/ # Utility scripts (cleanup, update-deps)
└── .github/workflows/ # CI/CD pipelines
```
### Aturan Submodule
- Setiap aplikasi di `apps/` adalah **submodule** ke repo terpisah.
- Perubahan kode aplikasi dilakukan di **repo masing-masing**, bukan di sini.
- Submodule pointer diupdate oleh CI/CD (bukan manual).
## Infrastructure Patterns
### Networking
- Semua service join **`app-shared-net`** (external Docker bridge)
- Service discovery via Docker DNS (container alias)
- Traefik sebagai ingress untuk HTTP/S eksternal
- Tailscale untuk cross-VPS (PostgreSQL, Redis)
### Compose File Pattern
```yaml
services:
<service>:
container_name: <service>
image: ghcr.io/asepharyana/asepharyana-hub/<service>:sha-<sha>
restart: always
networks:
app-shared-net:
aliases:
- <service>
env_file:
- ../../.env
networks:
app-shared-net:
name: app-shared-net
external: true
```
### Dapr Sidecar Pattern
```yaml
<service>-dapr:
container_name: <service>-dapr
image: daprio/daprd:latest
restart: always
depends_on:
nats:
condition: service_started
dapr-placement:
condition: service_started
networks:
- app-shared-net
command:
- './daprd'
- '--app-id=<service>'
- '--app-port=<port>'
- '--dapr-http-port=3500'
- '--dapr-grpc-port=50001'
- '--placement-host-address=dapr-placement:50005'
- '--resources-path=/components'
volumes:
- ../../infra/dapr/components:/components
```
### Traefik Routing
- Router + service definition di `infra/traefik/dynamic/apps.yaml`
- Subdomain pattern: `<service>.asepharyana.my.id` + `<service>.asepharyana.web.id`
- TLS cert dari volume mount (bukan auto-acme)
### CI/CD
- `docker-build-push.yml` — build per service, push ke GHCR, update compose manifest
- `deploy-docker.yml` — SSH ke orangevps, pull images, restart
- Selective deploy: hanya compose file yg berubah
## Deployment Order
1. `shared.yml` (Redis)
2. `nats.yml` (NATS message bus)
3. `dapr.yml` (Dapr placement)
4. `traefik.yml` (Reverse proxy)
5. Service compose files (apps + Dapr sidecar)
+7
View File
@@ -0,0 +1,7 @@
{
"image": "mcr.microsoft.com/devcontainers/universal:2",
"features": {
"ghcr.io/devcontainers/features/rust:1": {},
"ghcr.io/devcontainers/features/node:2": {}
}
}
+37 -22
View File
@@ -32,7 +32,7 @@ jobs:
if: github.event_name == 'workflow_dispatch' || github.event_name == 'push' || github.event.workflow_run.conclusion == 'success' if: github.event_name == 'workflow_dispatch' || github.event_name == 'push' || github.event.workflow_run.conclusion == 'success'
steps: steps:
- name: Checkout repository - name: Checkout repository
uses: actions/checkout@v7 uses: actions/checkout@v6
with: with:
fetch-depth: 1 fetch-depth: 1
submodules: false submodules: false
@@ -89,7 +89,7 @@ jobs:
git fetch origin main --depth=1 || true git fetch origin main --depth=1 || true
# Detect changed files before resetting # Detect changed files before resetting
ALL_COMPOSE_FILES="infra/compose/traefik.yml infra/compose/shared.yml infra/compose/scraper.yml infra/compose/hub.yml infra/compose/tools.yml infra/compose/llm-api.yml infra/compose/nats.yml infra/compose/dapr.yml infra/compose/observability.yml" ALL_COMPOSE_FILES="infra/compose/traefik.yml infra/compose/shared.yml infra/compose/scraper.yml infra/compose/elysia.yml infra/compose/react.yml infra/compose/rust-auth.yml"
TRAEFIK_DYNAMIC_DIR="infra/traefik/dynamic" TRAEFIK_DYNAMIC_DIR="infra/traefik/dynamic"
if git rev-parse HEAD >/dev/null 2>&1; then if git rev-parse HEAD >/dev/null 2>&1; then
@@ -142,6 +142,8 @@ jobs:
TARGET_COMPOSE="" TARGET_COMPOSE=""
fi fi
# Submodule update removed for faster VPS deployments
if command -v "docker" >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then if command -v "docker" >/dev/null 2>&1 && docker compose version >/dev/null 2>&1; then
COMPOSE_CMD="docker compose" COMPOSE_CMD="docker compose"
elif command -v docker-compose >/dev/null 2>&1; then elif command -v docker-compose >/dev/null 2>&1; then
@@ -151,28 +153,24 @@ jobs:
exit 1 exit 1
fi fi
# Always include ALL compose files for dependency resolution # Construct compose arguments
COMPOSE_ARGS="" docker rm -f imphenbot-app || true
for f in $ALL_COMPOSE_FILES; do
if [ -f "$f" ]; then
COMPOSE_ARGS="$COMPOSE_ARGS -f $f"
fi
done
if [ -n "$TARGET_COMPOSE" ]; then if [ -n "$TARGET_COMPOSE" ]; then
# Extract service names from target compose file(s) for selective up COMPOSE_ARGS=""
TARGET_SERVICES=""
for f in $TARGET_COMPOSE; do for f in $TARGET_COMPOSE; do
if [ -f "$f" ]; then if [ -f "$f" ]; then
svcs=$($COMPOSE_CMD -f "$f" config --services 2>/dev/null | tr '\n' ' ' | xargs) COMPOSE_ARGS="$COMPOSE_ARGS -f $f"
TARGET_SERVICES="$TARGET_SERVICES $svcs"
fi fi
done done
TARGET_SERVICES=$(echo "$TARGET_SERVICES" | xargs) # trim whitespace UP_FLAGS="-d" # No --remove-orphans for selective updates to avoid killing other services
echo "🎯 Selective update for services: $TARGET_SERVICES"
else else
echo "🚀 Performing full deployment of all services..." echo "🚀 Performing full deployment of all services..."
TARGET_SERVICES="" COMPOSE_ARGS=""
for f in $ALL_COMPOSE_FILES; do
COMPOSE_ARGS="$COMPOSE_ARGS -f $f"
done
UP_FLAGS="-d --remove-orphans"
fi fi
echo "📥 Pulling images for target services..." echo "📥 Pulling images for target services..."
@@ -181,7 +179,7 @@ jobs:
# Retry pull up to 3 times to handle transient Docker attestation lease errors # Retry pull up to 3 times to handle transient Docker attestation lease errors
for attempt in 1 2 3; do for attempt in 1 2 3; do
echo "Pull attempt $attempt/3..." echo "Pull attempt $attempt/3..."
if $COMPOSE_CMD $COMPOSE_ARGS --env-file .env pull $TARGET_SERVICES; then if $COMPOSE_CMD $COMPOSE_ARGS --env-file .env pull; then
echo "✅ Pull succeeded on attempt $attempt" echo "✅ Pull succeeded on attempt $attempt"
PULL_SUCCESS=true PULL_SUCCESS=true
break break
@@ -198,6 +196,10 @@ jobs:
echo "🧹 Clearing Git locks..." echo "🧹 Clearing Git locks..."
rm -f .git/shallow.lock || true rm -f .git/shallow.lock || true
# Cooldown to allow daemon to settle (optional but kept for safety)
# echo "⏳ Waiting for Docker daemon to settle..."
# sleep 2
echo "🧹 Removing stale target containers by container_name..." echo "🧹 Removing stale target containers by container_name..."
# Extract all explicitly defined container_names from compose files and remove them to prevent conflicts # Extract all explicitly defined container_names from compose files and remove them to prevent conflicts
if [ -n "$TARGET_COMPOSE" ]; then if [ -n "$TARGET_COMPOSE" ]; then
@@ -221,11 +223,7 @@ jobs:
echo "🆙 Starting services..." echo "🆙 Starting services..."
echo "🔍 Debug: Current docker containers:" echo "🔍 Debug: Current docker containers:"
docker ps -a docker ps -a
if [ -n "$TARGET_SERVICES" ]; then $COMPOSE_CMD $COMPOSE_ARGS --env-file .env up $UP_FLAGS
$COMPOSE_CMD $COMPOSE_ARGS --env-file .env up -d $TARGET_SERVICES
else
$COMPOSE_CMD $COMPOSE_ARGS --env-file .env up -d --remove-orphans
fi
# ── Traefik reload ── # ── Traefik reload ──
if [ "${RELOAD_TRAEFIK:-false}" = "true" ]; then if [ "${RELOAD_TRAEFIK:-false}" = "true" ]; then
@@ -237,3 +235,20 @@ jobs:
fi fi
EOF EOF
# - name: Restart coolify-proxy (delay 1 min)
# env:
# SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
# VPS_HOST: ${{ secrets.VPS_HOST }}
# VPS_USER: ${{ secrets.VPS_USER }}
# run: |
# mkdir -p ~/.ssh
# echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
# chmod 600 ~/.ssh/id_rsa
# ssh-keyscan -H "$VPS_HOST" >> ~/.ssh/known_hosts
#
# echo "⏳ Waiting 60s for containers to settle before restarting proxy..."
# sleep 60
#
# echo "🔄 Restarting coolify-proxy..."
# ssh "${VPS_USER}@${VPS_HOST}" "docker restart coolify-proxy"
# echo "✅ coolify-proxy restarted."
+32 -36
View File
@@ -5,10 +5,7 @@ on:
branches: branches:
- main - main
paths: paths:
- 'apps/scraper/**' - 'apps/**'
- 'apps/hub/**'
- 'apps/tools/**'
- 'apps/llm-api/**'
- '.github/workflows/docker-build-push.yml' - '.github/workflows/docker-build-push.yml'
- 'infra/**' - 'infra/**'
- '!infra/compose/**' - '!infra/compose/**'
@@ -37,11 +34,11 @@ jobs:
outputs: outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }} matrix: ${{ steps.set-matrix.outputs.matrix }}
scraper-api: ${{ steps.filter.outputs['scraper-api'] == 'true' || steps.dispatch.outputs['scraper-api'] == 'true' || github.event_name == 'workflow_dispatch' }} scraper-api: ${{ steps.filter.outputs['scraper-api'] == 'true' || steps.dispatch.outputs['scraper-api'] == 'true' || github.event_name == 'workflow_dispatch' }}
hub: ${{ steps.filter.outputs['hub'] == 'true' || steps.dispatch.outputs['hub'] == 'true' || github.event_name == 'workflow_dispatch' }} elysia-api: ${{ steps.filter.outputs['elysia-api'] == 'true' || steps.dispatch.outputs['elysia-api'] == 'true' || github.event_name == 'workflow_dispatch' }}
tools: ${{ steps.filter.outputs['tools'] == 'true' || steps.dispatch.outputs['tools'] == 'true' || github.event_name == 'workflow_dispatch' }} react-web: ${{ steps.filter.outputs['react-web'] == 'true' || steps.dispatch.outputs['react-web'] == 'true' || github.event_name == 'workflow_dispatch' }}
llm-api: ${{ steps.filter.outputs['llm-api'] == 'true' || steps.dispatch.outputs['llm-api'] == 'true' || github.event_name == 'workflow_dispatch' }} rust-auth: ${{ steps.filter.outputs['rust-auth'] == 'true' || steps.dispatch.outputs['rust-auth'] == 'true' || github.event_name == 'workflow_dispatch' }}
steps: steps:
- uses: actions/checkout@v7 - uses: actions/checkout@v6
with: with:
submodules: false submodules: false
fetch-depth: 2 fetch-depth: 2
@@ -66,9 +63,9 @@ jobs:
} }
echo "scraper-api=$(changed '^(apps/scraper(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/scraper\.Dockerfile$)')" >> "$GITHUB_OUTPUT" echo "scraper-api=$(changed '^(apps/scraper(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/scraper\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
echo "hub=$(changed '^(apps/hub(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/hub\.Dockerfile$)')" >> "$GITHUB_OUTPUT" echo "elysia-api=$(changed '^(apps/elysia(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/elysia\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
echo "tools=$(changed '^(apps/tools(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/tools\.Dockerfile$)')" >> "$GITHUB_OUTPUT" echo "react-web=$(changed '^(apps/react(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/react\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
echo "llm-api=$(changed '^(apps/llm-api(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/llm-api\.Dockerfile$)')" >> "$GITHUB_OUTPUT" echo "rust-auth=$(changed '^(apps/rust-auth(/|$)|infra/docker/rust\.Dockerfile$|\.github/workflows/docker-build-push\.yml$|\.gitmodules$)')" >> "$GITHUB_OUTPUT"
- name: Parse repository_dispatch payload - name: Parse repository_dispatch payload
id: dispatch id: dispatch
@@ -90,15 +87,15 @@ jobs:
fi fi
case "$SERVICE" in case "$SERVICE" in
scraper-api|hub|tools|llm-api) ;; scraper-api|elysia-api|react-web|rust-auth) ;;
*) *)
exit 1 exit 1
;; ;;
esac esac
if ! [[ "$SHA" =~ ^[0-9a-fA-F]{40}$ ]]; then echo "::error::Invalid sha '$SHA'. Expected 40 hex characters"; fi if ! [[ "$SHA" =~ ^[0-9a-fA-F]{40}$ ]]; then echo "::error::Invalid sha '$SHA'. Expected 40 hex characters"; exit 1; fi
SERVICES=(scraper-api hub tools llm-api) SERVICES=(scraper-api elysia-api react-web rust-auth)
for svc in "${SERVICES[@]}"; do for svc in "${SERVICES[@]}"; do
if [ "$SERVICE" = "$svc" ]; then if [ "$SERVICE" = "$svc" ]; then
echo "${svc}=true" >> "$GITHUB_OUTPUT" echo "${svc}=true" >> "$GITHUB_OUTPUT"
@@ -116,9 +113,9 @@ jobs:
} }
if [ "${{ steps.filter.outputs['scraper-api'] == 'true' || steps.dispatch.outputs['scraper-api'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "scraper-api" "docker-scraper" "apps/scraper"; fi if [ "${{ steps.filter.outputs['scraper-api'] == 'true' || steps.dispatch.outputs['scraper-api'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "scraper-api" "docker-scraper" "apps/scraper"; fi
if [ "${{ steps.filter.outputs['hub'] == 'true' || steps.dispatch.outputs['hub'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "hub" "docker-hub" "apps/hub"; fi if [ "${{ steps.filter.outputs['elysia-api'] == 'true' || steps.dispatch.outputs['elysia-api'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "elysia-api" "docker-elysia" "apps/elysia"; fi
if [ "${{ steps.filter.outputs['tools'] == 'true' || steps.dispatch.outputs['tools'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "tools" "docker-tools" "apps/tools"; fi if [ "${{ steps.filter.outputs['react-web'] == 'true' || steps.dispatch.outputs['react-web'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "react-web" "docker-react" "apps/react"; fi
if [ "${{ steps.filter.outputs['llm-api'] == 'true' || steps.dispatch.outputs['llm-api'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "llm-api" "docker-llm-api" "apps/llm-api"; fi if [ "${{ steps.filter.outputs['rust-auth'] == 'true' || steps.dispatch.outputs['rust-auth'] == 'true' || github.event_name == 'workflow_dispatch' }}" == "true" ]; then add_service "rust-auth" "docker-rust-auth" "apps/rust-auth"; fi
JSON_ARRAY="[$(IFS=,; echo "${SERVICES[*]}")]" JSON_ARRAY="[$(IFS=,; echo "${SERVICES[*]}")]"
echo "matrix=$JSON_ARRAY" >> $GITHUB_OUTPUT echo "matrix=$JSON_ARRAY" >> $GITHUB_OUTPUT
@@ -138,9 +135,9 @@ jobs:
case "$SERVICE" in case "$SERVICE" in
"scraper-api") REPO="https://github.com/asepharyana/asepharyana-hub-scraper.git" ;; "scraper-api") REPO="https://github.com/asepharyana/asepharyana-hub-scraper.git" ;;
"hub") REPO="https://github.com/asepharyana/asepharyana-hub-hub.git" ;; "elysia-api") REPO="https://github.com/asepharyana/asepharyana-hub-elysia.git" ;;
"tools") echo "tools is built from monorepo, no submodule wait needed"; exit 0 ;; "react-web") REPO="https://github.com/asepharyana/asepharyana-hub-react.git" ;;
"llm-api") REPO="https://github.com/asepharyana/asepharyana-hub-llm-api.git" ;; "rust-auth") REPO="https://github.com/asepharyana/asepharyana-hub-rust-auth.git" ;;
*) *)
echo "::error::Unsupported service '$SERVICE'" echo "::error::Unsupported service '$SERVICE'"
exit 1 exit 1
@@ -185,7 +182,7 @@ jobs:
contents: read contents: read
packages: write packages: write
steps: steps:
- uses: actions/checkout@v7 - uses: actions/checkout@v6
with: with:
submodules: false submodules: false
@@ -201,7 +198,6 @@ jobs:
cd "$SUBMODULE_PATH" cd "$SUBMODULE_PATH"
git fetch origin "$DISPATCH_SHA" git fetch origin "$DISPATCH_SHA"
git checkout "$DISPATCH_SHA" git checkout "$DISPATCH_SHA"
cd "${GITHUB_WORKSPACE}"
fi fi
- uses: docker/login-action@v4 - uses: docker/login-action@v4
@@ -219,13 +215,13 @@ jobs:
SVC_NAME="${{ matrix.id }}" SVC_NAME="${{ matrix.id }}"
SHORT=${GITHUB_SHA:0:7} SHORT=${GITHUB_SHA:0:7}
echo "image=${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}" >> $GITHUB_OUTPUT echo "image=${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}" >> $GITHUB_OUTPUT
echo "tags=${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}:sha-${SHORT}" >> $GITHUB_OUTPUT echo "tags=${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}:latest,${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}:sha-${SHORT}" >> $GITHUB_OUTPUT
echo "cache-registry=${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}:buildcache" >> $GITHUB_OUTPUT echo "cache-registry=${REGISTRY}/${IMAGE_NAME_PREFIX}/${SVC_NAME}:buildcache" >> $GITHUB_OUTPUT
case "$SVC_NAME" in case "$SVC_NAME" in
"scraper-api") echo "dockerfile=infra/docker/scraper.Dockerfile" >> $GITHUB_OUTPUT ;; "scraper-api") echo "dockerfile=infra/docker/scraper.Dockerfile" >> $GITHUB_OUTPUT ;;
"hub") echo "dockerfile=infra/docker/hub.Dockerfile" >> $GITHUB_OUTPUT ;; "elysia-api") echo "dockerfile=infra/docker/elysia.Dockerfile" >> $GITHUB_OUTPUT ;;
"tools") echo "dockerfile=infra/docker/tools.Dockerfile" >> $GITHUB_OUTPUT ;; "react-web") echo "dockerfile=infra/docker/react.Dockerfile" >> $GITHUB_OUTPUT ;;
"llm-api") echo "dockerfile=infra/docker/llm-api.Dockerfile" >> $GITHUB_OUTPUT ;; "rust-auth") echo "dockerfile=infra/docker/rust.Dockerfile" >> $GITHUB_OUTPUT ;;
esac esac
- name: Build and Push Docker image - name: Build and Push Docker image
@@ -256,7 +252,7 @@ jobs:
permissions: permissions:
contents: write contents: write
steps: steps:
- uses: actions/checkout@v7 - uses: actions/checkout@v6
with: with:
submodules: false submodules: false
token: ${{ secrets.GITHUB_TOKEN }} token: ${{ secrets.GITHUB_TOKEN }}
@@ -270,15 +266,15 @@ jobs:
declare -A SERVICES declare -A SERVICES
SERVICES["scraper-api"]="scraper.yml" SERVICES["scraper-api"]="scraper.yml"
SERVICES["hub"]="hub.yml" SERVICES["elysia-api"]="elysia.yml"
SERVICES["tools"]="tools.yml" SERVICES["react-web"]="react.yml"
SERVICES["llm-api"]="llm-api.yml" SERVICES["rust-auth"]="rust-auth.yml"
declare -A PATHS declare -A PATHS
PATHS["scraper-api"]="apps/scraper" PATHS["scraper-api"]="apps/scraper"
PATHS["hub"]="apps/hub" PATHS["elysia-api"]="apps/elysia"
PATHS["tools"]="apps/tools" PATHS["react-web"]="apps/react"
PATHS["llm-api"]="apps/llm-api" PATHS["rust-auth"]="apps/rust-auth"
# Use git config for possible commits # Use git config for possible commits
git config --local user.email "action@github.com" git config --local user.email "action@github.com"
@@ -287,9 +283,9 @@ jobs:
for id in "${!SERVICES[@]}"; do for id in "${!SERVICES[@]}"; do
SHOULD_HAVE_RUN=false SHOULD_HAVE_RUN=false
if [ "${{ needs.changes.outputs['scraper-api'] }}" == "true" ] && [ "$id" == "scraper-api" ]; then SHOULD_HAVE_RUN=true; fi if [ "${{ needs.changes.outputs['scraper-api'] }}" == "true" ] && [ "$id" == "scraper-api" ]; then SHOULD_HAVE_RUN=true; fi
if [ "${{ needs.changes.outputs['hub'] }}" == "true" ] && [ "$id" == "hub" ]; then SHOULD_HAVE_RUN=true; fi if [ "${{ needs.changes.outputs['elysia-api'] }}" == "true" ] && [ "$id" == "elysia-api" ]; then SHOULD_HAVE_RUN=true; fi
if [ "${{ needs.changes.outputs['tools'] }}" == "true" ] && [ "$id" == "tools" ]; then SHOULD_HAVE_RUN=true; fi if [ "${{ needs.changes.outputs['react-web'] }}" == "true" ] && [ "$id" == "react-web" ]; then SHOULD_HAVE_RUN=true; fi
if [ "${{ needs.changes.outputs['llm-api'] }}" == "true" ] && [ "$id" == "llm-api" ]; then SHOULD_HAVE_RUN=true; fi if [ "${{ needs.changes.outputs['rust-auth'] }}" == "true" ] && [ "$id" == "rust-auth" ]; then SHOULD_HAVE_RUN=true; fi
if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then SHOULD_HAVE_RUN=true; fi if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then SHOULD_HAVE_RUN=true; fi
+10 -8
View File
@@ -5,24 +5,26 @@ on:
pull_request: pull_request:
branches: [main] branches: [main]
paths: paths:
- 'apps/*/src/**/*.ts'
- 'apps/*/src/**/*.tsx'
- 'biome.json' - 'biome.json'
- '*.json'
- '*.js'
push: push:
branches: [main] branches: [main]
paths: paths:
- 'apps/*/src/**/*.ts'
- 'apps/*/src/**/*.tsx'
- 'biome.json' - 'biome.json'
- '*.json'
- '*.js'
jobs: jobs:
biome: biome:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 10 timeout-minutes: 10
steps: steps:
- uses: actions/checkout@v7 - uses: actions/checkout@v6
with: with:
submodules: recursive submodules: recursive
- uses: oven-sh/setup-bun@v2 - uses: actions/setup-node@v4
- run: bun install --frozen-lockfile with:
- run: bun run ci node-version: 22
- run: npm install -g @biomejs/biome
- run: biome ci . --no-errors-on-unmatched
+4 -14
View File
@@ -13,18 +13,8 @@ jobs:
permissions: permissions:
security-events: write security-events: write
steps: steps:
- uses: actions/checkout@v7 - uses: actions/checkout@v6
- uses: github/codeql-action/init@v3
with: with:
fetch-depth: 2 languages: javascript-typescript, rust
submodules: recursive - uses: github/codeql-action/analyze@v3
- uses: github/codeql-action/init@v4
with:
languages: rust
- name: Build Rust projects for CodeQL analysis
run: |
cargo build --manifest-path apps/scraper/Cargo.toml
cargo build --manifest-path apps/llm-api/Cargo.toml
- uses: github/codeql-action/analyze@v4
+36
View File
@@ -0,0 +1,36 @@
name: TypeCheck
on:
pull_request:
branches: [main]
paths:
- 'apps/react/src/**/*.ts'
- 'apps/react/src/**/*.tsx'
- 'apps/elysia/src/**/*.ts'
- 'apps/elysia/tsconfig.json'
- 'tsconfig.base.json'
jobs:
typecheck:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
with:
submodules: recursive
- uses: oven/setup-bun@v2
with:
bun-version: latest
- name: TypeCheck apps/react
working-directory: apps/react
run: |
bun install
npx tsc --noEmit
- name: TypeCheck apps/elysia
working-directory: apps/elysia
run: |
bun install
bun run typecheck
+7 -13
View File
@@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: 10 timeout-minutes: 10
steps: steps:
- uses: actions/checkout@v7 - uses: actions/checkout@v6
- name: Validate payload - name: Validate payload
env: env:
@@ -36,7 +36,7 @@ jobs:
fi fi
case "$SERVICE" in case "$SERVICE" in
scraper-api|hub|llm-api|tools) ;; scraper-api|elysia-api|react-web|rust-auth) ;;
*) *)
echo "::error::Unsupported service '$SERVICE'" echo "::error::Unsupported service '$SERVICE'"
exit 1 exit 1
@@ -52,20 +52,14 @@ jobs:
run: | run: |
set -euo pipefail set -euo pipefail
# Map service name to submodule path echo "Updating ${SERVICE} to ${SHA}"
case "$SERVICE" in git submodule update --init "apps/${SERVICE}"
scraper-api) SUBMODULE_PATH="apps/scraper" ;; cd "apps/${SERVICE}"
llm-api) SUBMODULE_PATH="apps/llm-api" ;; # full fetch so we get tree objects for the target SHA
*) SUBMODULE_PATH="apps/${SERVICE}" ;;
esac
echo "Updating ${SUBMODULE_PATH} to ${SHA}"
git submodule update --init "${SUBMODULE_PATH}"
cd "${SUBMODULE_PATH}"
git fetch --depth=1 origin master 2>/dev/null || git fetch --depth=1 origin main git fetch --depth=1 origin master 2>/dev/null || git fetch --depth=1 origin main
git checkout "${SHA}" git checkout "${SHA}"
cd "${GITHUB_WORKSPACE}" cd "${GITHUB_WORKSPACE}"
git add "${SUBMODULE_PATH}" git add "apps/${SERVICE}"
git diff --cached --quiet && exit 0 git diff --cached --quiet && exit 0
git config user.name "monrepo-bot" git config user.name "monrepo-bot"
+3 -4
View File
@@ -1,3 +1,4 @@
apps/gmw/
# See https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files for more about ignoring files. # See https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files for more about ignoring files.
# compiled output # compiled output
@@ -34,18 +35,16 @@ npm-debug.log
yarn-error.log yarn-error.log
testem.log testem.log
/typings /typings
.playwright-mcp/
# System Files # System Files
.DS_Store .DS_Store
Thumbs.db Thumbs.db
.claude/* .claude
!.claude/skills/
!.claude/settings.json
# Next.js # Next.js
.next .next
out out
**/.codegraph/** **/.codegraph/**
**/.claude/**
test-output test-output
**/**.env **/**.env
**/**.env.** **/**.env.**
+9 -12
View File
@@ -1,15 +1,12 @@
[submodule "apps/elysia"]
path = apps/elysia
url = https://github.com/asepharyana/asepharyana-hub-elysia.git
[submodule "apps/react"]
path = apps/react
url = https://github.com/asepharyana/asepharyana-hub-react.git
[submodule "apps/scraper"] [submodule "apps/scraper"]
path = apps/scraper path = apps/scraper
url = https://github.com/asepharyana/asepharyana-hub-scraper.git url = https://github.com/asepharyana/asepharyana-hub-scraper.git
[submodule "apps/hub"] [submodule "apps/rust-auth"]
path = apps/hub path = apps/rust-auth
url = https://github.com/asepharyana/asepharyana-hub-hub.git url = https://github.com/asepharyana/asepharyana-hub-rust-auth.git
[submodule "apps/tools"]
path = apps/tools
url = https://github.com/asepharyana/asepharyana-hub-tools.git
[submodule "plugins/hub-guide"]
path = plugins/hub-guide
url = https://github.com/asepharyana/asepharyana-hub-guide.git
[submodule "apps/llm-api"]
path = apps/llm-api
url = https://github.com/asepharyana/asepharyana-hub-llm-api.git
-65
View File
@@ -1,65 +0,0 @@
---
name: commit-convention
description: Commit message convention — type(scope): description for Asepharyana Hub
---
# Commit Convention — Asepharyana Hub
## Format
```
<type>(<scope>): <description>
[optional body]
[optional footer]
```
## Types
| Type | Usage |
| ---------- | ------------------------------------ |
| `feat` | Fitur baru |
| `fix` | Bug fix |
| `chore` | Maintenance, config, tooling |
| `docs` | Dokumentasi |
| `refactor` | Perubahan kode tanpa fungsional baru |
| `test` | Nambah/update test |
| `ci` | CI/CD workflows |
| `perf` | Optimasi performa |
| `style` | Formatting (tanda kutip, dll) |
## Scopes
| Scope | Area |
| ------------- | --------------------------------- |
| `scraper` | apps/scraper submodule |
| `infra` | infra/ (compose, traefik, docker) |
| `ci` | .github/workflows/ |
| `dapr` | Dapr config & sidecar |
| `nats` | NATS message bus |
| `docs` | Dokumentasi |
| `deps` | Dependencies |
| `scripts` | Utility scripts |
| `root` | Root config files |
## Contoh
```
feat(scraper): add anime detail caching via Dapr pubsub
fix(infra): correct NATS CLI flags for JetStream
chore(deps): update biome to v2.5.3
docs(infra): add deployment order for Dapr services
ci(deploy): add nats.yml to ALL_COMPOSE_FILES
refactor(scraper): migrate EventBus from tokio broadcast to Dapr pubsub
```
## Aturan
1. **Wajib** menyertakan scope dalam tanda kurung
2. **Wajib** `Co-Authored-By` untuk commit yang digenerate AI
3. **Gunakan imperative mood**: "add" bukan "added" / "adds"
4. **Jangan capitalize** type: `feat:` bukan `Feat:`
5. **No period** di akhir subject baris
6. Body explain **why** dan **what**, bukan **how**
7. Refer issue dengan `Closes #123` atau `Fixes #123` di footer
-97
View File
@@ -1,97 +0,0 @@
---
name: deploy-workflow
description: CI/CD pipeline, Docker build patterns, manual deploy steps, and troubleshooting for Asepharyana Hub
---
# Deploy & Workflow — Asepharyana Hub
## CI/CD Pipeline
### Build Pipeline (`docker-build-push.yml`)
Trigger: push ke `main` yang touch `apps/**`, `infra/**`, `infra/docker/**`
1. **changes** — detect service mana yg berubah via git diff
2. **wait-submodule-ref** — (repository_dispatch only) tunggu SHA commit fetchable
3. **build** — matrix build per service, push ke GHCR (`sha-<short>` + `latest`)
4. **update-manifest** — update image tag di compose file, commit + push
### Deploy Pipeline (`deploy-docker.yml`)
Trigger: build selesai, atau push ke `main` touch `infra/**`
1. SSH ke `orangevps` (via `secrets.VPS_HOST`)
2. Sync repo (`git fetch --depth=1 + reset`)
3. Login ke GHCR
4. Deteksi compose file yg berubah
5. Pull images + restart container selektif
### Secrets Required
| Secret | Untuk |
|--------|-------|
| `SSH_PRIVATE_KEY` | SSH ke VPS |
| `VPS_HOST` | IP/host VPS (tailscale IP) |
| `VPS_USER` | SSH user, biasanya `root` |
| `VPS_TARGET_DIR` | Lokasi repo di VPS |
| `ENV_FILE_PRODUCTION` | .env content untuk production |
### Selective Deployment
- Hanya compose file yg berubah yang di-redeploy
- Selective: `UP_FLAGS="-d"` (tanpa `--remove-orphans`)
- Full deploy: `UP_FLAGS="-d --remove-orphans"`
## Docker Patterns
### Build dengan cargo-chef (Rust)
```dockerfile
FROM lukemathwalker/cargo-chef:latest-rust-1.89.0 AS chef
WORKDIR /app
FROM chef AS planner
COPY apps/scraper .
RUN cargo chef prepare --recipe-path recipe.json
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json
COPY apps/scraper .
RUN cargo build --release
```
### Runtime minimal untuk Rust binary
```dockerfile
FROM debian:bookworm-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl libssl3 && rm -rf /var/lib/apt/lists/*
```
## Manual Deploy Steps
```bash
# 1. Login GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u asepharyana --password-stdin
# 2. Full stack
docker compose -f infra/compose/traefik.yml \
-f infra/compose/shared.yml \
-f infra/compose/nats.yml \
-f infra/compose/dapr.yml \
-f infra/compose/scraper.yml \
--env-file .env up -d --remove-orphans
# 3. Selective (hanya satu service)
docker compose -f infra/compose/scraper.yml --env-file .env up -d
```
## Troubleshooting
### Container reach Tailscale
Pastikan route ke Tailscale di main table:
```bash
ip route add 100.64.0.0/10 dev tailscale0 table main
systemctl restart tailscale-routes
```
### Healthcheck gagal di scratch images
NATS dan Dapr placement pake scratch — tidak bisa healthcheck. Cukup `service_started` di depends_on.
### Dapr sidecar crash
```bash
docker logs scraper-api-dapr | grep -iE "fatal|error"
```
Penyebab umum: komponen config salah, NATS/Dapr placement belum siap.
-133
View File
@@ -1,133 +0,0 @@
---
name: event-driven
description: Event-driven architecture patterns with Dapr + NATS for Asepharyana Hub services
---
# Event-Driven Architecture — Asepharyana Hub
## Stack
- **Message Backbone**: NATS + JetStream (untuk streaming & job queue)
- **Pub/Sub Runtime**: Dapr sidecar per service (pubsub via Redis built-in)
- **State Store**: Dapr → Redis
## Event Topics Convention
```
hub.<domain>.<action>
Contoh:
hub.image.cached → Image selesai di-cache ke CDN
hub.image.repaired → Image diperbaiki (CNAME change)
hub.scrape.anime.done → Scrape anime selesai
hub.system.alert → Error/alert dari service
```
## CloudEvents Format
```json
{
"specversion": "1.0",
"type": "hub.image.cached",
"source": "scraper-api",
"subject": "anime-poster",
"id": "uuid-v4",
"time": "2026-07-21T10:00:00Z",
"datacontenttype": "application/json",
"data": { ... }
}
```
## Publish Event (Rust via HTTP API)
Gunakan `reqwest` langsung ke Dapr sidecar (SDK Rust masih experimental):
```rust
let event = serde_json::json!({
"specversion": "1.0",
"type": "hub.image.cached",
"source": "scraper-api",
"id": Uuid::new_v4().to_string(),
"time": chrono::Utc::now().to_rfc3339(),
"datacontenttype": "application/json",
"data": { "original_url": url, "cdn_url": cdn_url }
});
reqwest::Client::new()
.post("http://localhost:3500/v1.0/publish/pubsub/hub.image.cached")
.json(&event)
.send()
.await?;
```
## Service Invocation
```bash
curl http://localhost:3500/v1.0/invoke/<app-id>/method/<path>
```
## State Store
```bash
# Set
curl -X POST http://localhost:3500/v1.0/state/statestore \
-H "Content-Type: application/json" \
-d '[{"key": "mykey", "value": "myvalue"}]'
# Get
curl http://localhost:3500/v1.0/state/statestore/mykey
# Delete
curl -X DELETE http://localhost:3500/v1.0/state/statestore/mykey
```
## Scraper Event Integration
File yang perlu dimodifikasi untuk event-driven:
| File | Perubahan |
|------|-----------|
| `src/events/bus.rs` | Ganti backend dari tokio broadcast ke Dapr pub/sub |
| `src/bootstrap/mod.rs` | Init DaprClient, inject ke AppState |
| `src/presentation/state.rs` | Tambah `dapr_client` field |
| `src/proxy/use_cases.rs` | Publish `ImageRepaired` & `ImageCached` events |
| `src/infrastructure/services/images/cache.rs` | Emit event tiap cache selesai |
| `Cargo.toml` | Tambah `reqwest`, `uuid`, `chrono` (jika belum ada) |
## Event Handlers (Subscribe)
Buat `src/subscribers/` untuk handler:
```rust
// src/subscribers/image_handler.rs
pub async fn handle_image_cached(event: CloudEvent) -> Result<()> {
// Log, notifikasi, update status
}
```
Daftarkan subscribers di `bootstrap/mod.rs` dengan spawn task:
```rust
tokio::spawn(async move {
let mut stream = dapr_client.subscribe("pubsub", "hub.image.cached");
while let Some(event) = stream.next().await {
handle_image_cached(event).await;
}
});
```
## Testing Event-Driven Code
```rust
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_publish_event() {
let client = MockDaprClient::new();
client.expect_publish()
.with(...)
.returning(|_| Ok(()));
// ... test
}
}
```
-101
View File
@@ -1,101 +0,0 @@
---
name: hub-rules
description: Repository structure, submodule strategy, infrastructure patterns, and architecture of Asepharyana Hub
---
# Asepharyana Hub — Repository Rules
## Struktur Repository
```
asepharyana-hub/
├── apps/ # Git submodules — source code aplikasi
├── docs/ # Dokumentasi, ADR, deployment guide
├── infra/ # Infrastructure as code
│ ├── compose/ # Satu compose file per service
│ ├── dapr/ # Dapr component configs
│ ├── docker/ # Dockerfiles per service
│ └── traefik/ # Static & dynamic Traefik config
├── scripts/ # Utility scripts (cleanup, update-deps)
└── .github/workflows/ # CI/CD pipelines
```
### Aturan Submodule
- Setiap aplikasi di `apps/` adalah **submodule** ke repo terpisah.
- Perubahan kode aplikasi dilakukan di **repo masing-masing**, bukan di sini.
- Submodule pointer diupdate oleh CI/CD (bukan manual).
## Infrastructure Patterns
### Networking
- Semua service join **`app-shared-net`** (external Docker bridge)
- Service discovery via Docker DNS (container alias)
- Traefik sebagai ingress untuk HTTP/S eksternal
- Tailscale untuk cross-VPS (PostgreSQL, Redis)
### Compose File Pattern
```yaml
services:
<service>:
container_name: <service>
image: ghcr.io/asepharyana/asepharyana-hub/<service>:sha-<sha>
restart: always
networks:
app-shared-net:
aliases:
- <service>
env_file:
- ../../.env
networks:
app-shared-net:
name: app-shared-net
external: true
```
### Dapr Sidecar Pattern
```yaml
<service>-dapr:
container_name: <service>-dapr
image: daprio/daprd:latest
restart: always
depends_on:
nats:
condition: service_started
dapr-placement:
condition: service_started
networks:
- app-shared-net
command:
- './daprd'
- '--app-id=<service>'
- '--app-port=<port>'
- '--dapr-http-port=3500'
- '--dapr-grpc-port=50001'
- '--placement-host-address=dapr-placement:50005'
- '--resources-path=/components'
volumes:
- ../../infra/dapr/components:/components
```
### Traefik Routing
- Router + service definition di `infra/traefik/dynamic/apps.yaml`
- Subdomain pattern: `<service>.asepharyana.my.id` + `<service>.asepharya.web.id`
- TLS cert dari volume mount (bukan auto-acme)
### Image Tagging
- `sha-<short-sha>` — immutable, untuk rollback
- `latest` — mutable, untuk convenience
- Registry: `ghcr.io/asepharyana/asepharyana-hub/<service>`
### CI/CD
- `docker-build-push.yml` — build per service, push ke GHCR, update compose manifest
- `deploy-docker.yml` — SSH ke orangevps, pull images, restart
- Selective deploy: hanya compose file yg berubah
## Deployment Order
1. `shared.yml` (Redis)
2. `nats.yml` (NATS message bus)
3. `dapr.yml` (Dapr placement)
4. `traefik.yml` (Reverse proxy)
5. Service compose files (apps + Dapr sidecar)
+8
View File
@@ -0,0 +1,8 @@
{
"mcpServers": {
"next-devtools": {
"command": "npx",
"args": ["-y", "next-devtools-mcp@latest"]
}
}
}
+81
View File
@@ -0,0 +1,81 @@
# Agent Protocol v10: Strategic Execution and Diagnostic Directives
1. **Mandatory Executability.**
- **Directive:** All generated outputs must be syntactically correct and directly executable by the target system's interpreter, compiler, or shell.
- **Constraint:** Placeholders and pseudo-code are forbidden. Every output must be a complete, functional artifact.
2. **Data and State Integrity.**
- **Directive:** All generated code must strictly adhere to declared data structures, schemas, and the target language's type system.
- **Constraint:** Any operation producing a type mismatch, schema violation, or logical inconsistency is an invalid operation and must be discarded.
3. **Atomic and Consistent State Modification.**
- **Directive:** Modification of a shared resource must be performed as an atomic operation or within an ACID-compliant transaction.
- **Constraint:** Operations that can lead to race conditions or inconsistent state are prohibited. Immutability is the required default.
4. **Zero-Trust Security (Inviolable Safety Constraint).**
- **Directive:** Secrets must not be stored as literal values in source code. They must be loaded at runtime from a secure external source.
- **Constraint:** Generated access policies must adhere to the Principle of Least Privilege.
5. **Supply Chain Security (Inviolable Safety Constraint).**
- **Directive:** All external dependencies must be sourced from trusted repositories and defined in a lockfile for deterministic resolution.
- **Constraint:** The dependency graph must be scanned for known CVEs. Dependencies with critical vulnerabilities are prohibited.
6. **Deterministic and Reproducible Builds.**
- **Directive:** From a given source commit, the build process must produce a byte-for-byte identical artifact in every execution.
- **Constraint:** All automated tests must be deterministic. A regression test codifying the fixed bug's failure condition must be included with the fix.
7. **Structured, Traceable Logging.**
- **Directive:** All processes must emit structured (JSON) logs for significant events. All log entries for a request must contain the same unique trace ID.
- **Constraint:** Error conditions must be explicitly logged with context and propagated. Errors must not be silently suppressed.
8. **Strict API Contract Enforcement.**
- **Directive:** All network communication must strictly conform to its published, versioned API contract.
- **Constraint:** Any network call violating the contract must be rejected. Breaking changes require a major version increment (SemVer).
9. **Distributed System Consensus.**
- **Directive:** Changes to shared state across a distributed system are committed only after a formal consensus algorithm confirms quorum.
- **Constraint:** Nodes in a minority partition must enter a read-only or unavailable state to prevent a split-brain scenario.
10. **Execution Planning and Pre-flight Validation (Think Before Acting).**
- **Directive:** For any multi-step task, a detailed execution plan (sequence of commands and file modifications) must be formulated before any state-modifying action is taken.
- **Constraint:** Before executing a command, the agent must first use a validation or dry-run flag (e.g., `--dry-run`, `--check`) if available. The operation may only proceed if the pre-flight check passes without error.
11. **Post-Failure Root Cause Analysis (Evaluate Mistakes from Logs).**
- **Directive:** Upon command execution failure (non-zero exit code), the current execution plan must be halted, and the agent must enter a diagnostic mode.
- **Constraint:** In diagnostic mode, the agent is required to: 1) Capture and parse the complete `stdout` and `stderr` logs. 2) Identify the specific error message or stack trace. 3) Correlate the error with the last command to form a root cause hypothesis. 4) Formulate a new, corrective execution plan based on the analysis.
12. **Context-Aware File System Operations.**
- **Directive:** Before modifying any file, its full content must be read to establish context. All edits must be based on an in-memory understanding of the file's current state.
- **Constraint:** Blind file operations, such as stream-based search-and-replace without structural validation, are strictly prohibited.
13. **Idempotent State Transitions.**
- **Directive:** Operations that modify state must be designed to be idempotent wherever the protocol allows.
- **Constraint:** Executing the same operation multiple times must result in the same final system state as executing it only once.
14. **Resource Lifecycle Management.**
- **Directive:** All finite system resources (e.g., file handles, network sockets) must be explicitly released after use.
- **Constraint:** The agent must generate code that prevents resource leaks, utilizing language-specific constructs like `try-with-resources` or `defer`.
15. **Configuration as Code (CaC).**
- **Directive:** All configuration must be defined and versioned in source-controlled files.
- **Constraint:** Manual, out-of-band configuration changes are prohibited. Versioned files are the single source of truth.
16. **Atomic and Semantic Version Control.**
- **Directive:** All code changes must be organized into logically atomic commits representing one complete unit of work.
- **Constraint:** Commit messages must adhere to a defined specification (e.g., Conventional Commits).
17. **User Authority and Command Primacy.**
- **Directive:** User-provided instructions and corrections are the definitive source of truth and have the highest operational priority.
- **Constraint:** The agent must immediately adapt its process to align with user directives. Rejected solutions must not be proposed again.
18. **Precedent-Based Improvement.**
- **Directive:** User-approved outputs and successful patterns must be recorded and prioritized as precedents for subsequent tasks.
- **Constraint:** Performance, security, and code quality must not degrade.
19. **Optimization by Explicit Consent.**
- **Directive:** The agent may identify and propose optimizations with a technical justification and supporting metrics.
- **Constraint:** The agent is prohibited from applying any self-initiated optimization without an explicit "approve" command from the user.
20. **System Hierarchy and Safety Overrides.**
- **Directive:** The operational control hierarchy is absolute: 1) **User Command**, 2) **Inviolable Safety Directives (#4, #5)**, 3) **Standard Operational Directives**.
- **Constraint:** If a command conflicts with an Inviolable Directive, the agent must halt, report the conflict and risk, and await a revised command.
+1
View File
@@ -0,0 +1 @@
22.11.0
+8
View File
@@ -0,0 +1,8 @@
{
"recommendations": [
"nrwl.angular-console",
"biomejs.biome",
"ms-playwright.playwright",
"firsttris.vscode-jest-runner"
]
}
+23
View File
@@ -0,0 +1,23 @@
{
"[javascript]": {
"editor.defaultFormatter": "biomejs.biome"
},
"[javascriptreact]": {
"editor.defaultFormatter": "biomejs.biome"
},
"[typescript]": {
"editor.defaultFormatter": "biomejs.biome"
},
"[typescriptreact]": {
"editor.defaultFormatter": "biomejs.biome"
},
"[json]": {
"editor.defaultFormatter": "biomejs.biome"
},
"[jsonc]": {
"editor.defaultFormatter": "biomejs.biome"
},
"editor.codeActionsOnSave": {
"source.organizeImports.biome": "explicit"
}
}
+70 -52
View File
@@ -5,6 +5,9 @@
``` ```
asepharyana-hub/ asepharyana-hub/
├── apps/ # Application services (Git submodules) ├── apps/ # Application services (Git submodules)
│ ├── elysia/ # Realtime API (Bun/Elysia/Drizzle/Redis)
│ ├── react/ # Frontend SPA (React/Vite/TanStack)
│ ├── rust-auth/ # IAM & auth service (Axum/SeaORM)
│ └── scraper/ # Web scraper service │ └── scraper/ # Web scraper service
├── docs/ # Documentation ├── docs/ # Documentation
│ ├── adr/ # Architecture Decision Records │ ├── adr/ # Architecture Decision Records
@@ -28,11 +31,19 @@ asepharyana-hub/
## Technology Stack ## Technology Stack
### Services ### Backend Services
| Service | Language/Runtime | Framework | Database | Key Libraries | | Service | Language/Runtime | Framework | Database | Key Libraries |
| --------- | ---------------- | --------- | -------- | ------------- | | ------------- | ---------------- | ---------- | ------------------------ | ------------------------------------------------------------ |
| **scraper** | _(submodule)_ | — | — | — | | **rust-auth** | Rust | Axum 0.8 | PostgreSQL (sqlx) | sqlx, jsonwebtoken, argon2, redis, opentelemetry, prometheus |
| **elysia** | TypeScript/Bun | Elysia 1.4 | PostgreSQL (Drizzle ORM) | Drizzle ORM, Redis (ioredis), JWT (jose), OTel, Swagger |
| **scraper** | _(submodule)_ | — | — | — |
### Frontend
| Service | Framework | Build Tool | Key Libraries |
| --------- | --------- | ---------- | --------------------------------------------------------------------------------------------------- |
| **react** | React 19 | Vite 7 | TanStack Router + Query, Three.js/React Three Fiber, Tailwind CSS 4, Zustand, Recharts, tsParticles |
### Infrastructure ### Infrastructure
@@ -42,9 +53,7 @@ asepharyana-hub/
| Container Runtime | Docker + Docker Compose | Service isolation and orchestration | | Container Runtime | Docker + Docker Compose | Service isolation and orchestration |
| Container Registry | GHCR (ghcr.io) | Docker image storage | | Container Registry | GHCR (ghcr.io) | Docker image storage |
| Networking | Tailscale | Secure overlay network between VPS nodes | | Networking | Tailscale | Secure overlay network between VPS nodes |
| Message Bus | NATS + JetStream | Event-driven pub/sub, job queues, streaming | | Cache | Redis (Alpine) | Session store, rate limit counters, caching |
| Runtime Sidecar | Dapr | Service invocation, pub/sub abstraction, state management |
| Cache & State | Redis (Alpine) | Session store, rate limit counters, caching, Dapr state store |
| CI/CD | GitHub Actions | Build, test, deploy automation | | CI/CD | GitHub Actions | Build, test, deploy automation |
## Infrastructure ## Infrastructure
@@ -53,7 +62,7 @@ asepharyana-hub/
Traefik runs as the entry point for all HTTP/S traffic. It is configured via: Traefik runs as the entry point for all HTTP/S traffic. It is configured via:
- **Static config**: CLI arguments in `infra/compose/traefik.yml` — entry points, providers, plugins - **Static config**: `infra/traefik/traefik.yaml` — entry points, providers, plugins
- **Dynamic config**: `infra/traefik/dynamic/` — routers, services, middlewares, TLS - **Dynamic config**: `infra/traefik/dynamic/` — routers, services, middlewares, TLS
- **Docker provider**: Auto-discovers containers with `traefik.enable=true` labels - **Docker provider**: Auto-discovers containers with `traefik.enable=true` labels
- **File provider**: Loads `apps.yaml` (routers/services), `middlewares.yaml`, `ssl.yaml` - **File provider**: Loads `apps.yaml` (routers/services), `middlewares.yaml`, `ssl.yaml`
@@ -81,7 +90,7 @@ Shared services:
Service compose files are combined during deployment: Service compose files are combined during deployment:
```bash ```bash
docker compose -f traefik.yml -f shared.yml -f scraper.yml up -d docker compose -f traefik.yml -f shared.yml -f elysia.yml -f react.yml ... up -d
``` ```
### Tailscale Networking ### Tailscale Networking
@@ -89,9 +98,10 @@ docker compose -f traefik.yml -f shared.yml -f scraper.yml up -d
```mermaid ```mermaid
graph TB graph TB
subgraph "Tailnet (100.64.0.0/10)" subgraph "Tailnet (100.64.0.0/10)"
IMRNES["imrnes (100.121.180.82)"] IMRNES["imrnes (100.108.1.124)"]
ORANGEVPS["orangevps (100.79.111.61)"] ORANGE["orange (100.96.248.86)"]
ARCH["archlinux (100.84.39.83)"] ARCH["archlinux (100.114.19.66)"]
LAPTOP["laptop-2f6e1iph (100.86.195.29)"]
end end
subgraph "imrnes Services" subgraph "imrnes Services"
@@ -99,16 +109,28 @@ graph TB
REDIS[Redis] REDIS[Redis]
end end
subgraph "orangevps Containers" subgraph "orange Containers"
TRAEFIK[Traefik :443] TRAEFIK[Traefik :443]
RUST_AUTH[rust-auth :3000]
ELYSIA[elysia-api :4092]
REACT[react-web :80]
SCRAPER[scraper-api :4091] SCRAPER[scraper-api :4091]
end end
TRAEFIK --> RUST_AUTH
TRAEFIK --> ELYSIA
TRAEFIK --> REACT
TRAEFIK --> SCRAPER TRAEFIK --> SCRAPER
RUST_AUTH -.->|Tailscale IP| PG
ELYSIA -.->|Tailscale IP| PG
RUST_AUTH -.->|Tailscale IP| REDIS
ELYSIA -.->|Tailscale IP| REDIS
style IMRNES fill:#3a7,color:#fff style IMRNES fill:#3a7,color:#fff
style ORANGEVPS fill:#37a,color:#fff style ORANGE fill:#37a,color:#fff
style ARCH fill:#773,color:#fff style ARCH fill:#773,color:#fff
style LAPTOP fill:#777,color:#fff
``` ```
Container-to-Tailscale connectivity requires a systemd service that adds a route to the main routing table: Container-to-Tailscale connectivity requires a systemd service that adds a route to the main routing table:
@@ -117,7 +139,7 @@ Container-to-Tailscale connectivity requires a systemd service that adds a route
ip route add 100.64.0.0/10 dev tailscale0 table main ip route add 100.64.0.0/10 dev tailscale0 table main
``` ```
This is managed by `/etc/systemd/system/tailscale-routes.service` on the `orangevps` VPS. This is managed by `/etc/systemd/system/tailscale-routes.service` on the `orange` VPS.
## Data Flow ## Data Flow
@@ -133,7 +155,7 @@ sequenceDiagram
participant Redis as Redis (imrnes via Tailscale) participant Redis as Redis (imrnes via Tailscale)
User->>DNS: asepharyana.my.id User->>DNS: asepharyana.my.id
DNS->>User: A/AAAA record → orangevps VPS IP DNS->>User: A/AAAA record → orange VPS IP
User->>Traefik: HTTPS request :443 User->>Traefik: HTTPS request :443
Traefik->>Traefik: TLS termination Traefik->>Traefik: TLS termination
Traefik->>Traefik: Middleware chain (headers, rate-limit, buffer) Traefik->>Traefik: Middleware chain (headers, rate-limit, buffer)
@@ -143,8 +165,8 @@ sequenceDiagram
App->>DB: sqlx/Drizzle query via Tailscale App->>DB: sqlx/Drizzle query via Tailscale
DB-->>App: Result set DB-->>App: Result set
else Cache lookup else Cache lookup
App->>Cache: GET/SET via Tailscale App->>Redis: GET/SET via Tailscale
Cache-->>App: Cached value Redis-->>App: Cached value
end end
App-->>Traefik: HTTP response App-->>Traefik: HTTP response
@@ -193,7 +215,7 @@ flowchart LR
### VPS Deployment ### VPS Deployment
The `orangevps` VPS (Tailscale `100.79.111.61`) hosts all application containers: The `orange` VPS (Tailscale `100.96.248.86`) hosts all application containers:
1. GitHub Actions SSHes into the VPS 1. GitHub Actions SSHes into the VPS
2. Production secrets are written as `.env` 2. Production secrets are written as `.env`
@@ -206,7 +228,7 @@ The `orangevps` VPS (Tailscale `100.79.111.61`) hosts all application containers
### Selective Deployment ### Selective Deployment
The deploy workflow supports selective updates — if only one compose file changed, only the corresponding service is pulled and recreated, avoiding disruption to other services. The deploy workflow supports selective updates — if only `infra/compose/elysia.yml` changed, only `elysia-api` is pulled and recreated, avoiding disruption to other services.
```mermaid ```mermaid
graph TB graph TB
@@ -243,7 +265,7 @@ Each application lives in its own Git repository and is imported as a submodule
### Submodule Lifecycle ### Submodule Lifecycle
1. Developer pushes to a submodule (e.g., `apps/scraper`) 1. Developer pushes to a submodule (e.g., `apps/elysia`)
2. Submodule's GitHub Action dispatches `repository_dispatch` to the super-repo with the service name and new SHA 2. Submodule's GitHub Action dispatches `repository_dispatch` to the super-repo with the service name and new SHA
3. Super-repo detects the dispatch, waits for the SHA to be fetchable, then builds only that service 3. Super-repo detects the dispatch, waits for the SHA to be fetchable, then builds only that service
4. The compose manifest is updated and committed with the new SHA tag 4. The compose manifest is updated and committed with the new SHA tag
@@ -253,24 +275,21 @@ Each application lives in its own Git repository and is imported as a submodule
```bash ```bash
# Update a single submodule to latest # Update a single submodule to latest
cd apps/scraper cd apps/elysia
git checkout main git checkout main
git pull git pull
cd ../.. cd ../..
git add apps/scraper git add apps/elysia
git commit -m "chore(scraper): update submodule to latest" git commit -m "chore(elysia): update submodule to latest"
# Update all submodules
git submodule update --remote --merge
``` ```
## Service Mesh & Inter-Service Communication ## Service Mesh & Inter-Service Communication
### HTTP (External + Internal via Traefik)
External traffic and internal HTTP calls route through Traefik. Services on `app-shared-net` can also communicate directly by container name.
### Event-Driven (NATS + Dapr)
NATS with JetStream provides a persistent message backbone. Each service has a Dapr sidecar that abstracts pub/sub, service invocation, and state management.
```mermaid ```mermaid
graph TB graph LR
subgraph "External" subgraph "External"
WWW[Internet] WWW[Internet]
end end
@@ -279,36 +298,35 @@ graph TB
TRAEFIK[Traefik :443] TRAEFIK[Traefik :443]
subgraph "app-shared-net" subgraph "app-shared-net"
NATS[NATS + JetStream<br/>:4222] REACT[react-web<br/>:80]
DAPR_PLACEMENT[Dapr Placement<br/>:50005] ELYSIA[elysia-api<br/>:4092]
RUST_AUTH[rust-auth<br/>:3000]
subgraph "Service: scraper-api" SCRAPER[scraper-api<br/>:4091]
SCRAPER[scraper-api<br/>:4091] REDIS[redis<br/>:6379]
DAPR_SIDECAR[Dapr Sidecar<br/>:3500]
SCRAPER --- DAPR_SIDECAR
end
end end
end
DAPR_SIDECAR -.->|gRPC pub/sub| NATS subgraph "Imrnes VPS"
DAPR_SIDECAR -.->|placement| DAPR_PLACEMENT PG[(PostgreSQL<br/>:5432)]
REDIS_IMR[Redis<br/>:6379]
end end
WWW -->|HTTPS| TRAEFIK WWW -->|HTTPS| TRAEFIK
TRAEFIK --> REACT
TRAEFIK --> ELYSIA
TRAEFIK --> RUST_AUTH
TRAEFIK --> SCRAPER TRAEFIK --> SCRAPER
ELYSIA -->|Tailscale| PG
RUST_AUTH -->|Tailscale| PG
ELYSIA -->|internal| REDIS
RUST_AUTH -->|internal| REDIS
ELYSIA -->|Tailscale| REDIS_IMR
RUST_AUTH -->|Tailscale| REDIS_IMR
``` ```
### Communication Patterns
| Pattern | Mechanism | Use Case |
|---------|-----------|----------|
| External HTTP | Traefik → Service | User requests, API calls |
| Internal HTTP | Service → Service (via Traefik or direct) | Synchronous queries |
| Pub/Sub Event | Dapr sidecar → NATS JetStream | Async notifications, image cache events |
| Service Invocation | Dapr sidecar gRPC | Cross-service RPC with retry & observability |
| State Store | Dapr → Redis | Shared state, job progress |
## Observability ## Observability
- **Traefik access logs**: JSON format, logged at INFO level - **Prometheus metrics**: Available on rust-auth via `axum-prometheus`
- **Traefik access logs**: JSON format, logged at INFO level - **Traefik access logs**: JSON format, logged at INFO level
- **Dashboard**: Traefik dashboard at `traefik.asepharyana.my.id` (secured) - **Dashboard**: Traefik dashboard at `traefik.asepharyana.my.id` (secured)
-14
View File
@@ -12,22 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Restructured the repository into a lightweight hub repo with standalone app submodules. - Restructured the repository into a lightweight hub repo with standalone app submodules.
- Simplified root tooling to plain `package.json` scripts and per-service commands. - Simplified root tooling to plain `package.json` scripts and per-service commands.
- Kept infrastructure, deployment workflows, and documentation in the root hub repo. - Kept infrastructure, deployment workflows, and documentation in the root hub repo.
- Cleaned up Traefik SSL config: removed legacy `asephstech`/`asephscloud` cert references, synced volume mounts, fixed `api.insecure`.
- Pruned `.env.example` from 145 legacy vars (Firebase, Discord, Coolify, Portainer, YouTube, etc.) to 30 focused vars.
- Optimized `scraper.Dockerfile`: removed Node.js and Chromium from runtime image.
- Simplified CI/CD workflows: removed orphan container reference, commented code blocks.
- Cleaned up scripts: removed stale MySQL config, fixed package references, simplified update-deps.
- Added NATS + JetStream message broker infrastructure (`infra/compose/nats.yml`).
- Added Dapr runtime infrastructure: placement service, sidecar pattern, pub/sub + state store components.
- Integrated Dapr sidecar into scraper service (`infra/compose/scraper.yml`).
- Updated deployment order: shared → NATS → Dapr → Traefik → apps.
- Added `docs/add-dapr-service.md` guide for adding Dapr to new services.
### Removed ### Removed
- Removed deprecated services from apps, compose files, Dockerfiles, Traefik routes, and workflows. - Removed deprecated services from apps, compose files, Dockerfiles, Traefik routes, and workflows.
- Removed stale monorepo orchestration configs and hook tooling from the root repo. - Removed stale monorepo orchestration configs and hook tooling from the root repo.
- Removed `.nvmrc` (duplicate of `.node-version`), `renovate.json` (using dependabot).
- Removed stale documentation: `dependency-map.md`, `observability.md`, `handoff-log.jsonl`, `quality-gates.json`, `workflow-state.json`.
- Removed deprecated `docs/superpowers/` design docs.
- Removed `infra/config/mysql/` (no active MySQL service) and `docs/config/squid.conf.archived`.
-131
View File
@@ -1,131 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Repository Overview
Asepharyana Hub is a **hub monorepo** for Asep Haryana Saputra's portfolio ecosystem. Application services live in separate repos imported as Git submodules under `apps/`. Infrastructure (Docker Compose, Traefik, Dapr) lives in `infra/`.
```
asepharyana-hub/
├── apps/ # Git submodules — each app is its own repo
│ ├── hub/ # Personal portfolio SPA (asepharyana-hub-hub)
│ └── scraper/ # Rust scraper API (asepharyana-hub-scraper)
├── docs/ # ADRs, deployment guide, new-app guide
├── infra/
│ ├── compose/ # One Docker Compose file per service
│ ├── dapr/ # Dapr config + component definitions
│ ├── docker/ # Dockerfiles per service
│ └── traefik/ # Reverse proxy config (static + dynamic)
├── scripts/ # Utility scripts (cleanup, update-deps, git hooks)
└── .github/workflows/ # CI/CD pipelines
```
### Submodule Strategy
- Each app in `apps/` is a separate Git repo imported as a submodule. Code changes happen in the submodule repo, not here.
- Submodule pointers are updated by CI/CD (via `repository_dispatch` or manual commit).
- Current submodules:
- `apps/hub``asepharyana/asepharyana-hub-hub`
- `apps/scraper``asepharyana/asepharyana-hub-scraper`
- `apps/llm-api``asepharyana/asepharyana-hub-llm-api`
- `apps/tools``asepharyana/asepharyana-hub-tools`.
### Infrastructure Stack
- **Traefik v3.6** — reverse proxy, TLS termination, middleware chain, Prometheus metrics (`--metrics.prometheus=true`)
- **NATS + JetStream** — message broker with persistent streaming
- **Dapr** — sidecar runtime (pub/sub abstraction, state management, service invocation)
- **Redis (Alpine)** — cache, session store, Dapr state store & pub/sub backend
- **Prometheus** — metrics backend with Docker service discovery (`docker_sd_configs`). Auto-discovers containers with `prometheus.io/scrape=true` label.
- **Jaeger** — distributed tracing backend (all-in-one), OTLP receiver
- **Tailscale** — secure overlay network between VPS nodes (PostgreSQL on `imrnes`, containers on `orangevps`)
### Monitoring
- **Hub dashboard** at `/dashboard` (Next.js client page, auto-refresh 15s)
- **Dashboard API** at `/api/dashboard` — returns JSON with Docker containers, Jaeger traces, Prometheus metrics (RPS, latency, errors, node CPU/RAM/Disk)
- **Docker socket** mounted on `hub` container (`--group-add 988`) for container discovery
- **Prometheus** auto-scrapes Traefik for per-service request metrics
### Networking
- All containers join `app-shared-net` (external Docker bridge network). Service discovery via Docker DNS (container name aliases).
- Traefik handles all external HTTP/S traffic on port 443.
- Cross-VPS traffic (DB, Redis) goes through Tailscale (`100.64.0.0/10`). Container-to-Tailscale connectivity requires a route in the main routing table (managed by `tailscale-routes.service`).
## Commands
```bash
make init-submodules # Initialize submodules after clone
make dev # Start shared dev infrastructure (Redis)
make update-submodules # Update all submodules to latest
bun run check # Biome lint + format + write
bun run ci # Biome CI mode (no writes, exit code on issues)
bun run format # Format only
bun run lint # Lint only
docker build -f infra/docker/scraper.Dockerfile -t scraper-api:latest . # Build image
```
### Validate YAML
```bash
python -c "import pathlib, yaml; [yaml.safe_load(open(p)) for p in pathlib.Path('infra').rglob('*.yml')]"
for f in infra/compose/*.yml; do docker compose -f "$f" config >/dev/null && echo "OK $f"; done
```
## CI/CD Workflows
| Workflow | Trigger | Action |
|----------|---------|--------|
| `lint.yml` | PR/push to main touching `*.json`, `*.js`, `biome.json` | `bun run ci` (Biome lint) |
| `docker-build-push.yml` | Push to main touching `apps/**`/`infra/**`, or `repository_dispatch` | Build Docker images per changed service, push to GHCR, update compose manifests |
| `deploy-docker.yml` | After build completes, or push touching `infra/**` | SSH to VPS (orangevps), pull images, restart containers selectively |
| `security.yml` | PR to main + weekly Monday | CodeQL analysis (Rust) |
| `update-submodule.yml` | `repository_dispatch` | Update submodule pointer in hub repo |
### Deployment Order
1. `shared.yml` (Redis)
2. `nats.yml` (NATS + JetStream)
3. `dapr.yml` (Dapr placement)
4. `traefik.yml` (Reverse proxy)
5. Service compose files (app + Dapr sidecar)
### Secrets Required for Deploy
`SSH_PRIVATE_KEY`, `VPS_HOST`, `VPS_USER`, `VPS_TARGET_DIR`, `ENV_FILE_PRODUCTION`
## Infrastructure Patterns
### Compose File Pattern
Each service gets one compose file. Containers join `app-shared-net` with a `container_name` alias for DNS. The network is declared `external: true`.
### Dapr Sidecar Pattern
Each app gets a companion `daprd` sidecar container. Dapr components (pubsub, statestore) are mounted from `infra/dapr/components/`. The sidecar communicates with NATS for pub/sub and Dapr placement for actor coordination.
### Traefik Routing
- Routers + services defined in `infra/traefik/dynamic/apps.yaml`
- Subdomain pattern: `<service>.asepharyana.my.id` and `<service>.asepharya.web.id`
- TLS certs from volume mounts (not auto-ACME)
- Middleware chain: `secure-headers``compress``retry``rate-limit``buffer`
### Image Tagging
- `sha-<short-sha>` — immutable, for deterministic rollbacks
- `latest` — mutable, for convenience
- Registry: `ghcr.io/asepharyana/asepharyana-hub/<service>`
- Build cache: `sha-<short>-buildcache` (registry-based caching)
## Adding a New Service
1. Create a separate repo for the app code
2. Add as submodule: `git submodule add <url> apps/<name>`
3. Create Dockerfile in `infra/docker/`
4. Create compose file in `infra/compose/` (app + Dapr sidecar)
5. Add Traefik router in `infra/traefik/dynamic/apps.yaml`
6. Add build job in `.github/workflows/docker-build-push.yml`
7. See `docs/add-new-app.md` for full guide
## Commit Convention
Format: `<type>(<scope>): <description>`
Types: `feat`, `fix`, `chore`, `docs`, `refactor`, `test`, `ci`, `perf`, `style`
Scopes: `scraper`, `infra`, `ci`, `dapr`, `nats`, `docs`, `deps`, `scripts`, `root`
Scope is required. Use imperative mood. No period at end of subject line. Co-Authored-By footer for AI-generated commits.
+69 -17
View File
@@ -14,8 +14,9 @@
## Prerequisites ## Prerequisites
- **Git** with LFS support - **Git** with LFS support
- **Node.js** >= 22.11.0 (via `.node-version`) - **Node.js** >= 22.11.0 (via `.node-version` or `.nvmrc`)
- **Bun** >= 1.3.11 (package manager) - **Bun** >= 1.3.11 (package manager)
- **Rust** >= 1.89.0 (for Rust services)
- **Docker** and **Docker Compose** (for shared infrastructure) - **Docker** and **Docker Compose** (for shared infrastructure)
## Local Setup ## Local Setup
@@ -39,14 +40,18 @@ This checks out all submodules at the pinned commit (not `main`). The submodules
| Path | Remote | | Path | Remote |
| ---------------- | --------------------------------------- | | ---------------- | --------------------------------------- |
| `apps/elysia` | `asepharyana/asepharyana-hub-elysia` |
| `apps/scraper` | `asepharyana/asepharyana-hub-scraper` | | `apps/scraper` | `asepharyana/asepharyana-hub-scraper` |
| `apps/react` | `asepharyana/asepharyana-hub-react` |
| `apps/rust-auth` | `asepharyana/asepharyana-hub-rust-auth` |
### 3. Install Dependencies per Service ### 3. Install Dependencies per Service
Install dependencies for TypeScript/Bun services: Install dependencies for TypeScript/Bun services:
```bash ```bash
cd apps/scraper && bun install && cd ../.. cd apps/elysia && bun install && cd ../..
cd apps/react && npm install && cd ../..
``` ```
### 4. Start Shared Infrastructure ### 4. Start Shared Infrastructure
@@ -70,42 +75,76 @@ Key variables to configure:
| Variable | Description | | Variable | Description |
| -------------- | ----------------------------------------------------- | | -------------- | ----------------------------------------------------- |
| `DATABASE_URL` | PostgreSQL connection (Tailscale IP to `imrnes` VPS) | | `DATABASE_URL` | PostgreSQL connection (Tailscale IP to `imrnes` VPS) |
| `REDIS_URL` | Redis connection (`redis://localhost:6379` for local) |
| `JWT_SECRET` | JWT signing secret |
| `GITHUB_TOKEN` | GitHub personal access token | | `GITHUB_TOKEN` | GitHub personal access token |
## Development Workflow ## Development Workflow
### Running Services ### Running Services
Refer to each service's own documentation for setup and development instructions. **Rust API (rust-auth):**
```bash
cd apps/rust-auth
cargo run
```
**Elysia API (elysia):**
```bash
cd apps/elysia
bun run dev
```
**React Frontend (react):**
```bash
cd apps/react
npm run dev
```
### API Documentation ### API Documentation
Refer to each service's own documentation for API docs and endpoints. - Rust OpenAPI: `http://localhost:4091/docs`
- Elysia Swagger: `http://localhost:4092/docs`
- Elysia AsyncAPI: `http://localhost:4092/docs-ws`
## Coding Standards ## Coding Standards
### Linting ### Linting
- **Biome** for TypeScript/JavaScript formatting and linting - **ESLint** with `@antfu/eslint-config` for TypeScript/JavaScript
- **Cargo Clippy** for Rust
Run linting: Run linting:
```bash ```bash
# TypeScript/JavaScript # TypeScript/JavaScript
bun run check eslint . --no-error-on-unmatched-pattern
# Rust specific
cd apps/rust-auth && cargo clippy -- -D warnings
``` ```
### Formatting ### Formatting
- **Biome** for TypeScript/JavaScript - **Prettier** for TypeScript/JavaScript/Markdown (config in `.prettierrc`)
- Single quotes, 100 print width, 2-space indent, trailing commas
- **Cargo fmt** for Rust
- **EditorConfig** for general formatting (`.editorconfig`) - **EditorConfig** for general formatting (`.editorconfig`)
```bash ```bash
# Format all # Prettier
bun run format prettier --write .
# Rust
cd apps/rust-auth && cargo fmt
``` ```
### Rust Configuration
Rust services use edition `2024` with stable toolchain (nightly features may be used).
## Commit Message Format ## Commit Message Format
@@ -138,15 +177,18 @@ This project enforces **Conventional Commits** for all commit messages.
### Examples ### Examples
``` ```
feat(scraper): add new data source integration feat(rust-auth): add OAuth2 Google login flow
chore: update biome config to v10 fix(elysia): handle null JWT payload in auth middleware
chore: update eslint config to v10
docs: add API endpoint documentation for scraper docs: add API endpoint documentation for scraper
refactor(react): extract Header component from App
test(elysia): add unit tests for rate limiter
ci: migrate to CodeQL v3 ci: migrate to CodeQL v3
``` ```
### Scopes ### Scopes
Common scopes: `scraper`, `infra`, `ci`, `deps` Common scopes: `rust-auth`, `elysia`, `react`, `scraper`, `infra`, `ci`, `deps`
## Pull Request Process ## Pull Request Process
@@ -159,9 +201,10 @@ Common scopes: `scraper`, `infra`, `ci`, `deps`
3. **Run checks locally** before pushing: 3. **Run checks locally** before pushing:
```bash ```bash
bun run check cd apps/react && npx tsc --noEmit
``` eslint . --no-error-on-unmatched-pattern
```
4. **Push and open a PR** against `main`. CI will automatically run: 4. **Push and open a PR** against `main`. CI will automatically run:
- **Lint** — ESLint across changed TypeScript files - **Lint** — ESLint across changed TypeScript files
@@ -175,7 +218,7 @@ Common scopes: `scraper`, `infra`, `ci`, `deps`
- Updates compose manifests to use the new SHA tags - Updates compose manifests to use the new SHA tags
6. **Deployment Pipeline** triggers after a successful Docker build: 6. **Deployment Pipeline** triggers after a successful Docker build:
- SSHes into the VPS (`orangevps`, Tailscale IP `100.79.111.61`) - SSHes into the VPS (`orange`, Tailscale IP `100.96.248.86`)
- Pulls updated Docker images - Pulls updated Docker images
- Recreates only the changed containers - Recreates only the changed containers
- All services share the `app-shared-net` Docker network - All services share the `app-shared-net` Docker network
@@ -184,4 +227,13 @@ Common scopes: `scraper`, `infra`, `ci`, `deps`
## Adding a New Service ## Adding a New Service
See `docs/add-new-app.md` for the complete step-by-step guide. See `docs/add-new-app.md` for the complete step-by-step guide. In summary:
1. Create the app in `apps/<name>`
2. Add it as a Git submodule in `.gitmodules`
3. Register it in `infra/compose/<name>.yml`
4. Add a Dockerfile at `infra/docker/<name>.Dockerfile`
5. Add Traefik routing config in `infra/traefik/dynamic/apps.yaml`
6. Add CI entries in `.github/workflows/docker-build-push.yml`
7. Add compose file to the deploy script in `deploy-docker.yml`
8. Add any required GitHub secrets for the service
+11 -1
View File
@@ -1,4 +1,4 @@
.PHONY: help dev update-submodules deploy init-submodules status .PHONY: help dev lint format test clean update-submodules deploy init-submodules status
SHELL := /bin/bash SHELL := /bin/bash
@@ -8,6 +8,16 @@ help: ## Show this help
dev: ## Start development infrastructure (Redis etc.) dev: ## Start development infrastructure (Redis etc.)
docker compose -f infra/compose/shared.yml up -d docker compose -f infra/compose/shared.yml up -d
lint: ## Run Biome linter
biome lint .
format: ## Format code with Biome
biome format --write .
clean: ## Clean build artifacts
rm -rf apps/*/dist apps/*/.next apps/*/target 2>/dev/null || true
rm -rf node_modules 2>/dev/null || true
update-submodules: ## Update all git submodules to latest remote update-submodules: ## Update all git submodules to latest remote
git submodule update --remote --merge --recursive git submodule update --remote --merge --recursive
+45 -21
View File
@@ -5,21 +5,20 @@ Aplikasi dipisah sebagai submodule agar frontend, API, dan service pendukung bis
## Services ## Services
| Service | Path | Notes | | Service | Path | Default Local Port | Notes |
| :------ | :------------- | :----------------------------- | | :---------- | :--------------- | :----------------- | :---------------------------------------------------------------------------- |
| Scraper | `apps/scraper` | Web scraper service + Dapr SDK | | Rust API | `apps/rust-auth` | `4091` | API utama (Axum + SeaORM), scraping, image proxy/cache, metrics, OpenAPI docs |
| NATS | — | Message broker + JetStream | | Elysia API | `apps/elysia` | `4092` | API realtime/auth/chat/quiz (Elysia + Bun + Drizzle + Redis) |
| Dapr | — | Sidecar runtime (per service) | | React Web | `apps/react` | `3000` | Frontend React/Vite |
| Scraper | `apps/scraper` | — | Web scraper service |
## Infrastructure ## Infrastructure
File compose berada di `infra/compose/`: File compose berada di `infra/compose/`:
- `traefik.yml`: reverse proxy Traefik untuk semua layanan. - `traefik.yml`: reverse proxy Traefik untuk semua layanan.
- `shared.yml`: Redis (cache + Dapr state store). - `shared.yml`: Redis.
- `nats.yml`: NATS message broker dengan JetStream persistence. - `rust-auth.yml`, `elysia.yml`, `react.yml`, `scraper.yml`: manifest deploy per service (image GHCR bertag SHA).
- `dapr.yml`: Dapr placement service untuk koordinasi sidecar.
- `scraper.yml`: manifest deploy per service (app + Dapr sidecar).
Dockerfile per service berada di `infra/docker/`. Dockerfile per service berada di `infra/docker/`.
@@ -28,6 +27,9 @@ Dockerfile per service berada di `infra/docker/`.
Build image via Dockerfile: Build image via Dockerfile:
```bash ```bash
docker build -f infra/docker/rust.Dockerfile -t rust-auth:latest .
docker build -f infra/docker/elysia.Dockerfile -t elysia-api:latest .
docker build -f infra/docker/react.Dockerfile -t react-web:latest .
docker build -f infra/docker/scraper.Dockerfile -t scraper-api:latest . docker build -f infra/docker/scraper.Dockerfile -t scraper-api:latest .
``` ```
@@ -36,8 +38,9 @@ Tag and push:
```bash ```bash
SHORT_SHA=$(git rev-parse --short HEAD) SHORT_SHA=$(git rev-parse --short HEAD)
docker tag scraper-api:latest ghcr.io/asepharyana/asepharyana-hub/scraper-api:sha-$SHORT_SHA docker tag rust-auth:latest ghcr.io/asepharyana/asepharyana-hub/rust-auth:sha-$SHORT_SHA
docker push ghcr.io/asepharyana/asepharyana-hub/scraper-api:sha-$SHORT_SHA docker push ghcr.io/asepharyana/asepharyana-hub/rust-auth:sha-$SHORT_SHA
# repeat for elysia-api, react-web, scraper-api
``` ```
## Local Development ## Local Development
@@ -50,11 +53,27 @@ docker compose -f infra/compose/shared.yml up -d
### 2) Jalankan service yang dibutuhkan ### 2) Jalankan service yang dibutuhkan
Refer to each service's own documentation for development setup. ```bash
# Rust API
cd apps/rust-auth
cargo run
# Elysia API
cd apps/elysia
bun install
bun run dev
# React web
cd apps/react
npm install
npm run dev
```
## API Docs and Monitoring ## API Docs and Monitoring
Refer to each service's own documentation for API docs. - Rust OpenAPI: `/docs`
- Elysia Swagger: `/docs`
- Elysia AsyncAPI viewer: `/docs-ws`
## Deployment Notes ## Deployment Notes
@@ -67,11 +86,12 @@ Refer to each service's own documentation for API docs.
Semua VPS terhubung via **Tailscale**. Setiap VPS punya IP Tailscale dan service berkomunikasi antar VPS melalui Tailscale network (`100.64.0.0/10`). Semua VPS terhubung via **Tailscale**. Setiap VPS punya IP Tailscale dan service berkomunikasi antar VPS melalui Tailscale network (`100.64.0.0/10`).
| VPS | Tailscale IP | Service | | VPS | Tailscale IP | Service |
| :--------- | :-------------- | :------------------------------------- | | :---------------- | :-------------- | :------------------------------------- |
| `imrnes` | `100.121.180.82` | PostgreSQL, Redis | | `imrnes` | `100.108.1.124` | PostgreSQL (`hub`), Redis |
| `orangevps` | `100.79.111.61` | App containers (Traefik, scraper-api) | | `orange` | `100.96.248.86` | App containers (Traefik, 9Router, dll) |
| `archlinux` | `100.84.39.83` | _(development machine)_ | | `archlinux` | `100.114.19.66` | _(development machine)_ |
| `laptop-2f6e1iph` | `100.86.195.29` | _(offline)_ |
### Container → Tailscale Connectivity ### Container → Tailscale Connectivity
@@ -98,10 +118,10 @@ Service yang connect ke Tailscale IP:
```env ```env
# PostgreSQL di imrnes # PostgreSQL di imrnes
DATABASE_URL=postgres://user:pass@100.121.180.82:5432/dbname DATABASE_URL=postgres://user:pass@100.108.1.124:5432/dbname
# Redis di imrnes # Redis di imrnes
REDIS_URL=redis://100.121.180.82:6379 REDIS_URL=redis://100.108.1.124:6379
``` ```
#### Persistent Systemd Service #### Persistent Systemd Service
@@ -163,7 +183,7 @@ docker exec <container> node -e "
const net = require('net'); const net = require('net');
const c = new net.Socket(); const c = new net.Socket();
c.setTimeout(5000); c.setTimeout(5000);
c.connect(5432, '100.121.180.82', () => { console.log('OK'); c.end(); }); c.connect(5432, '100.108.1.124', () => { console.log('OK'); c.end(); });
c.on('error', e => { console.log('FAIL:', e.code); }); c.on('error', e => { console.log('FAIL:', e.code); });
c.on('timeout', () => { console.log('TIMEOUT'); c.destroy(); }); c.on('timeout', () => { console.log('TIMEOUT'); c.destroy(); });
" "
@@ -179,3 +199,7 @@ Panduan langkah demi langkah untuk menambahkan aplikasi baru ada di `docs/add-ne
## License ## License
MIT MIT
## Test Section
This is a test for PR-Agent auto-review.
Submodule
+1
Submodule apps/elysia added at b8f4b5806e
Submodule apps/hub deleted from 51798f483f
Submodule apps/llm-api deleted from d8425ea3b3
Submodule
+1
Submodule apps/react added at 124d2f88bd
+1
Submodule apps/rust-auth added at 6e695f07e8
Submodule apps/tools deleted from 3956b90c3c
-238
View File
@@ -1,238 +0,0 @@
# Arsitektur asepharyana-hub
## Topologi Fisik
Dua node terhubung via **Tailscale** overlay network:
```
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ orangevps (VPS) │ │ imrnes (Bare-metal) │
│ IP: 45.127.35.244 │ │ Tailscale: 100.121.180.82 │
│ Tailscale: 100.x.x.x │◄──────┤ │
│ │ │ Layanan: │
│ Layanan: │ │ ├─ PostgreSQL (port 6432) │
│ ├─ Traefik (port 80/443) │ │ └─ Redis (port 6379) │
│ ├─ NATS + JetStream │ │ │
│ ├─ Dapr Placement │ └──────────────────────────────┘
│ ├─ Redis (cache, Dapr) │
│ ├─ Scraper API + Dapr │
│ └─ Hub (Next.js SPA) │
└──────────────────────────────┘
```
### Konektivitas Container ke Tailscale
Container di `orangevps` tidak bisa langsung mencapai IP Tailscale (`100.x.x.x`). Route Tailscale harus ditambahkan ke tabel routing utama (`main`) via `tailscale-routes.service` agar traffic dari container bisa melewati host ke Tailscale.
## Alur Request HTTP (External)
```
Internet
▼ Port 443
Traefik (v3.6)
├─ TLS termination (sertifikat dari volume mount)
├─ Middleware chain: secure-headers → compress → retry → rate-limit → buffer
├─ Plugin: real-ip (Cloudflare), block-sensitive-paths
▼ Router matching
Host(`asepharyana.my.id`) || Host(`www.asepharyana.my.id`) → hub
host(`hub.asepharyana.my.id`) → hub (SPA + dashboard)
Host(`scraper.asepharyana.my.id`) || Host(`api.asepharyana.my.id`) → scraper-api
├─ hub (Next.js, port 3000)
│ ├─ / — Portfolio SPA
│ ├─ /dashboard — Ops dashboard (client-side, auto-refresh 15s)
│ ├─ /api/dashboard — JSON: Docker containers, Jaeger traces, Prometheus metrics
│ └─ Docker socket mounted (:ro) for container discovery
▼ Service load balancer
http://scraper-api:4091
Scraper API (Rust / Axum)
├─ Health check: GET /, respon 200
├─ REST endpoints
├─ Database via `DATABASE_URL` (Tailscale → PostgreSQL di imrnes)
├─ Cache via `REDIS_URL` (Redis lokal di container)
└─ Pub/sub via Dapr sidecar (localhost:3500)
```
## Infrastruktur Internal
### Docker Compose Project
Semua service berjalan dalam satu Docker Compose project bernama `compose` dan bergabung di network `app-shared-net`:
| File | Service | Peran |
|------|---------|-------|
| `traefik.yml` | `traefik` | Reverse proxy + TLS + metrics Prometheus |
| `shared.yml` | `redis` | Cache, session store, backend Dapr pub/sub & state |
| `nats.yml` | `nats` | Message broker + JetStream persistent streaming |
| `dapr.yml` | `dapr-placement` | Koordinasi actor placement untuk sidecar Dapr |
| `scraper.yml` | `scraper-api` + `scraper-api-dapr` | Aplikasi Rust + sidecar Dapr |
| `hub.yml` | `hub` | Next.js SPA portfolio + dashboard + Docker socket |
| `observability.yml` | `otel-collector`, `jaeger`, `prometheus`, `node-exporter` | Tracing, metrics, observability |
### Dapr Sidecar Pattern
Setiap aplikasi yang menggunakan Dapr mendapat sidecar container `daprd`:
```
┌─────────────────────┐
│ scraper-api │
│ (app port 4091) │
└────────┬────────────┘
│ localhost:3500 (HTTP)
│ localhost:50001 (gRPC)
┌────────▼────────────┐
│ scraper-api-dapr │
│ (daprd sidecar) │
│ │
│ Dapr components: │
│ ├─ pubsub.redis │
│ └─ state.redis │
└─────────────────────┘
```
Komponen Dapr:
| Komponen | Tipe | Backend |
|----------|------|---------|
| `pubsub` | `pubsub.redis` | `redis:6379` |
| `statestore` | `state.redis` | `redis:6379` (prefix `dapr`) |
### Monitoring & Auto-Discovery
#### Prometheus Docker Auto-Discovery
Prometheus menggunakan `docker_sd_configs` untuk auto-detect container yang perlu di-scrape. Cukup tambah label pada container:
```yaml
labels:
- 'prometheus.io/scrape=true'
- 'prometheus.io/port=8080' # port metrics endpoint
- 'prometheus.io/path=/metrics' # optional, default /metrics
```
Prometheus akan auto-detect dan mulai scrape container dalam 15 detik.
#### Traefik Metrics
Traefik mengekspos metrics Prometheus di port 8080 (`--metrics.prometheus=true`). Metrics yang tersedia:
| Metric | Query untuk dashboard |
|--------|----------------------|
| Request rate | `sum(rate(traefik_service_requests_total[1m]))` |
| Latency | `avg(traefik_service_request_duration_seconds_sum / traefik_service_request_duration_seconds_count) * 1000` |
| Error rate | `sum(rate(traefik_service_requests_total{code=~"5.."}[1m]))` |
Dashboard di `/api/dashboard` returns node metrics + Traefik range data untuk 4 sparkline charts (RPS, latency, errors, trace volume).
#### Docker Socket Access
Container yang perlu akses Docker socket (`/var/run/docker.sock`) harus punya group docker (GID 988):
```yaml
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
group_add:
- '988'
```di compose atau `--group-add 988` via CLI. Berlaku untuk `hub` (container list) dan `prometheus` (Docker SD).
### NATS + JetStream
NATS berjalan dengan flag `-js` untuk mengaktifkan JetStream. Persistent stream disimpan di volume `nats_data`. Dapr pub/sub routing:
```
Service → Dapr sidecar (pubsub.redis) → Redis streams
```
> **Catatan:** Saat ini Dapr pub/sub menggunakan Redis, bukan NATS. Jika ingin migrasi ke NATS untuk pub/sub, komponen Dapr perlu diganti dengan `pubsub.nats`.
## Arsitektur CI/CD
```
Push ke main (apps/**, infra/**)
docker-build-push.yml
├─ Phase 1: Detect changed services
├─ Phase 2: Build & Push image ke GHCR
└─ Phase 3: Update compose manifest + submodule pointer
▼ (workflow_run trigger)
deploy-docker.yml
├─ SSH ke orangevps
├─ Git sync, pull images
├─ Remove stale containers
└─ Selective restart service
```
Submodule update dari remote repo via `repository_dispatch`:
```
Push ke asepharyana-hub-scraper
▼ (repository_dispatch)
update-submodule.yml
├─ Update submodule pointer
└─ Commit & push ke hub repo
▼ (repository_dispatch trigger)
docker-build-push.yml
└─ Build, push, deploy
```
## Image Tagging Strategy
| Tag | Contoh | Penggunaan |
|-----|--------|------------|
| `sha-<short>` | `sha-a3c5d74` | Immutable, deterministic rollback |
| `latest` | `latest` | Mutable, convenience |
| `buildcache` | `sha-a3c5d74-buildcache` | Registry-based build cache (internal) |
## Networking
### Port Map
| Port | Service | Deskripsi |
|------|---------|-----------|
| 443 | Traefik | HTTPS eksternal |
| 80 | Traefik | Redirect ke HTTPS |
| 4222 | NATS | Client connections |
| 8222 | NATS | HTTP monitor / health |
| 6379 | Redis | Internal container network |
| 3500 | Dapr sidecar | Dapr HTTP API (per service) |
| 50001 | Dapr sidecar | Dapr gRPC API (per service) |
| 50005 | Dapr placement | Actor placement |
| 4091 | Scraper API | Aplikasi HTTP |
## Event Topics Convention
Semua event menggunakan prefix `hub.`:
| Topic | Payload | Deskripsi |
|-------|---------|-----------|
| `hub.image.cached` | `{original_url, cdn_url, source}` | Image selesai di-cache |
| `hub.image.repaired` | `{old_url, new_url}` | CNAME image diperbaiki |
| `hub.scrape.anime.done` | `{source, slug, duration}` | Scrape anime selesai |
| `hub.system.alert` | `{service, level, message}` | Error/alert dari service |
## Service Registry (Traefik)
Domain routing:
| Subdomain | Service | URL Backend |
|-----------|---------|-------------|
| `asepharyana.my.id` (root) | Hub SPA + dashboard | `http://hub:3000` |
| `www.*` | Hub (alias) | `http://hub:3000` |
| `hub.*` | Hub (alias) | `http://hub:3000` |
| `scraper.*` | Scraper API | `http://scraper-api:4091` |
| `api.*` | Scraper API (alias) | `http://scraper-api:4091` |
| `traefik.*` | Traefik Dashboard | `api@internal` |
| `jaeger.*` | Jaeger UI | `http://jaeger:16686` |
Semua domain tersedia di:
- `<service>.asepharyana.my.id`
- `<service>.asepharyana.web.id`
-681
View File
@@ -1,681 +0,0 @@
# Deployment Guide
Panduan deploy aplikasi apapun menggunakan **Docker + Docker Compose + GitHub Actions + VPS**.
## Arsitektur
```
GitHub Repo ──► GitHub Actions ──► Registry (GHCR / Docker Hub / ECR / dll.)
VPS (<VPS_HOST>)
docker compose pull + up
```
## Prerequisites
- Docker Engine >= 24.x
- Docker Compose v2 (plugin)
- Git
- Akun GitHub dengan akses repo
- SSH key di `~/.ssh/<KEY_NAME>` (default: `id_ed25519`)
## Konfigurasi VPS Target
Buat berkas `~/orangevps` (atau sesuaikan dengan env Anda):
```text
ssh <USER>@<VPS_HOST>
```
Contoh isi `~/orangevps`:
```text
ssh root@45.127.35.244
```
| Parameter | Nilai | Contoh |
|-----------|-------|--------|
| User | `<USER>` | `root` |
| Host | `<VPS_HOST>` | `45.127.35.244` |
| SSH Key | `~/.ssh/<KEY_NAME>` | `~/.ssh/id_ed25519` |
| Target Dir di VPS | `<VPS_TARGET_DIR>` | `/opt/app` atau `/root/app` |
> Tip: Jika SSH key menggunakan nama selain default, sesuaikan path dan `ssh -i` sesuai.
## Registry
Pilih registry untuk menyimpan image Docker. Sesuaikan dengan proyek:
| Registry | URL | Auth |
|----------|-----|------|
| GitHub Container Registry | `ghcr.io` | `GITHUB_TOKEN` |
| Docker Hub | `docker.io` | username / PAT |
| AWS ECR | `<account>.dkr.ecr.<region>.amazonaws.com` | `aws ecr get-login-password` |
| Google GCR | `gcr.io` | `gcloud auth print-access-token` |
| Azure ACR | `<registry>.azurecr.io` | `az acr login` |
Contoh namespace untuk GHCR:
```text
Registry : ghcr.io
Namespace: <GITHUB_USERNAME_OR_ORG>
Repo : <REPO_NAME>
```
Pastikan package/visibility di registry mengizinkan akses pull dari VPS.
---
## Deploy Otomatis (Recommended)
Gunakan GitHub Actions untuk otomatisasi build, push, dan deploy.
### Workflow 1: Build dan Push Image
File: `.github/workflows/docker-build-push.yml`
```yaml
name: Build and Push Docker Images
on:
push:
branches: [main]
workflow_dispatch:
permissions:
contents: read
packages: write
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/setup-buildx-action@v4
- uses: docker/build-push-action@v7
with:
context: .
file: Dockerfile
push: true
tags: |
ghcr.io/${{ github.repository }}/<SERVICE_NAME>:latest
ghcr.io/${{ github.repository }}/<SERVICE_NAME>:sha-${{ github.sha }}
cache-from: type=registry,ref=ghcr.io/${{ github.repository }}/<SERVICE_NAME>:buildcache
cache-to: type=registry,ref=ghcr.io/${{ github.repository }}/<SERVICE_NAME>:buildcache,mode=max
```
Ubah `<SERVICE_NAME>` sesuai service (misal: `app`, `web`, `api`). Jika monorepo, gunakan matrix strategy untuk build beberapa service sekaligus.
### Workflow 2: Deploy ke VPS
File: `.github/workflows/deploy-docker.yml`
```yaml
name: Deploy Docker to VPS
on:
workflow_run:
workflows: ['Build and Push Docker Images']
types: [completed]
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Deploy to VPS
env:
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
VPS_HOST: ${{ secrets.VPS_HOST }}
VPS_USER: ${{ secrets.VPS_USER }}
VPS_TARGET_DIR: ${{ secrets.VPS_TARGET_DIR }}
ENV_FILE_PRODUCTION: ${{ secrets.ENV_FILE_PRODUCTION }}
run: |
set -euo pipefail
mkdir -p ~/.ssh
echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -H -t ed25519,rsa "$VPS_HOST" >> ~/.ssh/known_hosts
SSH_OPTS=(-o ControlMaster=auto -o ControlPath=/tmp/ssh-%r@%h:%p -o ControlPersist=600 -o StrictHostKeyChecking=yes)
ssh "${SSH_OPTS[@]}" "$VPS_USER@$VPS_HOST" "mkdir -p $VPS_TARGET_DIR && mkdir -p $VPS_TARGET_DIR/infra/compose"
echo "$ENV_FILE_PRODUCTION" > .env.prod
scp "${SSH_OPTS[@]}" .env.prod "$VPS_USER@$VPS_HOST:$VPS_TARGET_DIR/.env"
ssh "${SSH_OPTS[@]}" "$VPS_USER@$VPS_HOST" bash -s <<'EOF'
set -euo pipefail
cd "$VPS_TARGET_DIR"
docker network inspect app-shared-net >/dev/null 2>&1 || docker network create app-shared-net
if [ ! -d ".git" ]; then
git init
git remote add origin https://github.com/<GITHUB_USER>/<REPO_NAME>.git
fi
git fetch origin main --depth=1 || true
git reset --hard FETCH_HEAD
docker compose --env-file .env pull
docker compose --env-file .env up -d --remove-orphans
EOF
```
### Secrets GitHub yang Diperlukan
Buka **Settings > Secrets and variables > Actions**:
| Secret | Deskripsi |
|--------|-----------|
| `SSH_PRIVATE_KEY` | Isi dengan `cat ~/.ssh/<KEY_NAME>` |
| `VPS_HOST` | IP atau domain VPS |
| `VPS_USER` | User SSH (misal: `root`, `ubuntu`, `deploy`) |
| `VPS_TARGET_DIR` | Direktori aplikasi di VPS |
| `ENV_FILE_PRODUCTION` | Isi dengan environment production |
### Trigger Manual
```bash
gh workflow run deploy-docker.yml
```
---
## Dockerfile Patterns
Pilih pattern sesuai jenis aplikasi.
### Pattern 1: Multi-stage Build (SPA / static assets)
```dockerfile
FROM oven/bun:1 AS builder
WORKDIR /app
COPY package.json bun.lock ./
RUN bun install --frozen-lockfile
COPY . .
RUN bun run build
FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
```
### Pattern 2: Single-stage (runtime image)
```dockerfile
FROM oven/bun:1
WORKDIR /app
COPY package.json bun.lock ./
RUN bun install --frozen-lockfile
COPY . .
EXPOSE 3000
CMD ["bun", "run", "start"]
```
### Pattern 3: Compiled binary (Rust / Go / Zig)
```dockerfile
FROM rust:1 AS builder
WORKDIR /app
COPY . .
RUN cargo build --release
FROM debian:bookworm-slim
COPY --from=builder /app/target/release/app /usr/local/bin/app
EXPOSE 8080
CMD ["app"]
```
---
## Docker Compose Patterns
### Single service
```yaml
services:
app:
container_name: app
image: registry.example.com/org/app:latest
restart: always
ports:
- "3000:3000"
environment:
- NODE_ENV=production
```
### Multi-service dengan shared network
```yaml
services:
app:
container_name: app
image: registry.example.com/org/app:latest
restart: always
networks: [app-shared-net]
redis:
container_name: redis
image: redis:7-alpine
restart: always
networks: [app-shared-net]
networks:
app-shared-net:
name: app-shared-net
external: true
```
### Dengan reverse proxy (Traefik / Caddy / Nginx)
```yaml
services:
app:
container_name: app
image: registry.example.com/org/app:latest
restart: always
networks: [app-shared-net]
labels:
- 'traefik.enable=true'
- 'traefik.http.routers.app.rule=Host(`app.example.com`)'
- 'traefik.http.routers.app.entrypoints=websecure'
- 'traefik.http.routers.app.tls=true'
- 'traefik.http.services.app.loadbalancer.server.port=3000'
networks:
app-shared-net:
name: app-shared-net
external: true
```
---
## Deploy Manual (Lokal)
### 1. Build dan Push ke Registry
Login ke registry:
```bash
echo $GITHUB_TOKEN | docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
```
Build dan push:
```bash
docker build -t ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:latest -f Dockerfile .
docker push ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:latest
```
Tag tambahan dengan SHA commit:
```bash
SHORT_SHA=$(git rev-parse --short HEAD)
docker tag ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:latest \
ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:sha-${SHORT_SHA}
docker push ghcr.io/<GITHUB_USERNAME>/<REPO_NAME>/<SERVICE_NAME>:sha-${SHORT_SHA}
```
### 2. Pull dan Deploy di VPS
SSH ke VPS:
```bash
ssh -i ~/.ssh/<KEY_NAME> <USER>@<VPS_HOST>
```
Clone repo (jika belum):
```bash
git clone https://github.com/<GITHUB_USER>/<REPO_NAME>.git <VPS_TARGET_DIR>
cd <VPS_TARGET_DIR>
```
Buat shared network (hanya sekali):
```bash
docker network create app-shared-net
```
Siapkan environment:
```bash
cp .env.example .env
# Edit .env sesuai nilai production
nano .env
```
Login ke registry di VPS:
```bash
echo $GITHUB_TOKEN | docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
```
Pull gambar terbaru:
```bash
cd <VPS_TARGET_DIR>
docker compose -f docker-compose.yml --env-file .env pull
```
Deploy (up):
```bash
docker compose -f docker-compose.yml --env-file .env up -d --remove-orphans
```
Verifikasi:
```bash
docker compose -f docker-compose.yml ps
docker compose -f docker-compose.yml logs -f <SERVICE_NAME>
```
---
## Deployment Order (Manual)
Jika deploy bertahap, gunakan urutan ini:
```bash
# 1. Shared services (Redis, database, dll.)
docker compose -f infra/compose/shared.yml up -d
# 2. Reverse proxy
docker compose -f infra/compose/traefik.yml up -d
# 3. Aplikasi
docker compose \
-f infra/compose/app1.yml \
-f infra/compose/app2.yml \
up -d
```
---
## Perintah Berguna di VPS
```bash
# Lihat semua container
docker ps -a
# Log service
docker logs -f <container_name>
# Restart satu service
docker compose -f <compose_file> up -d --force-recreate
# Hapus network lama (hati-hati)
docker network rm app-shared-net
docker network create app-shared-net
# Bersihkan image unused
docker image prune -a -f
docker system prune -a -f
```
---
## Troubleshooting
### Image tidak bisa di-pull
Pastikan sudah login ke registry di VPS:
```bash
docker logout ghcr.io
echo $GITHUB_TOKEN | docker login ghcr.io -u <GITHUB_USERNAME> --password-stdin
```
Periksa visibility package di registry (harus `Public` atau akses diberikan).
### Port sudah dipakai
```bash
docker ps | grep :80
docker ps | grep :443
```
### Reverse proxy tidak routing
Periksa label di compose file dan pastikan shared network ada:
```bash
docker network inspect app-shared-net
docker logs traefik
```
---
## Environment Variable Management
### Pola 1: `.env` di VPS (recommended untuk production)
```bash
# Di VPS
cd <VPS_TARGET_DIR>
cp .env.example .env
# Edit sesuai production
nano .env
```
CI/CD upload `.env` via secret, tidak simpan di repo.
### Pola 2: Docker secrets (Swarm mode)
```yaml
services:
app:
image: app:latest
secrets:
- db_password
secrets:
db_password:
file: ./secrets/db_password.txt
```
### Pola 3: External secret manager
- **HashiCorp Vault**: inject via env atau file
- **AWS Secrets Manager**: `aws secretsmanager get-secret-value`
- **Doppler / Infisical**: unified secret management
---
## Tagging dan Versioning
### Strategy yang umum
| Strategy | Contoh tag | Kegunaan |
|----------|-----------|----------|
| Latest + SHA | `latest`, `sha-abc1234` | CI/CD cepat, traceable |
| SemVer | `1.2.3`, `1.2`, `1` | Release publik |
| Git tag mirror | `v1.2.3` | Sync dengan git tag |
| Branch mirror | `main`, `develop` | Preview / staging |
### Contoh git tag driven deploy
```bash
git tag v1.2.3
git push origin v1.2.3
```
CI/CD membaca tag, build image dengan tag yang sama, dan deploy.
---
## Rollback
### Rollback via registry
```bash
# Lihat tag yang tersedia
docker manifest inspect ghcr.io/org/app:latest
# atau lihat UI registry
# Di VPS, edit compose file ke tag sebelumnya
# lalu:
docker compose --env-file .env pull
docker compose --env-file .env up -d --remove-orphans
```
### Rollback via git
```bash
git revert HEAD
git push origin main
# CI/CD otomatis build dan deploy versi sebelumnya
```
---
## Health Checks
### Di Dockerfile
```dockerfile
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD curl -f http://localhost:3000/health || exit 1
```
### Di Docker Compose
```yaml
services:
app:
image: app:latest
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
interval: 30s
timeout: 3s
retries: 3
start_period: 10s
```
---
## Monitoring & Observability
```bash
# Log aggregated
docker compose logs -f --tail=100
# Resource usage
docker stats
# Disk usage
docker system df
# Cleanup
docker system prune -a -f
```
---
## Catatan Keamanan
- Jangan commit `.env` atau SSH private key ke repo.
- Gunakan GitHub Secrets (atau secret manager) untuk credential di CI/CD.
- Rotate token dan key secara berkala.
- Batasi akses SSH ke VPS (ubah port default, gunakan fail2ban).
- Set `StrictHostKeyChecking=yes` pada SSH opsional deployment.
---
---
## Proyek Ini: asepharyana-hub
> Dokumentasi spesifik untuk repo ini. Lihat juga [ADR-0002](adr/0002-env-file-via-github-secret.md).
### Topologi
| Host | IP | Peran |
|------|----|-------|
| `orangevps` (VPS) | `45.127.35.244` | Docker host: Traefik, scraper-api, Redis, NATS, Dapr |
| `imrnes` (bare-metal) | `100.121.180.82` (Tailscale) | PostgreSQL (port 6432), Redis (port 6379) |
### Environment Variables
**Production `.env` tidak pernah di-commit.** File ini disimpan sebagai GitHub secret `ENV_FILE_PRODUCTION` dan di-SCP ke VPS saat deploy via `deploy-docker.yml`.
Cara update:
```bash
# Baca current .env dari VPS
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env"
# Update GitHub secret (dari output di atas)
cat > /tmp/env-updated << 'EOF'
<paste content, edit, lalu>
EOF
cat /tmp/env-updated | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub
```
**Jangan manual edit `.env` di VPS tanpa update GitHub secret juga** — nanti ke- overwrite pas deploy berikutnya.
### Database
| Variable | Value |
|----------|-------|
| `DATABASE_URL` | `postgres://asephs:hunterz@100.121.180.82:6432/hub` |
| `REDIS_URL` | `redis://redis:6379` (Docker network) |
### Kompose
Proyek compose bernama `compose`, terdiri dari 5 file yang selalu di-include bersamaan:
```bash
/root/asepharyana-hub/infra/compose/
├── traefik.yml # Reverse proxy
├── shared.yml # Redis
├── nats.yml # NATS
├── dapr.yml # Dapr placement
├── scraper.yml # Scraper API
└── observability.yml # OTel Collector, Jaeger, Dashboard
```bash
cd /root/asepharyana-hub
docker compose \
-p compose \
--env-file .env \
-f infra/compose/traefik.yml \
-f infra/compose/shared.yml \
-f infra/compose/scraper.yml \
-f infra/compose/nats.yml \
-f infra/compose/dapr.yml \
-f infra/compose/observability.yml \
up -d --remove-orphans
```
---
## Checklist Deploy Proyek Baru
1. [ ] Dockerfile ditest lokal (`docker build`, `docker run`)
2. [ ] Docker Compose file valid (`docker compose config`)
3. [ ] `.dockerignore` sesuai (node_modules, .git, .env)
4. [ ] Registry dibuat (GHCR package / Docker Hub repo / ECR / dll.)
5. [ ] GitHub Actions workflow dibuat dengan permission `packages: write`
6. [ ] VPS siap: Docker, Docker Compose, SSH key
7. [ ] Shared network dibuat (`docker network create`)
8. [ ] `.env` production di-VPS atau via secret manager
9. [ ] Reverse proxy (Traefik / Caddy / Nginx) routing ke container
10. [ ] Health check endpoint aktif
-206
View File
@@ -1,206 +0,0 @@
# Development Guide
Panduan setup lingkungan development lokal untuk kontributor `asepharyana-hub`.
## Prasyarat
| Tool | Versi Minimal | Catatan |
|------|---------------|---------|
| Git | 2.40+ | Submodule support |
| Docker | 24+ | Dengan Docker Compose v2 plugin |
| Rust | 1.85+ | Hanya untuk `apps/scraper` |
| Bun | 1.x | Root tooling (Biome) |
| Dapr CLI | 1.14+ | Opsional, untuk development dengan Dapr |
## Setup Awal
```bash
# 1. Clone repo
git clone https://github.com/asepharyana/asepharyana-hub.git
cd asepharyana-hub
# 2. Init submodules
make init-submodules
# 3. Setup environment
cp .env.example .env
# Edit .env sesuai kebutuhan lokal
# 4. Install root dependencies
bun install
```
## Menjalankan Infrastruktur Lokal
Beberapa service membutuhkan Redis. Jalankan dengan:
```bash
make dev
# atau equivalen:
docker compose -f infra/compose/shared.yml up -d
```
Ini akan menjalankan Redis Alpine di `localhost:6379`.
### (Opsional) NATS Lokal
Jika service membutuhkan pub/sub:
```bash
docker compose -f infra/compose/nats.yml up -d
# NATS client: localhost:4222
# NATS monitor: localhost:8222
```
### (Opsional) Dapr Placement Lokal
Jika service membutuhkan sidecar Dapr:
```bash
docker compose -f infra/compose/dapr.yml up -d
# Dapr placement: localhost:50005
```
## Menjalankan Service Lokal
### Scraper API (Rust)
```bash
# Pastikan Redis sudah running (make dev)
cd apps/scraper
# Cargo run
cargo run
# Dengan Dapr sidecar (jika placement running)
dapr run \
--app-id scraper-api \
--app-port 4091 \
--dapr-http-port 3500 \
--resources-path ../../infra/dapr/components \
-- cargo run
```
### Dengan Docker Compose (Full Stack)
Untuk menjalankan semua service sekaligus:
```bash
docker compose \
-f infra/compose/shared.yml \
-f infra/compose/nats.yml \
-f infra/compose/dapr.yml \
-f infra/compose/scraper.yml \
--env-file .env \
up -d
```
Untuk service baru, tambahkan compose file-nya ke daftar.
## Update Submodules
### Pull latest dari semua submodule
```bash
make update-submodules
# atau:
git submodule update --remote --merge --recursive
```
### Check status submodule
```bash
make status
# atau:
git submodule status
```
### Sync .env ke submodule
```bash
bash scripts/2updateenv.sh
# Copy .env root ke apps/*/
```
## Linting & Formatting
Root repo menggunakan **Biome** untuk linting dan formatting:
```bash
bun run check # Lint + format + write
bun run ci # CI mode (no write, exit code on issues)
bun run lint # Lint only
bun run format # Format only
```
## Build Docker Image Lokal
```bash
# Scraper API
docker build -f infra/docker/scraper.Dockerfile -t scraper-api:local .
# Service baru: tambahkan Dockerfile di infra/docker/
```
## Testing
Saat ini belum ada test runner di root level. Masing-masing submodule mengelola testing sendiri:
```bash
# Scraper API (Rust)
cd apps/scraper && cargo test
```
## Validasi YAML
Sebelum commit perubahan infra, validasi semua file YAML:
```bash
python -c "
import pathlib, yaml
for p in pathlib.Path('infra').rglob('*.yml'):
with open(p) as f: yaml.safe_load(f)
print(f'OK {p}')
for p in pathlib.Path('infra').rglob('*.yaml'):
with open(p) as f: yaml.safe_load(f)
print(f'OK {p}')
"
for f in infra/compose/*.yml; do
docker compose -f "$f" config >/dev/null && echo "OK $f"
done
```
## Git Workflow
### Commit Convention
```
<type>(<scope>): <description>
```
Type: `feat`, `fix`, `chore`, `docs`, `refactor`, `test`, `ci`, `perf`, `style`
Scope: `scraper`, `infra`, `ci`, `dapr`, `nats`, `docs`, `deps`, `scripts`, `root`
Contoh:
```
feat(scraper): add image cache endpoint
fix(infra): correct Traefik rate-limit config
chore(deps): bump biome to 2.5.0
```
### Branch Strategy
- `main` — production branch, push triggers CI/CD
- Fitur baru: branch dari `main`, PR ke `main`
- Submodule development: dilakukan di repo masing-masing, hub hanya update pointer
## Deployment ke VPS
Push ke `main` otomatis trigger CI/CD. Untuk trigger manual:
```bash
gh workflow run deploy-docker.yml
```
Lihat `docs/DEPLOYMENT.md` untuk detail.
-162
View File
@@ -1,162 +0,0 @@
# Menambahkan Dapr ke Service Baru
Panduan integrasi Dapr runtime sidecar untuk service di `asepharyana-hub`.
## Prasyarat
- NATS server berjalan (`infra/compose/nats.yml`)
- Dapr placement service berjalan (`infra/compose/dapr.yml`)
## 1. Compose File
Setiap service butuh sidecar container Dapr. Contoh:
```yaml
services:
app:
container_name: app
image: ghcr.io/asepharyana/asepharyana-hub/app:latest
restart: always
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
networks:
app-shared-net:
aliases:
- app
env_file:
- ../../.env
app-dapr:
container_name: app-dapr
image: daprio/daprd:latest
restart: always
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
networks:
- app-shared-net
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
otel-collector:
condition: service_started
networks:
- app-shared-net
command:
- './daprd'
- '--app-id=app'
- '--app-port=3000'
- '--dapr-http-port=3500'
- '--dapr-grpc-port=50001'
- '--placement-host-address=dapr-placement:50005'
- '--config=/dapr/config.yaml'
- '--resources-path=/dapr/components'
volumes:
- ../../infra/dapr:/dapr:ro
networks:
app-shared-net:
name: app-shared-net
external: true
```
## 2. Mengakses Dapr dari Service
### Via HTTP API (semua bahasa)
Sidecar listen di `localhost:3500`:
```bash
# Publish event
curl -X POST http://localhost:3500/v1.0/publish/pubsub/hub.event.type \
-H "Content-Type: application/json" \
-d '{"key": "value"}'
# Service invocation
curl http://localhost:3500/v1.0/invoke/app/method/endpoint
# State store
curl -X POST http://localhost:3500/v1.0/state/statestore \
-H "Content-Type: application/json" \
-d '[{"key": "mykey", "value": "myvalue"}]'
```
### Via Dapr SDK (Rust)
Tambah ke `Cargo.toml`:
```toml
dapr-sdk = { version = "0.15", features = ["pubsub", "http"] }
tokio-stream = "0.1"
```
Contoh publish event:
```rust
use dapr_sdk::client::{Client, Event};
use dapr_sdk::DaprClient;
let client = DaprClient::new("127.0.0.1", 3500).await?;
client.publish_event("pubsub", "hub.image.cached", serde_json::json!({
"original_url": url,
"cdn_url": cdn_url,
})).await?;
```
Contoh subscribe event:
```rust
let mut stream = client.subscribe_events("pubsub", "hub.image.cached").await?;
while let Some(event) = stream.next().await {
let data: MyEvent = serde_json::from_slice(&event.data)?;
// handle event
}
```
## 3. Event Topics Convention
Gunakan prefix `hub.` untuk semua event:
| Topic | Payload | Description |
|-------|---------|-------------|
| `hub.image.cached` | `{original_url, cdn_url, source}` | Image selesai di-cache |
| `hub.image.repaired` | `{old_url, new_url}` | CNAME image diperbaiki |
| `hub.scrape.anime.done` | `{source, slug, duration}` | Scrape anime selesai |
| `hub.system.alert` | `{service, level, message}` | Error/alert dari service |
## 4. Local Development
Untuk development tanpa Docker:
```bash
# 1. Install Dapr CLI
# 2. Init Dapr local
dapr init
# 3. Run service dengan sidecar
dapr run --app-id app --app-port 3000 --dapr-http-port 3500 \
--resources-path ./infra/dapr/components \
-- cargo run
```
## 5. Verifikasi
```bash
# Sidecar health
curl http://localhost:3500/v1.0/healthz
# Publish test event
curl -X POST http://localhost:3500/v1.0/publish/pubsub/hub.test \
-H "Content-Type: application/json" \
-d '{"test": true}'
# NATS stream stats
curl http://localhost:8222/jszetstream
```
-43
View File
@@ -52,49 +52,6 @@ networks:
Gunakan `app-shared-net` agar service dapat diakses oleh Traefik dan service lain. Gunakan `app-shared-net` agar service dapat diakses oleh Traefik dan service lain.
## 3.5. Tambahkan Dapr sidecar (wajib untuk pub/sub)
Setiap service yang ingin menggunakan Dapr pub/sub atau service invocation harus punya sidecar.
Tambah di `infra/compose/<nama-app>.yml`:
```yaml
<nama-app>-dapr:
container_name: <nama-app>-dapr
image: daprio/daprd:latest
restart: always
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
otel-collector:
condition: service_started
networks:
- app-shared-net
command:
- './daprd'
- '--app-id=<nama-app>'
- '--app-port=<port>'
- '--dapr-http-port=3500'
- '--dapr-grpc-port=50001'
- '--placement-host-address=dapr-placement:50005'
- '--config=/dapr/config.yaml'
- '--resources-path=/dapr/components'
volumes:
- ../../infra/dapr:/dapr:ro
```
Pastikan juga app container punya `depends_on` ke dapr-placement, nats, dan otel-collector:
```yaml
depends_on:
dapr-placement:
condition: service_healthy
nats:
condition: service_healthy
otel-collector:
condition: service_started
```
## 4. Tambahkan route Traefik ## 4. Tambahkan route Traefik
Update `infra/traefik/dynamic/apps.yaml`: Update `infra/traefik/dynamic/apps.yaml`:
@@ -22,9 +22,12 @@ Use `asepharyana-hub` as the root hub repository.
Current app submodules: Current app submodules:
| Service | Path | Remote | | Service | Path | Remote |
| ----------- | -------------- | --------------------------------------- | | -------------- | ---------------- | --------------------------------------- |
| Scraper API | `apps/scraper` | `asepharyana/asepharyana-hub-scraper` | | Elysia API | `apps/elysia` | `asepharyana/asepharyana-hub-elysia` |
| React frontend | `apps/react` | `asepharyana/asepharyana-hub-react` |
| Rust auth API | `apps/rust-auth` | `asepharyana/asepharyana-hub-rust-auth` |
| Scraper API | `apps/scraper` | `asepharyana/asepharyana-hub-scraper` |
## Consequences ## Consequences
-124
View File
@@ -1,124 +0,0 @@
# ADR 0002: Production `.env` via GitHub Encrypted Secret
## Status
Accepted
## Context
The project runs on a remote VPS (`orangevps`, IP `45.127.35.244`) that hosts multiple services via Docker Compose. These services require environment variables (database credentials, API keys, tokens) that must not be committed to the repository.
The production `.env` file on the VPS is **not** a copy of the committed `.env` in the repo root — it contains additional secrets (Portainer tokens, Discord bot tokens, etc.) that only exist in production.
Previously, the `.env` file on the VPS was edited manually via SSH, which led to drift between the local `.env` and the production `.env`. When the database server IP or port changed in the local `.env`, the production `.env` was not updated, causing service outages.
## Decision
The production `.env` file is stored as a **GitHub Actions encrypted secret** named `ENV_FILE_PRODUCTION`. During deployment, the `.github/workflows/deploy-docker.yml` workflow writes this secret to a file and SCPs it to the VPS.
### Flow
```
GitHub Secret (ENV_FILE_PRODUCTION)
▼ (deploy-docker.yml)
echo "$ENV_FILE_PRODUCTION" > .env.prod
scp .env.prod → VPS:$VPS_TARGET_DIR/.env
▼ (docker compose --env-file .env up)
Container reads $DATABASE_URL, $JWT_SECRET, etc.
```
### How to update
```bash
# 1. Read current content from the VPS
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env"
# 2. Pipe updated content to the GitHub secret
# (requires gh CLI with repo access)
cat /path/to/updated-env | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub
# 3. Trigger a redeploy to push it to the VPS
gh workflow run deploy-docker.yml
# OR apply immediately on the VPS (for hotfix):
ssh root@45.127.35.244 "sed -i 's|OLD_VALUE|NEW_VALUE|' /root/asepharyana-hub/.env"
# Then restart affected containers
```
## Server Topology
| Host | IP | Role |
|------|----|------|
| `orangevps` (VPS) | `45.127.35.244` | Docker host: Traefik, scraper-api, Redis, NATS, Dapr |
| `imrnes` (bare-metal) | `100.121.180.82` (Tailscale) | PostgreSQL (port 6432), Redis (port 6379), Browserless |
## Database
| Variable | Value |
|----------|-------|
| `DATABASE_URL` | `postgres://asephs:hunterz@100.121.180.82:6432/hub` |
| `REDIS_URL` | `redis://redis:6379` (Docker network, overridden per-service) |
| `EXTERNAL_BROWSERLESS_WS` | `ws://43.134.105.109:3001/?token=...` (external proxy) |
> **Important:** The Docker Compose `environment:` section uses variable interpolation (`${DATABASE_URL}`), which is resolved from the `--env-file .env` at compose time — NOT from the service's `env_file`. Both must be kept in sync.
## Docker Compose Project Structure
The VPS runs a single Docker Compose project named `compose` composed of multiple files:
```bash
/root/asepharyana-hub/infra/compose/
├── traefik.yml # Reverse proxy (TLS termination, routing)
├── shared.yml # Redis
├── nats.yml # NATS message broker + JetStream
├── dapr.yml # Dapr placement service
├── scraper.yml # Scraper API + Dapr sidecar
└── observability.yml # OTel Collector + Jaeger + Dashboard
```
All files are always included together for dependency resolution:
```bash
docker compose \
--env-file .env \
-f infra/compose/traefik.yml \
-f infra/compose/shared.yml \
-f infra/compose/scraper.yml \
-f infra/compose/nats.yml \
-f infra/compose/dapr.yml \
-f infra/compose/observability.yml \
up -d
```
## GitHub Secrets Required
| Secret | Description |
|--------|-------------|
| `SSH_PRIVATE_KEY` | SSH key for VPS access |
| `VPS_HOST` | `45.127.35.244` |
| `VPS_USER` | `root` |
| `VPS_TARGET_DIR` | `/root/asepharyana-hub` |
| `ENV_FILE_PRODUCTION` | Full `.env` content for production |
## Consequences
### Positive
- Environment is version-controlled via GitHub Secrets audit log.
- No risk of committing secrets to the repo.
- Deployment is fully automated — `.env` is pushed on every deploy.
- Easy to rotate secrets: update `ENV_FILE_PRODUCTION` and redeploy.
### Negative
- The secret is opaque — you cannot diff it or review changes via PR.
- If the secret falls out of sync with the local `.env`, services silently break on next deploy.
- Requires `gh` CLI or GitHub UI to update — not a simple file edit.
### Mitigations
- Keep the **committed `.env`** in the repo root as the source of truth for non-secret values (database URL, ports, API endpoints).
- Document any manual SSH hotfix at the same time as updating the GitHub secret.
- Run `gh secret set ENV_FILE_PRODUCTION` with the latest server `.env` content after any hotfix.
-234
View File
@@ -1,234 +0,0 @@
# Backup & Disaster Recovery
## Aset yang Perlu di-Backup
| Aset | Lokasi | Frekuensi | Metode |
|------|--------|-----------|--------|
| Database PostgreSQL | `imrnes` (100.121.180.82:6432) | Harian | `pg_dump` |
| Volume Redis | `orangevps` (Docker volume) | Opsional | Redis RDB / AOF |
| Volume NATS JetStream | `orangevps` (Docker volume) | Opsional | File copy |
| Docker Compose manifests | GitHub (hub repo) | Real-time | Git |
| Environment variables | GitHub secret `ENV_FILE_PRODUCTION` | Manual | `gh secret set` |
| TLS certificates | `orangevps` (`/root/*.pem`, `*.key`) | Saat renew | SCP |
| Tailscale auth | Tailscale admin console | - | Cloud-managed |
| GitHub Actions secrets | GitHub UI | Manual | Backup list |
## Database PostgreSQL (Prioritas Tertinggi)
### Backup Manual
```bash
# Dari orangevps (via Tailscale)
pg_dump -h 100.121.180.82 -p 6432 -U asephs -d hub \
--no-owner --no-acl \
-F c -f /root/db-backups/hub-$(date +%Y%m%d-%H%M%S).dump
# Atau dari imrnes langsung
pg_dump -U asephs -d hub \
-F c -f /backup/hub/hub-$(date +%Y%m%d-%H%M%S).dump
```
### Restore
```bash
# Drop dan recreate database
dropdb -h 100.121.180.82 -p 6432 -U asephs hub
createdb -h 100.121.180.82 -p 6432 -U asephs hub
# Restore dari dump
pg_restore -h 100.121.180.82 -p 6432 -U asephs -d hub \
--no-owner --no-acl \
/path/to/backup/hub-20260101-120000.dump
```
### Backup Otomatis (via Cron di imrnes)
```bash
# /etc/cron.d/hub-db-backup
0 2 * * * root pg_dump -U asephs -d hub -F c -f /backup/hub/hub-$(date +\%Y\%m\%d).dump && find /backup/hub -name "hub-*.dump" -mtime +30 -delete
```
## Volume Docker
### Redis
Redis data bisa di-recover dari NATS events (event sourcing). Jika tidak ada persistence requirement, cukup restart:
```bash
docker volume rm redis_data
docker compose -f infra/compose/shared.yml up -d
```
Jika perlu backup:
```bash
# Save RDB snapshot
docker exec redis redis-cli SAVE
# Copy dari volume
docker run --rm -v redis_data:/data -v /backup:/backup alpine cp /data/dump.rdb /backup/redis-$(date +%Y%m%d).rdb
```
### NATS JetStream
```bash
# Backup volume
docker run --rm -v nats_data:/data -v /backup:/backup alpine \
tar czf /backup/nats-$(date +%Y%m%d).tar.gz -C /data .
```
## Environment Variables
### Backup `.env` dari VPS
```bash
# Simpan current .env dari VPS
ssh root@45.127.35.244 "cat /root/asepharyana-hub/.env" > .env.backup.$(date +%Y%m%d)
# Update GitHub secret
cat .env.backup.$(date +%Y%m%d) | gh secret set ENV_FILE_PRODUCTION --repo asepharyana/asepharyana-hub
```
### Restore `.env` jika hilang
```bash
# Buat .env baru dari template
cp .env.example .env
# Edit secrets (manual dari password manager atau GitHub secret)
# Atau download dari GitHub secret
gh secret list --repo asepharyana/asepharyana-hub
```
## TLS Certificates
### Backup
```bash
# Di orangevps
tar czf /root/cert-backup-$(date +%Y%m%d).tar.gz \
/root/asepharyana.my.id.pem \
/root/asepharyana.my.id.key \
/root/asepharyana.web.id.pem \
/root/asepharyana.web.id.key \
/root/asepharyana-hub/infra/traefik/dynamic/ssl.yaml
# SCP ke local
scp root@45.127.35.244:/root/cert-backup-*.tar.gz .
```
### Restore
```bash
# SCP ke VPS
scp cert-backup-20260101.tar.gz root@45.127.35.244:/root/
# Extract
ssh root@45.127.35.244 "tar xzf /root/cert-backup-20260101.tar.gz -C / && docker restart traefik"
```
## Disaster Recovery Scenarios
### Skenario 1: VPS (orangevps) mati total
**Dampak:** Semua service down.
**Recovery:**
```bash
# 1. Provision VPS baru (atau restore dari snapshot)
# 2. Install Docker + Tailscale
# 3. Clone repo
git clone https://github.com/asepharyana/asepharyana-hub.git /root/asepharyana-hub
# 4. Setup Tailscale, route service
# 5. Restore .env
echo "<ENV_FILE_PRODUCTION>" > /root/asepharyana-hub/.env
# 6. Restore TLS certs
# 7. Create network
docker network create app-shared-net
# 8. Start services sesuai urutan
cd /root/asepharyana-hub
for f in shared.yml nats.yml dapr.yml traefik.yml scraper.yml; do
docker compose -f infra/compose/$f --env-file .env up -d
done
# 9. Update DNS jika IP baru
```
### Skenario 2: Database (imrnes) mati total
**Dampak:** Semua service yang butuh database error.
**Recovery:**
```bash
# 1. Fix imrnes atau provision server baru
# 2. Setup PostgreSQL
# 3. Restore dari backup terakhir
# 4. Update Tailscale IP jika perlu
# 5. Update .env dan GitHub secret
# 6. Redeploy
```
### Skenario 3: GitHub repository hilang
**Dampak:** Kehilangan CI/CD, tapi Docker images masih ada di GHCR.
**Recovery:**
```bash
# 1. Create repo baru di GitHub
# 2. Push dari local clone
git remote add origin-new https://github.com/asepharyana/asepharyana-hub-new.git
git push origin-new main
# 3. Re-create GitHub secrets
# 4. Re-create workflows
# 5. Update VPS remote
ssh root@45.127.35.244 "cd /root/asepharyana-hub && git remote set-url origin https://github.com/asepharyana/asepharyana-hub-new.git"
```
### Skenario 4: GHCR registry tidak bisa diakses
**Dampak:** Tidak bisa pull image.
**Recovery:**
```bash
# 1. Build image langsung di VPS
docker build -f infra/docker/scraper.Dockerfile -t ghcr.io/asepharyana/asepharyana-hub/scraper-api:local .
# 2. Update compose file untuk sementara
sed -i 's|image: ghcr.io/.*|image: ghcr.io/asepharyana/asepharyana-hub/scraper-api:local|' infra/compose/scraper.yml
# 3. Start
docker compose -f infra/compose/scraper.yml up -d
```
### Skenario 5: Semua server mati (total loss)
**Recovery:**
```bash
# 1. Provision VPS baru
# 2. Provision server database baru
# 3. Setup Tailscale
# 4. Clone repo, restore .env, certs
# 5. Restore database dari backup (jika ada)
# 6. Jika tidak ada backup database:
# - Build image dari GHCR
# - Start service dengan database kosong
# - Data akan terisi ulang dari scraping
```
## Checklist Pencegahan
- [ ] Cron job backup database berjalan
- [ ] Backup `.env` disimpan di luar VPS (password manager)
- [ ] TLS certificates backup disimpan di luar VPS
- [ ] GitHub secrets terdaftar (tidak hanya diingat)
- [ ] Docker images bisa di-rebuild dari CI (GHCR sebagai source of truth)
- [ ] Tailscale admin access via multiple accounts
-248
View File
@@ -1,248 +0,0 @@
# CI/CD Pipeline
Dokumentasi pipeline CI/CD untuk `asepharyana-hub`. Terdiri dari 5 GitHub Actions workflow yang saling terhubung.
## Workflow Overview
```
┌─────────────┐
│ Lint │ (PR/push → Biome)
└──────┬──────┘
Push ke main ─────┼────── repository_dispatch
┌──────▼──────────────────┐
│ docker-build-push.yml │
│ │
│ Phase 1: Detect │
│ Phase 2: Build & Push │
│ Phase 3: Update │
│ manifests │
└──────┬──────────────────┘
│ workflow_run
┌──────▼──────────────┐
│ deploy-docker.yml │
│ SSH → VPS │
│ Pull → Restart │
└─────────────────────┘
repository_dispatch ──► update-submodule.yml
(dari submodule) (update pointer → commit)
docker-build-push.yml
(triggered by push)
```
## Workflow Detail
### 1. Lint (`lint.yml`)
**Trigger:** PR/push ke `main` yang mengubah `*.json`, `*.js`, `biome.json`
**Aksi:**
- Checkout repo dengan submodules
- Setup Bun
- `bun install --frozen-lockfile`
- `bun run ci` (Biome CI mode)
**Permissions:** read-only
### 2. Build and Push Docker Images (`docker-build-push.yml`)
**Trigger:**
- Push ke `main` yang mengubah `apps/**`, `infra/**`, atau file workflow
- `repository_dispatch` tipe `submodule-updated`
- `workflow_dispatch` (manual)
**Concurrency:** Satu workflow per branch (cancel-in-progress=false)
#### Phase 1: Detect Changes
Job `changes` mendeteksi service mana yang perlu di-build:
- **Push event:** `git diff --name-only` antara `before` dan `after` SHA
- **repository_dispatch:** Parse payload `{service, sha}` dan validasi
- **workflow_dispatch:** Build semua service
Output format matrix:
```json
[{"id":"scraper-api","target":"docker-scraper","path":"apps/scraper"}]
```
#### Phase 2: Build & Push (Matrix)
Job `build` berjalan paralel per service (matrix strategy):
1. Checkout repo + sync submodule
2. Jika `repository_dispatch`, checkout submodule ke SHA tertentu
3. Login ke GHCR
4. Setup Docker Buildx
5. Build & push dengan tag:
- `ghcr.io/asepharyana/asepharyana-hub/<service>:latest`
- `ghcr.io/asepharyana/asepharyana-hub/<service>:sha-<shortsha>`
6. Build cache: registry-based (`:<service>:buildcache`)
#### Phase 3: Update Manifests
Job `update-manifest`:
1. Update image tag di compose file (`infra/compose/<service>.yml`)
2. Jika `repository_dispatch`, update submodule pointer
3. Commit dengan message `chore: update manifests and submodules [skip ci]`
4. Push dengan retry (3 attempts, rebase jika conflict)
### 3. Deploy Docker to VPS (`deploy-docker.yml`)
**Trigger:**
- `workflow_run` setelah `docker-build-push.yml` selesai
- Push ke `main` yang mengubah `infra/**`
- `workflow_dispatch` (manual)
**Concurrency:** Satu deployment dalam satu waktu (`group: deploy-vps`)
**Aksi di VPS (via SSH):**
```
1. Setup SSH multiplexing
2. SCP .env dari GitHub secret ke VPS
3. Docker login ke GHCR
4. Git sync (fetch + reset --hard)
5. Detect changed files:
├─ Compose stack changes → selective container update
├─ Traefik dynamic config → SIGHUP
└─ Other infra → full deploy
6. Pull images (retry 3x)
7. Remove stale containers
8. Up services
9. SIGHUP Traefik jika perlu
```
### 4. Security Scan (`security.yml`)
**Trigger:**
- PR ke `main`
- Jadwal: Setiap Senin (`0 6 * * 1`)
**Aksi:**
- Checkout dengan fetch-depth 2
- CodeQL init untuk Rust
- `cargo build` di `apps/scraper`
- CodeQL analyze
### 5. Update Submodule Pointer (`update-submodule.yml`)
**Trigger:** `repository_dispatch` tipe `submodule-updated`
**Aksi:**
1. Validasi payload (`service`, `sha`)
2. Map service ke submodule path (e.g., `scraper-api``apps/scraper`)
3. Update submodule ke SHA yang diberikan
4. Commit sebagai `monrepo-bot` dengan message:
`chore: update <service> to <shortsha>`
5. Push dengan retry (3 attempts)
## Flow Submodule Update
Flow lengkap ketika code berubah di submodule repo:
```
1. Developer push ke asepharyana-hub-scraper
2. GitHub Action di scraper repo kirim repository_dispatch
ke asepharyana-hub
3. update-submodule.yml terima dispatch, update pointer
4. Commit masuk ke hub repo main
5. Commit ini trigger docker-build-push.yml
(push ke main dengan path apps/scraper/**)
6. Build image baru, update compose file
7. Deploy ke VPS
```
## Secrets yang Diperlukan
| Secret | Workflow | Deskripsi |
|--------|----------|-----------|
| `SSH_PRIVATE_KEY` | deploy-docker | SSH key untuk akses VPS |
| `VPS_HOST` | deploy-docker | IP VPS (`45.127.35.244`) |
| `VPS_USER` | deploy-docker | User SSH (`root`) |
| `VPS_TARGET_DIR` | deploy-docker | Dir di VPS (`/root/asepharyana-hub`) |
| `ENV_FILE_PRODUCTION` | deploy-docker | Full `.env` production |
## Menambahkan Service Baru ke Pipeline
Untuk menambahkan service baru, update:
### `docker-build-push.yml`
1. **Phase 1 — `changes` job:** Tambah detection logic untuk service baru:
```yaml
echo "new-service=$(changed '^(apps/new-service(/|$)|\.github/workflows/docker-build-push\.yml$|infra/docker/new-service\.Dockerfile$)')" >> "$GITHUB_OUTPUT"
```
2. **Phase 1 — `repository_dispatch`:** Tambah case:
```yaml
case "$SERVICE" in
scraper-api|new-service) ;;
```
3. **Phase 1 — `set-matrix`:** Tambah service:
```bash
if [ "${{ ...['new-service'] == 'true' ... }}" == "true" ]; then add_service "new-service" "docker-new-service" "apps/new-service"; fi
```
4. **Phase 2 — `meta` step:** Tambah mapping Dockerfile:
```bash
"new-service") echo "dockerfile=infra/docker/new-service.Dockerfile" >> $GITHUB_OUTPUT ;;
```
5. **Phase 3 — `update-manifest`:** Tambah mapping:
```bash
SERVICES["new-service"]="new-service.yml"
PATHS["new-service"]="apps/new-service"
```
### `deploy-docker.yml`
Tambah compose file ke `ALL_COMPOSE_FILES`:
```bash
ALL_COMPOSE_FILES="infra/compose/traefik.yml infra/compose/shared.yml infra/compose/scraper.yml infra/compose/nats.yml infra/compose/dapr.yml infra/compose/new-service.yml"
```
## Rollback
### Rollback Image
```bash
# Cari SHA tag sebelumnya di GHCR packages
# Update compose file ke tag tersebut
sed -i 's|sha-badcommit|sha-goodcommit|g' infra/compose/scraper.yml
git commit -am "fix: rollback scraper-api to sha-goodcommit"
git push
```
### Rollback via Git Revert
```bash
git revert HEAD
git push origin main
# Pipeline otomatis build dan deploy
```
## Monitoring Pipeline
```bash
# Cek status workflow terbaru
gh run list --limit 5
# Lihat log workflow tertentu
gh run view <run-id> --log
# Trigger workflow manual
gh workflow run deploy-docker.yml
```
+39
View File
@@ -0,0 +1,39 @@
# Squid Proxy Configuration (Archived)
**Status**: Archived; not referenced by active infrastructure.
**Date archived**: 2026-06-04
**Original location**: `infra/squid.conf`
**Reason**: No active Docker Compose service mounts or starts Squid. The config is preserved here for reference if Squid proxying is restored later.
## Original configuration
```squid
# Allow localhost access only
acl localhost src 127.0.0.1/32 ::1/128
http_access allow localhost
# Allow Docker bridge subnets (Adjusted for our expected subnets)
acl docker_bridge src 172.16.0.0/12
http_access allow docker_bridge
# Deny all other access
http_access deny all
# Standard port - Explicitly bind to all interfaces
http_port 0.0.0.0:3128
# Hide source info
forwarded_for off
via off
```
## Restore checklist
1. Copy this config back to `infra/squid.conf`.
2. Add a Squid service to an active compose file.
3. Mount the config into the Squid container.
4. Document which services should use the proxy.
5. Validate network exposure and authentication before production use.
-286
View File
@@ -1,286 +0,0 @@
# NATS + JetStream Guide
Dokumentasi konfigurasi, penggunaan, dan troubleshooting NATS di infrastruktur `asepharyana-hub`.
## Arsitektur
NATS berjalan di container `nats` dengan JetStream diaktifkan (`-js`). Data persistent disimpan di volume Docker `nats_data`.
```
Service ──► NATS (port 4222) ──► JetStream (disk)
├─ Monitoring HTTP: port 8222
└─ Client connections: port 4222
```
### Hubungan dengan Dapr
Saat ini Dapr pub/sub menggunakan **Redis** (`pubsub.redis`), bukan NATS. NATS berfungsi sebagai message broker independen untuk:
- Event streaming antar service
- Persistent job queues
- Pub/sub untuk service yang tidak menggunakan Dapr
Jika ingin Dapr menggunakan NATS sebagai backend pub/sub, ganti komponen `pubsub.yaml`:
```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.nats
version: v1
metadata:
- name: natsURL
value: nats://nats:4222
```
## Konfigurasi Compose
File: `infra/compose/nats.yml`
```yaml
services:
nats:
container_name: nats
image: nats:latest
restart: always
networks:
app-shared-net:
aliases:
- nats
ports:
- '4222:4222' # client connections
- '8222:8222' # HTTP monitor
command:
- '-js' # enable JetStream
- '-sd'
- '/data' # storage directory
volumes:
- nats_data:/data
```
## CLI Tools
### Install NATS CLI
```bash
# Linux
curl -sf https://bin.nats.dev/nats | sh
sudo mv nats /usr/local/bin/
# Atau via package manager
# brew install nats-io/nats-tools/nats (macOS)
```
### Koneksi ke NATS
```bash
# Dari host (port 4222 ter-expose)
nats context save hub --server nats://localhost:4222 --description "Hub Production"
nats context select hub
# Test koneksi
nats server check
nats server info
```
### Manage Streams (JetStream)
```bash
# List semua stream
nats stream list
# Lihat detail stream
nats stream info <stream-name>
# Buat stream
nats stream add <stream-name> \
--subjects "hub.>" \
--storage file \
--max-msgs 1000000 \
--max-bytes 1G \
--retention limits
# Hapus stream
nats stream rm <stream-name>
# Purge (hapus semua message, retain stream)
nats stream purge <stream-name>
```
### Pub/Sub
```bash
# Subscribe ke subject
nats sub "hub.>"
nats sub "hub.image.cached"
# Publish message
nats pub "hub.test" '{"message": "hello"}'
nats pub "hub.image.cached" '{"original_url": "https://example.com/img.jpg", "cdn_url": "https://cdn.example.com/img.jpg"}'
# Request-reply
nats request "hub.service.do" '{"task": "process"}'
```
### Monitoring via HTTP API
```bash
# Server info
curl http://localhost:8222/
# JetStream info
curl http://localhost:8222/jszetstream
# Stream detail
curl http://localhost:8222/jszetstream?stream=<stream-name>
# Consumer info
curl http://localhost:8222/jszetstream?stream=<stream-name>&consumer=<consumer-name>
# Server stats
curl http://localhost:8222/varz
# Connections
curl http://localhost:8222/connz
```
## Event Topics Convention
Semua topik menggunakan prefix `hub.`:
| Subject | Payload | Deskripsi |
|---------|---------|-----------|
| `hub.image.cached` | `{original_url, cdn_url, source}` | Image selesai di-cache |
| `hub.image.repaired` | `{old_url, new_url}` | CNAME image diperbaiki |
| `hub.scrape.anime.done` | `{source, slug, duration}` | Scrape anime selesai |
| `hub.system.alert` | `{service, level, message}` | Error/alert dari service |
| `hub.test` | Any | Testing |
### Wildcard Subjects
NATS mendukung wildcard:
- `hub.>` — semua event hub (multi-level)
- `hub.image.*` — semua event image (single-level)
- `hub.*.done` — semua event yang selesai (single-level)
## JetStream Configuration
### Storage
Data JetStream disimpan di volume Docker `nats_data`.
Lokasi di VPS:
```bash
docker volume inspect nats_data
# atau
ls -la /var/lib/docker/volumes/nats_data/_data/
```
### Memory & Limits
NATS tidak memiliki konfigurasi limit memori default. Untuk production, pertimbangkan:
```yaml
command:
- '-js'
- '-sd'
- '/data'
- '--max_pending_size=64MB'
- '--max_payload=1MB'
```
Atau gunakan NATS configuration file:
```yaml
# nats-server.conf
jetstream:
max_memory_store: 256MB
max_file_store: 10GB
```
## Troubleshooting
### Stream data tidak muncul
```bash
# 1. Cek koneksi NATS
nats server check
# 2. Cek apakah JetStream aktif
curl http://localhost:8222/jszetstream
# 3. Cek stream dan message count
nats stream list
# 4. Subscribe langsung untuk test
nats sub ">"
```
### NATS tidak bisa start
```bash
# Cek log
docker logs nats
# Cek apakah port 4222 sudah dipakai
ss -tlnp | grep 4222
# Cek volume data korup
docker run --rm -v nats_data:/data alpine ls -la /data
# Restart
docker compose -f infra/compose/nats.yml up -d --force-recreate
```
### Disk JetStream penuh
```bash
# Cek ukuran volume
docker system df -v | grep nats_data
# Purge stream jika perlu
nats stream purge <stream-name>
# Atau hapus volume (data hilang!)
docker compose -f infra/compose/nats.yml down
docker volume rm nats_data
docker compose -f infra/compose/nats.yml up -d
```
### Slow consumer
```bash
# Cek consumer lag
nats stream info <stream-name>
# Lihat fields: "Pending" dan "Acknowledgment"
# Lihat stats server
curl http://localhost:8222/varz | jq '.slow_consumers'
```
## Migration: Redis Pub/Sub ke NATS
Jika ingin migrasi dari Dapr pub/sub Redis ke NATS:
1. Buat stream NATS untuk topik `hub.>`
2. Update `infra/dapr/components/pubsub.yaml` dari `pubsub.redis` ke `pubsub.nats`
3. Deploy ulang semua service (Dapr sidecar akan reconnect)
4. Verifikasi event flow
```yaml
# infra/dapr/components/pubsub.yaml (setelah migrasi)
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.nats
version: v1
metadata:
- name: natsURL
value: nats://nats:4222
```
-60
View File
@@ -1,60 +0,0 @@
# Tools — Document Scanner & Media Processing Hub
Self-hosted, no-install document scanner dan media processing tools yang jalan di browser. Alternatif dari CamScanner, ilovepdf, compressjpeg — tanpa upload ke pihak ketiga.
## Visi
Satu platform dengan tools manipulasi file yang **beneran dipake orang setiap hari**. Semua proses di backend Rust — cepat, hemat memory, ga perlu install software.
## Fitur Utama
### Phase 1 — Document Scanner (Prioritas)
- Foto dokumen pake HP → auto-detect tepi → lurusin (perspective correction)
- Enhance: iluminasi merata, contrast, sharpen, B&W
- OCR → searchable PDF (teks bisa di-copy, dicari)
- Batch: multi-page → satu PDF
- Fallback crop manual (kalau auto-detect gagal)
### Phase 2 — Image Tools
- Compress JPEG/PNG/WebP (lossy + lossless, atur kualitas %)
- Resize batch (atur dimensi, semua foto disamain)
- Convert format (HEIC→JPEG, PNG→WebP, SVG→PNG)
- Remove background (ONNX model, Rust runtime)
### Phase 3 — PDF Tools
- Merge PDF (gabung file)
- Split PDF (ekstrak halaman tertentu)
- Images→PDF (kumpulan foto jadi 1 file)
- PDF→Images (tiap halaman jadi gambar)
- PDF compress (turunkin kualitas embedded images)
### Phase 4 — Video/Audio Tools
- Compress video (bitrate + resolusi)
- Extract audio (MP4→MP3)
- Trim/crop
- GIF maker
- Audio convert + trim
## Target User
Orang yang:
- Punya HP/PC, paham teknologi dasar (buka browser, upload file)
- Butuh scan dokumen tanpa install aplikasi
- Butuh kompres file buat kirim WA/email
- Butuh manipulasi PDF sesekali
- Peduli privasi — ga mau upload file ke server pihak ketiga
## Prinsip Desain
1. **Satu task selesai dalam <5 detik** — ga ada loading lama
2. **Drag & drop + preview** — liat hasil sebelum download
3. **Progress realtime** via WebSocket — tau lagi di tahap mana
4. **Batch processing** — banyak file, satu klik
5. **Privasi first** — file otomatis dihapus setelah 1 jam
6. **WASM fallback** — tools ringan jalan di client (tanpa upload)
## Domain & Branding
- **Domain**: `tools.asepharyana.my.id` | `tools.asepharyana.web.id`
- **Design**: Twilight Terminal theme (sama kaya portfolio), konsisten visual
- **Dashboard**: Link dari hub dashboard → tools stats (total files processed, storage used)
-451
View File
@@ -1,451 +0,0 @@
# Architecture
## System Overview
```
┌────────────────────────────────────────────────────────────────┐
│ BROWSER │
│ ┌────────────┐ ┌────────────┐ ┌────────────────────────┐ │
│ │ Upload │ │ Camera │ │ Preview + Download │ │
│ │ (drag/drop)│ │ (PWA) │ │ (streaming) │ │
│ └─────┬──────┘ └─────┬──────┘ └───────────┬────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ WebSocket (progress: processing/step/percentage) │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────┬───────────────────────────────────┘
│ HTTPS / WSS
┌────────────────────────────────────────────────────────────────┐
│ TRAEFIK (tools.asepharyana.my.id) │
│ Middleware chain: secure-headers → compress → rate-limit │
└──────────────────────────┬────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────┐
│ tools-app (Next.js 16 / TypeScript) │
│ │
│ ┌──────────────────┐ ┌─────────────────┐ │
│ │ Pages/Routes │ │ API Routes │ │
│ │ / → home │ │ POST /api/upload ──▶ file │
│ │ /scan → scanner │ │ GET /api/job/:id ─▶ status │
│ │ /image → image │ │ WS /api/job/:id/ws ─▶ progress │
│ │ /pdf → pdf tools │ │ GET /api/download/:id ─▶ file │
│ └──────────────────┘ └─────────────────┘ │
│ │
│ Upload validation: MIME type, size limit (50MB), virus scan │
│ Temp storage bridge ke worker via HTTP/NATS │
└──────────────────┬────────────────────────────────────────────┘
│ HTTP (internal)
┌────────────────────────────────────────────────────────────────┐
│ API GATEWAY (Rust / Axum) │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────┐ │
│ │ Upload │ │ Job Manager │ │ Download │ │
│ │ (streaming │ │ (CRUD job │ │ (stream file, │ │
│ │ chunked) │ │ status) │ │ auto-delete) │ │
│ └──────┬───────┘ └──────┬───────┘ └────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ NATS JetStream │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
│ │ │ jobs │ │ jobs │ │ jobs │ │ │
│ │ └────┬─────┘ └────┬─────┘ └──────┬───────┘ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ scan. │ │ image. │ │ pdf. │ │ │
│ │ │ progress │ │ progress │ │ progress │ │ │
│ │ └──────────┘ └──────────┘ └──────────────┘ │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Cache (Redis) │ │
│ │ - Job metadata (status, progress, timestamps) │ │
│ │ - Rate limiting (sliding window per IP/tool) │ │
│ │ - Result metadata (file path, size, type) │ │
│ └────────────────────────────────────────────────────┘ │
└──────────────────┬────────────────────────────────────────────┘
│ consume NATS queue
┌────────────────────────────────────────────────────────────────┐
│ WORKER POOL (Rust / Tokio + Rayon) │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
│ │ Scan Worker │ │ Image Worker │ │ PDF Worker │ │
│ │ ×4 instances │ │ ×2 instances │ │ ×2 instances│ │
│ │ │ │ │ │ │ │
│ │ 1. Load image │ │ 1. Load image │ │ 1. Load PDF │ │
│ │ 2. Edge detect │ │ 2. Compress │ │ 2. Merge/ │ │
│ │ 3. Warp │ │ /resize/ │ │ split │ │
│ │ 4. Enhance │ │ convert │ │ 3. Save │ │
│ │ 5. OCR │ │ 3. Save │ │ 4. Update │ │
│ │ 6. Gen PDF │ │ 4. Update │ │ job │ │
│ │ 7. Update job │ │ job status │ │ status │ │
│ │ └───────────────┘ └─────────────────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Temp Storage (filesystem volume / S3-compatible) │ │
│ │ Auto-cleanup: job TTL 1 jam, NATS cron tiap 10m │ │
│ └────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────┘
```
## Component Diagram
```
┌────────────────────────────────────────────┐
│ apps/tools │
│ │
│ ├── frontend/ │
│ │ ├── pages/ ← Next.js pages │
│ │ ├── components/ ← React components │
│ │ ├── lib/ ← utilities │
│ │ └── public/ ← static assets │
│ │ │
│ ├── backend/ ← Rust workspace │
│ │ ├── gateway/ ← Axum API server │
│ │ ├── workers/ ← Processing workers │
│ │ │ ├── scanner/ ← Document scanner │
│ │ │ ├── image/ ← Image tools │
│ │ │ └── pdf/ ← PDF tools │
│ │ └── common/ ← Shared libs │
│ │ │
│ └── Dockerfile │
└────────────────────────────────────────────┘
```
## Data Flow (Document Scanner — Flow Lengkap)
```
1. User buka tools.asepharyana.my.id/scan
2. Upload foto via drag-drop atau kamera HP (PWA)
3. Next.js route handler menerima file
├─ Validasi: MIME type (image/*), max 50MB, virus header scan
└─ Upload chunked ke Gateway internal (HTTP POST)
4. Gateway menerima stream:
├─ Simpan ke temp storage
├─ Buat job record di Redis: {id, tool: "scan", status: "queued", progress: 0}
└─ Publish ke NATS: tools.scan.jobs {job_id, file_path, options}
5. Scan Worker consume dari NATS:
├─ Update Redis: status = "processing", progress = 10
├─ Load image (image-rs)
├─ Pipeline (detail di pipeline.md):
│ 1. Edge detection ──▶ progress 25
│ 2. Perspective warp ──▶ progress 40
│ 3. Shadow removal ──▶ progress 55
│ 4. Binarization ──▶ progress 70
│ 5. Contrast/sharpen ──▶ progress 80
│ 6. OCR ──▶ progress 90
│ 7. Generate PDF ──▶ progress 95
├─ Simpan file hasil ke temp storage
├─ Update Redis: status = "completed", progress = 100, result_path, ocr_text
└─ Publish ke NATS: tools.scan.progress {job_id, status, progress}
6. WebSocket handler di Gateway:
├─ Subscribe NATS topics tools.scan.progress
├─ Forward ke browser user (per-job-id filter)
└─ Browser update progress bar + preview
7. User download PDF:
├─ GET /api/download/:job_id
├─ Gateway stream file dari temp storage
└─ Browser save file
```
## Tech Stack
### Frontend (Next.js + TypeScript)
| Library | Fungsi |
|---------|--------|
| Next.js 16 | App router, API routes |
| shadcn/ui + Tailwind v4 | UI components |
| Framer Motion | Animasi progress, transisi |
| Canvas API | Preview crop manual, image manipulation client-side |
| WebSocket API | Real-time progress |
### Backend (Rust)
| Crate | Fungsi |
|-------|--------|
| `axum` | HTTP server (Gateway) |
| `tokio` | Async runtime |
| `image` | Image I/O, resize, convert, compress |
| `imageproc` | Edge detection, contour, thresholding |
| `lopdf` | PDF generation, merge, split, compress |
| `leptess` | Tesseract OCR binding |
| `ort` | ONNX Runtime (background removal) |
| `async-nats` | NATS JetStream client |
| `deadpool-redis` | Redis connection pool |
| `redis` | Redis async client |
| `rayon` | Parallel processing (batch, pixel ops) |
| `serde` | Serialization |
| `tracing` + `opentelemetry` | Observability |
| `uuid` | Job ID generation |
### Infrastructure
| Komponen | Fungsi |
|----------|--------|
| NATS JetStream | Job queue, progress pub/sub, scheduler |
| Redis | Job metadata, rate limiting, cache |
| PostgreSQL | Opsional — audit log, usage statistics |
| Tesseract | OCR engine (data files di Docker image) |
| Prometheus | Metrics (jobs/min, queue depth, latency per stage) |
## Job Queue (NATS Streams & Consumers)
### Streams
```
tools-scan-jobs → 1 stream, mirror to all scan workers
tools-image-jobs → 1 stream, mirror to all image workers
tools-pdf-jobs → 1 stream, mirror to all pdf workers
tools-progress → 1 stream, all progress events (key-value by job_id)
tools-scheduler → 1 stream, cron events
```
### Subjects
```
tools.scan.jobs.{job_id} → job submission
tools.scan.progress.{job_id} → progress update (fan-out ke Gateway)
tools.image.jobs.{job_id} → job submission
tools.image.progress.{job_id} → progress update
tools.pdf.jobs.{job_id} → job submission
tools.pdf.progress.{job_id} → progress update
tools.scheduler.cleanup → cleanup expired files (every 10 min)
```
## Redis Schema
```
job:{id} → Hash {status, tool, progress, file_path, result_path, ocr_text, created_at, ttl}
rate_limit:{ip}:{tool} → Sorted Set (sliding window)
file_meta:{hash} → String {original_name, size, mime}
```
## Metrics (Prometheus)
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `tools_jobs_total` | Counter | `tool`, `status` | Total jobs processed |
| `tools_jobs_in_flight` | Gauge | `tool` | Currently processing jobs |
| `tools_queue_depth` | Gauge | `tool` | NATS queue depth |
| `tools_processing_duration` | Histogram | `tool`, `stage` | Duration per stage |
| `tools_file_size_bytes` | Histogram | `tool` | Upload file size distribution |
| `tools_rate_limit_hits` | Counter | `tool` | Rate limit violations |
## Directory Structure
```
apps/tools/
├── frontend/
│ ├── src/
│ │ ├── app/
│ │ │ ├── page.tsx # Landing page
│ │ │ ├── scan/
│ │ │ │ ├── page.tsx # Scanner page
│ │ │ │ └── result/[id]/
│ │ │ │ └── page.tsx # Result page
│ │ │ ├── image/
│ │ │ │ ├── compress/page.tsx
│ │ │ │ ├── resize/page.tsx
│ │ │ │ ├── convert/page.tsx
│ │ │ │ └── remove-bg/page.tsx
│ │ │ ├── pdf/
│ │ │ │ ├── merge/page.tsx
│ │ │ │ ├── split/page.tsx
│ │ │ │ ├── images-to-pdf/page.tsx
│ │ │ │ └── compress/page.tsx
│ │ │ ├── api/
│ │ │ │ ├── upload/route.ts
│ │ │ │ ├── job/[id]/route.ts
│ │ │ │ │ └── ws/route.ts
│ │ │ │ └── download/[id]/route.ts
│ │ │ ├── layout.tsx
│ │ │ └── globals.css
│ │ ├── components/
│ │ │ ├── upload-zone.tsx # Drag & drop area
│ │ │ ├── progress-bar.tsx # WebSocket-connected progress
│ │ │ ├── preview.tsx # Before/after preview
│ │ │ ├── crop-editor.tsx # Manual corner adjustment
│ │ │ ├── tool-layout.tsx # Consistent tool page layout
│ │ │ └── camera-capture.tsx # PWA camera interface
│ │ ├── hooks/
│ │ │ ├── use-job-status.ts # WebSocket connection
│ │ │ ├── use-upload.ts # Upload with progress
│ │ │ └── use-camera.ts # Camera access
│ │ └── lib/
│ │ ├── utils.ts
│ │ └── types.ts
│ ├── next.config.ts
│ ├── package.json
│ └── tsconfig.json
├── backend/
│ ├── Cargo.toml
│ ├── gateway/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── main.rs
│ │ ├── routes/
│ │ │ ├── mod.rs
│ │ │ ├── upload.rs
│ │ │ ├── job.rs
│ │ │ ├── download.rs
│ │ │ └── ws.rs
│ │ ├── nats/
│ │ │ ├── mod.rs
│ │ │ └── publisher.rs
│ │ ├── redis/
│ │ │ ├── mod.rs
│ │ │ ├── job.rs
│ │ │ └── ratelimit.rs
│ │ ├── metrics.rs
│ │ └── config.rs
│ │
│ ├── workers/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── main.rs
│ │ ├── scanner/
│ │ │ ├── mod.rs
│ │ │ ├── pipeline.rs
│ │ │ ├── edge.rs # Edge detection
│ │ │ ├── warp.rs # Perspective correction
│ │ │ ├── enhance.rs # Shadow removal, B&W, contrast
│ │ │ ├── ocr.rs # Tesseract wrapper
│ │ │ └── pdf.rs # Generate searchable PDF
│ │ ├── image/
│ │ │ ├── mod.rs
│ │ │ ├── compress.rs
│ │ │ ├── resize.rs
│ │ │ ├── convert.rs
│ │ │ └── remove_bg.rs
│ │ ├── pdf/
│ │ │ ├── mod.rs
│ │ │ ├── merge.rs
│ │ │ ├── split.rs
│ │ │ ├── extract.rs
│ │ │ └── compress.rs
│ │ ├── nats/
│ │ │ ├── mod.rs
│ │ │ └── consumer.rs
│ │ └── config.rs
│ │
│ └── common/
│ ├── Cargo.toml
│ └── src/
│ ├── lib.rs
│ ├── types.rs # Shared types (JobStatus, Job, etc.)
│ ├── error.rs # Error types
│ └── nats.rs # NATS subject constants
├── Dockerfile
├── compose.yml # Local dev compose
└── README.md
```
## API Design
### Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/upload` | Upload file, create job |
| `GET` | `/api/job/:id` | Get job status + result metadata |
| `WS` | `/api/job/:id/ws` | WebSocket — realtime progress |
| `GET` | `/api/download/:id` | Download result file |
| `DELETE` | `/api/job/:id` | Cancel job, delete files |
| `GET` | `/health` | Health check |
### Upload Request
```
POST /api/upload
Content-Type: multipart/form-data
{
file: <binary>,
tool: "scan" | "image-compress" | "image-resize" | "image-convert" | "remove-bg" |
"pdf-merge" | "pdf-split" | "images-to-pdf" | "pdf-compress",
options?: { // tool-specific options
quality?: 80, // compress quality
width?: 1920, // resize width
format?: "webp", // convert format
pages?: "1,3-5", // PDF split pages
dpi?: 300, // scan DPI
enhance?: true, // scan auto-enhance
ocr?: true // scan OCR
}
}
```
### Response (202 Accepted)
```json
{
"job_id": "uuid",
"status": "queued",
"tool": "scan",
"ws_url": "/api/job/uuid/ws",
"created_at": "2026-07-24T10:00:00Z",
"estimated_seconds": 5
}
```
### WebSocket Messages
```json
// Server → Client
{
"type": "progress",
"job_id": "uuid",
"status": "processing",
"progress": 45,
"stage": "warp",
"message": "Meluruskan perspektif dokumen..."
}
{
"type": "complete",
"job_id": "uuid",
"status": "completed",
"progress": 100,
"result": {
"download_url": "/api/download/uuid",
"file_name": "scan_20260724.pdf",
"file_size": 1245678,
"pages": 1,
"ocr_text": "Nama: Asep...",
"preview_url": "/api/job/uuid/preview"
}
}
{
"type": "error",
"job_id": "uuid",
"status": "failed",
"error": "Edge detection failed: cannot find document boundary"
}
```
## Integration with Existing Portfolio
| Area | Detail |
|------|--------|
| **Domain** | `tools.asepharyana.my.id` — tambah entry di `infra/traefik/dynamic/apps.yaml` |
| **Dashboard** | Link ke tools stats di dashboard hub yang sudah ada |
| **Docker Compose** | `infra/compose/tools.yml` — pola sama kaya `hub.yml` |
| **CI/CD** | Tambah service `tools` di `docker-build-push.yml` |
| **Style** | Ulang Twilight Terminal theme dari hub, konsisten visual branding |
| **Monitoring** | Reuse existing Prometheus + Grafana, tambah metrics tools |

Some files were not shown because too many files have changed in this diff Show More