fix(sla): 100% fleet SLA (64/64) - per-challenge SSH login, phew checker, sidecar detection

Three independent root causes, all found by measuring instead of assuming:

1. SSH failed on 10/16 challenges while state.json looked perfect.
   Only the 6 native GEMASTIK XVIII images provision 'ctfuser'; every imported
   XVI/XVII image does 'echo root:${PASSWORD} | chpasswd' and logs in as root.
   set_ssh_passwords() hardcoded ctfuser, so chpasswd set a password nobody
   used -> 'Permission denied' on every team. Registry gains a per-challenge
   'ssh_user'; chpasswd targets the real login and reports failures loudly.

2. phew SLA timed out on a healthy service, four bugs stacked:
   - chall.py block-buffers stdout through the exec pipe (PYTHONUNBUFFERED now
     set) and does a fresh Pailier keygen (~12 s) before printing its menu;
   - _read_until read a TEXT pipe, so read(1) pulled 8 KB into Python's
     TextIOWrapper buffer and select() then blocked on data already in memory;
   - its buffer was per-call, so the read satisfying 'pt (hex)' also swallowed
     the '> ' the next call waited for -> a race that failed intermittently;
   - reaping killed chall.py it did not own: a blanket pkill -f, a
     snapshot-diff (concurrent sessions diff against the same pre-spawn set),
     and a class-level _children shared across uvicorn's thread pool. The child
     now prints its own pid so exactly one session is reaped.
   Also: ONE interactive session per check instead of five spawns (Paillier is
   randomized per ciphertext, not per process) - 5 keygens were the CPU load
   that starved the checks. And the 6 orphan single-node containers from the
   original deploy were removed; one held 58 leaked chall.py and drove load
   average 76 on 2 CPUs.

3. missing_sidecars() matched compose-generated names (teamN-<svc>-1) while
   every service sets an explicit container_name, so it reported all 16 running
   challenges as missing and hid the one real gap (anti-alchemy-db, which has
   no container_name). Now reads container_name when present and falls back to
   the compose default otherwise.

Verified: 64/64 SLA across 4 teams; 64/64 real SSH logins succeed with
correct <chall>_teamN hostnames; phew 3/3 sequential with no process leak.

Adds panel/verify_ssh_creds.py, audit_ssh_users.sh, reset_runtime.sh,
sla_sweep.sh, fix_sidecars.sh, phew_concurrency_test.sh, exec_probe_i.py.

panel/exec_probe.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+# How many concurrent `docker exec` sessions does this host tolerate?
+# The Phew SLA checker's 5-way concurrency test failed with
+# "No such exec instance" + "failed to open stdin fifo", which points at an
+# exec-session ceiling rather than at the checker.
+set -uo pipefail
+C=${1:-phew_container_team1}
+N=${2:-8}
+echo "container: $C"
+echo "--- $N concurrent trivial execs ---"
+fail=0
+for i in $(seq 1 "$N"); do
+  ( out=$(docker exec "$C" true 2>&1); rc=$?
+    if [ $rc -ne 0 ]; then echo "  exec$i FAILED: $out"; fi ) &
+done
+wait
+echo "--- done ---"
+echo "dockerd max concurrent execs:"
+docker info 2>/dev/null | grep -iE 'exec|containerd' | head -3

panel/exec_probe_i.py

@@ -0,0 +1,54 @@
+#!/usr/bin/env python3
+"""How many concurrent INTERACTIVE (`docker exec -i`) sessions does this host take?
+
+Trivial non-interactive execs succeed 8-way, but the Phew SLA checker's 5-way
+interactive test failed with
+
+    failed to open stdin fifo: error creating fifo ... -stdin: no such file
+    No such exec instance: <id>
+
+which is a containerd exec-session limit, not a checker bug. Interactive execs
+allocate a fifo + a tracked exec instance, so they hit a ceiling that plain
+`docker exec <c> true` never approaches.
+
+    python3 panel/exec_probe_i.py [container] [N]
+"""
+import subprocess
+import sys
+import time
+from concurrent.futures import ThreadPoolExecutor
+
+CONT = sys.argv[1] if len(sys.argv) > 1 else "phew_container_team1"
+N = int(sys.argv[2]) if len(sys.argv) > 2 else 8
+
+
+def interactive(i: int) -> tuple[int, str]:
+    """Mimic the checker: `exec -i`, write a line, read the reply, exit."""
+    p = subprocess.Popen(
+        ["docker", "exec", "-i", CONT, "sh", "-c",
+         "echo $$; read line; echo \"got:$line\""],
+        stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT, text=True, bufsize=0)
+    try:
+        out, _ = p.communicate("hello\n", timeout=45)
+    except subprocess.TimeoutExpired:
+        p.kill()
+        out = "TIMEOUT"
+    return p.returncode, (out or "").strip().replace("\n", " | ")[:150]
+
+
+def main() -> None:
+    print(f"container={CONT}  N={N} interactive execs")
+    ok = 0
+    with ThreadPoolExecutor(max_workers=N) as ex:
+        for rc, out in ex.map(interactive, range(N)):
+            if rc == 0 and "got:hello" in out:
+                ok += 1
+            else:
+                print(f"  FAIL rc={rc}: {out}")
+    print(f"  {ok}/{N} interactive execs OK")
+    print("RESULT:", "PASS" if ok == N else f"limit reached at {ok}/{N}")
+
+
+if __name__ == "__main__":
+    main()

panel/fix_sidecars.sh

@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+# Bring up only the MISSING sidecars of each team (compose default `up -d`
+# would also rebuild the 16 main services, which is slow and unnecessary).
+set -uo pipefail
+BASE=/opt/gemastik18-final
+cd "$BASE/panel"
+for t in 1 2 3 4; do
+  missing=$(python3 -c "
+import sys; sys.path.insert(0,'.')
+import teams
+print(' '.join(teams.missing_sidecars($t)))
+" 2>/dev/null)
+  if [ -z "$missing" ]; then
+    echo "team$t: nothing missing"
+    continue
+  fi
+  echo "team$t missing: $missing"
+  # map container name -> compose service name
+  for cont in $missing; do
+    svc=${cont#team${t}-}
+    svc=${svc%-1}
+    echo "  starting $svc"
+    (cd "$BASE/teams/team$t/services" && \
+     docker compose -p "team$t" -f docker-compose.yml up -d --no-deps "$svc" 2>&1 | tail -2)
+  done
+done
+echo "=== verify ==="
+python3 -c "
+import sys; sys.path.insert(0,'.')
+import teams
+for i in (1,2,3,4):
+    print('team%d still missing:' % i, teams.missing_sidecars(i))
+"

panel/phew_chall_ref.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+
+import os
+from Pailier import *
+from Crypto.Util.number import *
+
+with open("/flag.txt", "rb") as f:
+    flag_bytes = f.read()
+
+key = os.urandom(66)              
+key_int = bytes_to_long(key)
+
+cipher = pailier()
+
+while True:
+    print("1. encrypt")
+    print("2. bingo")
+    print("3. decrypt")
+    print("4. key?")
+    try:
+        inp = int(input("> "))
+    except (ValueError, EOFError):
+        print("Invalid input")
+        continue
+
+    if inp == 1:
+        print("pt (hex)")
+        try:
+            inp = input("> ")
+            ct = cipher.encrypt(int(inp, 16))
+            print('ct : ', '{0:x}'.format(ct))
+        except (ValueError, EOFError):
+            print("Invalid hex input")
+
+    elif inp == 2:
+        print("key (hex)")
+        try:
+            user_hex = input("> ").strip()
+            user_key = bytes.fromhex(user_hex)
+            
+            if len(user_key) == 66 and user_key == key:
+                try:
+                    print(flag_bytes.decode())
+                except Exception:
+                    print(flag_bytes.hex())
+            else:
+                print("nope")
+        except (ValueError, EOFError):
+            print("nope")
+
+    elif inp == 3:
+        print("ct (hex)")
+        try:
+            inp = input("> ")
+            pt = cipher.decrypt(int(inp, 16))
+            print('pt : ', '{0:x}'.format(pt))
+        except (ValueError, EOFError):
+            print("Invalid hex input")
+
+    elif inp == 4:
+        try:
+            ct = cipher.encrypt(key_int)
+            print('ct : ', '{0:x}'.format(ct))
+        except Exception:
+            print("Encryption error")
+
+    else:
+        exit()

panel/phew_concurrency_test.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Concurrency regression test for the Phew checker.
+#
+# The bug this guards against: the checker reaped chall.py processes it did not
+# own, so two overlapping checks killed each other's session and the victim
+# reported "Process ended while waiting for '> '" on a healthy service. Firing
+# several checks at once is the only way to reproduce it — sequential runs pass
+# even with the bug present.
+set -uo pipefail
+BASE=/opt/gemastik18-final
+TEAM=${1:-1}
+N=${2:-5}
+case "$TEAM" in 1) PORT=31080;; 2) PORT=32080;; 3) PORT=33080;; 4) PORT=34080;; esac
+U=$(grep -oP '^ADMIN_USERNAME=\K.*' "$BASE/teams/team$TEAM/receiver/.env")
+P=$(grep -oP '^ADMIN_PASSWORD=\K.*' "$BASE/teams/team$TEAM/receiver/.env")
+
+count() { docker exec "phew_container_team$TEAM" sh -c 'ps ax | grep -c "[c]hall.py"'; }
+echo "before: $(count) chall.py (1 = socat service only)"
+
+pids=()
+for i in $(seq 1 "$N"); do
+  ( out=$(timeout 300 curl -sS -u "$U:$P" "http://127.0.0.1:$PORT/check/phew")
+    echo "  req$i: $out" ) &
+  pids+=($!)
+done
+for p in "${pids[@]}"; do wait "$p"; done
+
+sleep 5
+after=$(count)
+echo "after:  $after chall.py"
+if [ "$after" -le 1 ]; then
+  echo "RESULT: PASS (no leak)"
+else
+  echo "RESULT: FAIL (leaked $((after - 1)))"
+fi

panel/phew_trace.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""Replay the Phew checker's exact interaction, timing every step.
+
+Purpose: find WHICH read exceeds its budget. The receiver log only says
+"Timeout waiting for '> '. Got so far: <empty>", which cannot distinguish a
+slow keygen from a hang. This prints per-step wall time and the buffer state
+at the moment of the timeout.
+"""
+import os
+import select
+import subprocess
+import sys
+import time
+
+CONT = os.environ.get("PHEW_CONT", "phew_container_team1")
+CMD = ["docker", "exec", "-i", "-e", "PYTHONUNBUFFERED=1", CONT,
+       "python3", "/home/ctfuser/chall/src/chall.py"]
+
+
+def read_until(proc, needle, timeout):
+    """Mirror of the checker's _read_until, but reporting the buffer."""
+    buf = ""
+    end = time.time() + timeout
+    raw = proc.stdout.buffer if hasattr(proc.stdout, "buffer") else proc.stdout
+    while needle not in buf:
+        left = end - time.time()
+        if left <= 0:
+            raise TimeoutError(f"timeout after {timeout}s, buffer={buf!r}")
+        r, _, _ = select.select([raw], [], [], min(left, 1.0))
+        if not r:
+            continue
+        chunk = raw.read1(4096) if hasattr(raw, "read1") else raw.read(4096)
+        if not chunk:
+            raise TimeoutError(f"EOF, buffer={buf!r}")
+        buf += chunk.decode(errors="replace")
+    return buf
+
+
+def step(label, fn):
+    t0 = time.time()
+    try:
+        out = fn()
+        print(f"  {label:<34} {time.time()-t0:6.2f}s  ok")
+        return out
+    except TimeoutError as e:
+        print(f"  {label:<34} {time.time()-t0:6.2f}s  TIMEOUT {e}")
+        raise
+
+
+def main():
+    proc = subprocess.Popen(CMD, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+                            stderr=subprocess.STDOUT, text=True, bufsize=0)
+    try:
+        print(f"container={CONT}")
+        step("boot -> first menu (budget 45s)", lambda: read_until(proc, "> ", 45))
+
+        for name, send, budget in (("encrypt", "1", 30), ("decrypt", "3", 30),
+                                   ("key?", "4", 30)):
+            proc.stdin.write(send + "\n")
+            proc.stdin.flush()
+            step(f"send '{send}' -> prompt (budget 30s)",
+                 lambda: read_until(proc, "> ", 30))
+            if name == "encrypt":
+                step("  send plaintext -> prompt",
+                     lambda: read_until(proc, "> ", 30)) if False else None
+                proc.stdin.write("414243\n")
+                proc.stdin.flush()
+                step("  send plaintext -> prompt",
+                     lambda: read_until(proc, "> ", 30))
+            elif name == "key?":
+                proc.stdin.write("\n")
+                proc.stdin.flush()
+                step("  send blank -> prompt",
+                     lambda: read_until(proc, "> ", 30))
+            else:
+                proc.stdin.write("0\n")
+                proc.stdin.flush()
+                step("  send ct -> prompt",
+                     lambda: read_until(proc, "> ", 30))
+        print("ALL STEPS WITHIN BUDGET")
+    except TimeoutError:
+        print("=> a step needs a bigger budget (or a different prompt)")
+        raise SystemExit(1)
+    finally:
+        try:
+            subprocess.run(["docker", "exec", CONT, "pkill", "-f", "chall.py"],
+                           capture_output=True, timeout=20)
+        except Exception:
+            pass
+        if proc.poll() is None:
+            proc.kill()
+
+
+if __name__ == "__main__":
+    main()

panel/sla_sweep.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# Full SLA sweep across every team, SEQUENTIALLY.
+#
+# Sequential is not optional: the receivers are sync Flask apps, so hitting
+# several at once makes them contend for the same 2 CPUs and report false
+# timeouts (a pitfall already documented, and re-violated once here).
+set -uo pipefail
+BASE=/opt/gemastik18-final
+declare -A RPORT=( [1]=31080 [2]=32080 [3]=33080 [4]=34080 )
+
+echo "load: $(cut -d' ' -f1-3 /proc/loadavg)"
+for t in 1 2 3 4; do
+  ENVF=$BASE/teams/team$t/receiver/.env
+  [ -f "$ENVF" ] || { echo "=== team$t: no receiver env ==="; continue; }
+  U=$(grep -oP '^ADMIN_USERNAME=\K.*' "$ENVF")
+  P=$(grep -oP '^ADMIN_PASSWORD=\K.*' "$ENVF")
+  port=${RPORT[$t]}
+  S=$(date +%s)
+  code=$(timeout 900 curl -sS -u "$U:$P" \
+      "http://127.0.0.1:$port/check/all" \
+      -o "/tmp/sla_t$t.json" -w '%{http_code}' 2>/dev/null)
+  took=$(( $(date +%s) - S ))
+  echo "=== team$t (receiver :$port, HTTP $code, ${took}s) ==="
+  python3 - "$t" <<'PY'
+import json, sys
+t = sys.argv[1]
+try:
+    d = json.load(open(f"/tmp/sla_t{t}.json"))
+except Exception as e:
+    print("  no/invalid result:", e); raise SystemExit
+r = d.get("results", d)
+if not isinstance(r, dict):
+    print("  ", json.dumps(d)[:300]); raise SystemExit
+ok  = sorted(k for k, v in r.items() if v is True)
+bad = sorted(k for k, v in r.items() if v is not True)
+print(f"  PASS {len(ok)}/{len(r)}")
+if ok:  print("   ok :", ", ".join(ok))
+if bad: print("   BAD:", ", ".join(bad))
+PY
+done

panel/teams.py

@@ -506,29 +506,62 @@ def missing_sidecars(idx: int) -> list[str]:
     compose = svc_dir / "docker-compose.yml"
     if not compose.exists():
         return []
-    declared = _compose_service_names(compose)
+    declared = _compose_service_names(compose, project=f"team{idx}")
     if not declared:
         return []
-    want = {f"team{idx}-{n}-1" for n in declared}
+    # _compose_service_names returns the REAL container_name values
+    # (`<chall>_container_teamN`), so compare them as-is — do NOT re-wrap them
+    # in the compose default `teamN-<svc>-1`, which no service here uses.
+    want = set(declared)
     running = set(subprocess.run(["docker", "ps", "--format", "{{.Names}}"],
                                  capture_output=True, text=True).stdout.split())
     return sorted(want - running)
 
 
-def _compose_service_names(compose: Path) -> list[str]:
-    """Top-level service names from a compose file, without needing PyYAML."""
+def _compose_service_names(compose: Path, project: str = "") -> list[str]:
+    """Container names the compose will create, without needing PyYAML.
+
+    Two shapes exist in these composes and BOTH must be caught:
+
+    * services with an explicit `container_name:` (`<chall>_container_teamN`) —
+      that name is what the SLA checkers and the receiver env key off, so it
+      must be matched as-is;
+    * sidecars WITHOUT a `container_name:` (anti-alchemy-db, gemas-notes-db) —
+      compose names those `teamN-<svc>-1`, and they are exactly the ones that
+      go missing, because a per-service `up <main>` never starts them.
+
+    Matching only the first shape reported "[] missing" while the db sidecar
+    was absent on every team, and matching the compose default for everything
+    reported all 16 running challenges as missing. Both are wrong; return the
+    real name when there is one and the compose default when there isn't.
+    """
     names: list[str] = []
     in_services = False
+    pending: str | None = None
+    pfx = f"{project}-" if project else ""
     for line in compose.read_text().splitlines():
         if not line.strip() or line.lstrip().startswith("#"):
             continue
-        if not line.startswith((" ", "\t")):
-            in_services = line.rstrip() == "services:"
+        if re.match(r"^services:\s*$", line):
+            in_services = True
             continue
-        if in_services and line.startswith("  ") and not line.startswith("   "):
-            name = line.strip().rstrip(":")
-            if name and not name.startswith("-"):
-                names.append(name)
+        if in_services and re.match(r"^[a-zA-Z]", line):
+            break  # next top-level key
+        if not in_services:
+            continue
+        m = re.match(r"^  ([A-Za-z0-9_.-]+):\s*$", line)
+        if m:
+            if pending is not None:
+                # previous service had no container_name -> compose default
+                names.append(f"{pfx}{pending}-1")
+            pending = m.group(1)
+            continue
+        m = re.match(r'^\s+container_name:\s*"?([A-Za-z0-9_.-]+)"?\s*$', line)
+        if m and pending is not None:
+            names.append(m.group(1))
+            pending = None
+    if pending is not None:
+        names.append(f"{pfx}{pending}-1")
     return names
 
 

panel/verify_ssh_creds.py

@@ -40,8 +40,9 @@ def main():
             jobs.append((n, t["ports"][n]["ssh"], t.get("chall_passwords", {}).get(n)))
         ok = bad = 0
         details = []
+        users = orch.challenge_ssh_users()
         with ThreadPoolExecutor(max_workers=6) as ex:
-            futs = {ex.submit(probe, t.get("ssh_user", "ctfuser"), pw, port): n
+            futs = {ex.submit(probe, users.get(n, "root"), pw, port): n
                     for n, port, pw in jobs if pw}
             for fut, n in futs.items():
                 good, out, err = fut.result()
@@ -52,7 +53,7 @@ def main():
                 else:
                     bad += 1
                     details.append((n, f"FAIL {err}"))
-        print(f"team{idx} ({t.get('label')}): ssh {ok} ok / {bad} fail  (user={t.get('ssh_user')})")
+        print(f"team{idx} ({t.get('label')}): ssh {ok} ok / {bad} fail")
         for n, info in details:
             if info == "FAIL" or info.startswith("FAIL"):
                 print(f"    {n}: {info}")

panel/watch_load.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+# Watch the host recover after the orphan single-node containers were removed.
+# 2 CPUs + ~92 containers means the 6 leftovers (one with 58 leaked chall.py
+# processes) were the dominant load source; SLA timeouts on a healthy service
+# were a symptom of that, not of the service.
+for i in 1 2 3 4 5 6; do
+  LOAD=$(cut -d' ' -f1-3 /proc/loadavg)
+  IDLE=$(vmstat 1 2 | tail -1 | awk '{print $15}')
+  CHALL=$(ps -eo args --no-headers | grep -c '[c]hall.py')
+  echo "t+$((i * 20))s  load=$LOAD  idle=${IDLE}%  chall.py=$CHALL"
+  sleep 20
+done