fix(sla): 100% fleet SLA (64/64) - per-challenge SSH login, phew checker, sidecar detection
Three independent root causes, all found by measuring instead of assuming:
1. SSH failed on 10/16 challenges while state.json looked perfect.
Only the 6 native GEMASTIK XVIII images provision 'ctfuser'; every imported
XVI/XVII image does 'echo root:${PASSWORD} | chpasswd' and logs in as root.
set_ssh_passwords() hardcoded ctfuser, so chpasswd set a password nobody
used -> 'Permission denied' on every team. Registry gains a per-challenge
'ssh_user'; chpasswd targets the real login and reports failures loudly.
2. phew SLA timed out on a healthy service, four bugs stacked:
- chall.py block-buffers stdout through the exec pipe (PYTHONUNBUFFERED now
set) and does a fresh Pailier keygen (~12 s) before printing its menu;
- _read_until read a TEXT pipe, so read(1) pulled 8 KB into Python's
TextIOWrapper buffer and select() then blocked on data already in memory;
- its buffer was per-call, so the read satisfying 'pt (hex)' also swallowed
the '> ' the next call waited for -> a race that failed intermittently;
- reaping killed chall.py it did not own: a blanket pkill -f, a
snapshot-diff (concurrent sessions diff against the same pre-spawn set),
and a class-level _children shared across uvicorn's thread pool. The child
now prints its own pid so exactly one session is reaped.
Also: ONE interactive session per check instead of five spawns (Paillier is
randomized per ciphertext, not per process) - 5 keygens were the CPU load
that starved the checks. And the 6 orphan single-node containers from the
original deploy were removed; one held 58 leaked chall.py and drove load
average 76 on 2 CPUs.
3. missing_sidecars() matched compose-generated names (teamN-<svc>-1) while
every service sets an explicit container_name, so it reported all 16 running
challenges as missing and hid the one real gap (anti-alchemy-db, which has
no container_name). Now reads container_name when present and falls back to
the compose default otherwise.
Verified: 64/64 SLA across 4 teams; 64/64 real SSH logins succeed with
correct <chall>_teamN hostnames; phew 3/3 sequential with no process leak.
Adds panel/verify_ssh_creds.py, audit_ssh_users.sh, reset_runtime.sh,
sla_sweep.sh, fix_sidecars.sh, phew_concurrency_test.sh, exec_probe_i.py.
This commit is contained in:
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env python3
|
||||
"""How many concurrent INTERACTIVE (`docker exec -i`) sessions does this host take?
|
||||
|
||||
Trivial non-interactive execs succeed 8-way, but the Phew SLA checker's 5-way
|
||||
interactive test failed with
|
||||
|
||||
failed to open stdin fifo: error creating fifo ... -stdin: no such file
|
||||
No such exec instance: <id>
|
||||
|
||||
which is a containerd exec-session limit, not a checker bug. Interactive execs
|
||||
allocate a fifo + a tracked exec instance, so they hit a ceiling that plain
|
||||
`docker exec <c> true` never approaches.
|
||||
|
||||
python3 panel/exec_probe_i.py [container] [N]
|
||||
"""
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
CONT = sys.argv[1] if len(sys.argv) > 1 else "phew_container_team1"
|
||||
N = int(sys.argv[2]) if len(sys.argv) > 2 else 8
|
||||
|
||||
|
||||
def interactive(i: int) -> tuple[int, str]:
|
||||
"""Mimic the checker: `exec -i`, write a line, read the reply, exit."""
|
||||
p = subprocess.Popen(
|
||||
["docker", "exec", "-i", CONT, "sh", "-c",
|
||||
"echo $$; read line; echo \"got:$line\""],
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, text=True, bufsize=0)
|
||||
try:
|
||||
out, _ = p.communicate("hello\n", timeout=45)
|
||||
except subprocess.TimeoutExpired:
|
||||
p.kill()
|
||||
out = "TIMEOUT"
|
||||
return p.returncode, (out or "").strip().replace("\n", " | ")[:150]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
print(f"container={CONT} N={N} interactive execs")
|
||||
ok = 0
|
||||
with ThreadPoolExecutor(max_workers=N) as ex:
|
||||
for rc, out in ex.map(interactive, range(N)):
|
||||
if rc == 0 and "got:hello" in out:
|
||||
ok += 1
|
||||
else:
|
||||
print(f" FAIL rc={rc}: {out}")
|
||||
print(f" {ok}/{N} interactive execs OK")
|
||||
print("RESULT:", "PASS" if ok == N else f"limit reached at {ok}/{N}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user