Cyrene
aa0bb45633
Three independent root causes, all found by measuring instead of assuming:
1. SSH failed on 10/16 challenges while state.json looked perfect.
Only the 6 native GEMASTIK XVIII images provision 'ctfuser'; every imported
XVI/XVII image does 'echo root:${PASSWORD} | chpasswd' and logs in as root.
set_ssh_passwords() hardcoded ctfuser, so chpasswd set a password nobody
used -> 'Permission denied' on every team. Registry gains a per-challenge
'ssh_user'; chpasswd targets the real login and reports failures loudly.
2. phew SLA timed out on a healthy service, four bugs stacked:
- chall.py block-buffers stdout through the exec pipe (PYTHONUNBUFFERED now
set) and does a fresh Pailier keygen (~12 s) before printing its menu;
- _read_until read a TEXT pipe, so read(1) pulled 8 KB into Python's
TextIOWrapper buffer and select() then blocked on data already in memory;
- its buffer was per-call, so the read satisfying 'pt (hex)' also swallowed
the '> ' the next call waited for -> a race that failed intermittently;
- reaping killed chall.py it did not own: a blanket pkill -f, a
snapshot-diff (concurrent sessions diff against the same pre-spawn set),
and a class-level _children shared across uvicorn's thread pool. The child
now prints its own pid so exactly one session is reaped.
Also: ONE interactive session per check instead of five spawns (Paillier is
randomized per ciphertext, not per process) - 5 keygens were the CPU load
that starved the checks. And the 6 orphan single-node containers from the
original deploy were removed; one held 58 leaked chall.py and drove load
average 76 on 2 CPUs.
3. missing_sidecars() matched compose-generated names (teamN-<svc>-1) while
every service sets an explicit container_name, so it reported all 16 running
challenges as missing and hid the one real gap (anti-alchemy-db, which has
no container_name). Now reads container_name when present and falls back to
the compose default otherwise.
Verified: 64/64 SLA across 4 teams; 64/64 real SSH logins succeed with
correct <chall>_teamN hostnames; phew 3/3 sequential with no process leak.
Adds panel/verify_ssh_creds.py, audit_ssh_users.sh, reset_runtime.sh,
sla_sweep.sh, fix_sidecars.sh, phew_concurrency_test.sh, exec_probe_i.py.
55 lines
1.8 KiB
Python
55 lines
1.8 KiB
Python
#!/usr/bin/env python3
|
|
"""How many concurrent INTERACTIVE (`docker exec -i`) sessions does this host take?
|
|
|
|
Trivial non-interactive execs succeed 8-way, but the Phew SLA checker's 5-way
|
|
interactive test failed with
|
|
|
|
failed to open stdin fifo: error creating fifo ... -stdin: no such file
|
|
No such exec instance: <id>
|
|
|
|
which is a containerd exec-session limit, not a checker bug. Interactive execs
|
|
allocate a fifo + a tracked exec instance, so they hit a ceiling that plain
|
|
`docker exec <c> true` never approaches.
|
|
|
|
python3 panel/exec_probe_i.py [container] [N]
|
|
"""
|
|
import subprocess
|
|
import sys
|
|
import time
|
|
from concurrent.futures import ThreadPoolExecutor
|
|
|
|
CONT = sys.argv[1] if len(sys.argv) > 1 else "phew_container_team1"
|
|
N = int(sys.argv[2]) if len(sys.argv) > 2 else 8
|
|
|
|
|
|
def interactive(i: int) -> tuple[int, str]:
|
|
"""Mimic the checker: `exec -i`, write a line, read the reply, exit."""
|
|
p = subprocess.Popen(
|
|
["docker", "exec", "-i", CONT, "sh", "-c",
|
|
"echo $$; read line; echo \"got:$line\""],
|
|
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
|
|
stderr=subprocess.STDOUT, text=True, bufsize=0)
|
|
try:
|
|
out, _ = p.communicate("hello\n", timeout=45)
|
|
except subprocess.TimeoutExpired:
|
|
p.kill()
|
|
out = "TIMEOUT"
|
|
return p.returncode, (out or "").strip().replace("\n", " | ")[:150]
|
|
|
|
|
|
def main() -> None:
|
|
print(f"container={CONT} N={N} interactive execs")
|
|
ok = 0
|
|
with ThreadPoolExecutor(max_workers=N) as ex:
|
|
for rc, out in ex.map(interactive, range(N)):
|
|
if rc == 0 and "got:hello" in out:
|
|
ok += 1
|
|
else:
|
|
print(f" FAIL rc={rc}: {out}")
|
|
print(f" {ok}/{N} interactive execs OK")
|
|
print("RESULT:", "PASS" if ok == N else f"limit reached at {ok}/{N}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|