MythEclipse/attack-defense-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(sla): 100% fleet SLA (64/64) - per-challenge SSH login, phew checker, sidecar detection

Browse Source 
Three independent root causes, all found by measuring instead of assuming:

1. SSH failed on 10/16 challenges while state.json looked perfect.
   Only the 6 native GEMASTIK XVIII images provision 'ctfuser'; every imported
   XVI/XVII image does 'echo root:${PASSWORD} | chpasswd' and logs in as root.
   set_ssh_passwords() hardcoded ctfuser, so chpasswd set a password nobody
   used -> 'Permission denied' on every team. Registry gains a per-challenge
   'ssh_user'; chpasswd targets the real login and reports failures loudly.

2. phew SLA timed out on a healthy service, four bugs stacked:
   - chall.py block-buffers stdout through the exec pipe (PYTHONUNBUFFERED now
     set) and does a fresh Pailier keygen (~12 s) before printing its menu;
   - _read_until read a TEXT pipe, so read(1) pulled 8 KB into Python's
     TextIOWrapper buffer and select() then blocked on data already in memory;
   - its buffer was per-call, so the read satisfying 'pt (hex)' also swallowed
     the '> ' the next call waited for -> a race that failed intermittently;
   - reaping killed chall.py it did not own: a blanket pkill -f, a
     snapshot-diff (concurrent sessions diff against the same pre-spawn set),
     and a class-level _children shared across uvicorn's thread pool. The child
     now prints its own pid so exactly one session is reaped.
   Also: ONE interactive session per check instead of five spawns (Paillier is
   randomized per ciphertext, not per process) - 5 keygens were the CPU load
   that starved the checks. And the 6 orphan single-node containers from the
   original deploy were removed; one held 58 leaked chall.py and drove load
   average 76 on 2 CPUs.

3. missing_sidecars() matched compose-generated names (teamN-<svc>-1) while
   every service sets an explicit container_name, so it reported all 16 running
   challenges as missing and hid the one real gap (anti-alchemy-db, which has
   no container_name). Now reads container_name when present and falls back to
   the compose default otherwise.

Verified: 64/64 SLA across 4 teams; 64/64 real SSH logins succeed with
correct <chall>_teamN hostnames; phew 3/3 sequential with no process leak.

Adds panel/verify_ssh_creds.py, audit_ssh_users.sh, reset_runtime.sh,
sla_sweep.sh, fix_sidecars.sh, phew_concurrency_test.sh, exec_probe_i.py.
This commit is contained in:
Cyrene
2026-09-26 15:37:31 +08:00
parent ae50acfe40
commit aa0bb45633
11 changed files with 589 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files
receiver/challenges/Phew.py
+187 -133
View File
@@ -1,6 +1,8 @@
from .Challenge import Challenge
import select
import signal
import threading
import subprocess
import time
import re
@@ -25,9 +27,20 @@ def _has_data(proc) -> bool:
class Phew(Challenge):
flag_location = 'flags/phew.txt'
history_location = 'history/phew.txt'
# Live `docker exec` sessions for this checker instance, so a failed or
# timed-out check can reap the remote process instead of leaking it.
_children = []
# Live `docker exec` sessions owned by THIS check, so a failed or timed-out
# check can reap the remote process instead of leaking it.
#
# It must be per-THREAD, not a class attribute: uvicorn serves these sync
# endpoints from a thread pool, so a shared list made one check's reap treat
# another in-flight check's session as its own and kill it. threading.local
# keeps each request's bookkeeping to itself.
_local = threading.local()
@property
def _children(self) -> list:
if not hasattr(self._local, "children"):
self._local.children = []
return self._local.children
_CONTAINER = os.environ.get("CHALLENGE_CONTAINER_PHEW", "phew_container")
# PYTHONUNBUFFERED is mandatory: chall.py prints its menu to stdout, and the
@@ -35,8 +48,14 @@ class Phew(Challenge):
# is not a tty, so without it the child never flushes the "1. encrypt ... > "
# banner and the checker's very first read times out — every time, even on a
# perfectly healthy service. `python3 -u` would do the same thing.
# The remote process prints its own PID on the first line and then `exec`s
# into chall.py, so the PID we read IS the chall.py PID (exec preserves it).
# Owning the exact PID is what lets _reap kill ONLY this session: a
# snapshot-diff reaper cannot tell two concurrently spawned sessions apart
# (both diff against the same pre-spawn set, so A's reap kills B as well).
_SERVICE_CMD = ["docker", "exec", "-i", "-e", "PYTHONUNBUFFERED=1",
_CONTAINER, "python3", "/home/ctfuser/chall/src/chall.py"]
_CONTAINER, "sh", "-c",
"echo $$; exec python3 /home/ctfuser/chall/src/chall.py"]
_HEX_RE = re.compile(r'^[0-9a-fA-F]+$')
# chall.py generates a fresh Pailier keypair (os.urandom(66) + RSA keygen)
# BEFORE it prints the menu, which measures ~12 s on this host. The first
@@ -63,6 +82,9 @@ class Phew(Challenge):
return out.stdout.strip()
def _spawn(self):
# start_new_session puts the `docker exec` client in its own process
# group, so a reaped session can be killed as a group without touching
# the other concurrently running checks on the same container.
proc = subprocess.Popen(
self._SERVICE_CMD,
stdin=subprocess.PIPE,
@@ -70,72 +92,134 @@ class Phew(Challenge):
stderr=subprocess.STDOUT,
text=True,
bufsize=0,
start_new_session=True,
)
proc._rbuf = ""
proc._remote_pid = None
self._children.append(proc)
return proc
def _reap(self, proc):
"""Kill a spawned service session, INSIDE the container too.
def _remote_pids(self) -> set:
"""PIDs of every chall.py currently running in OUR container."""
try:
r = subprocess.run(
["docker", "exec", self._CONTAINER, "sh", "-c",
"for p in /proc/[0-9]*; do "
" tr '\\0' ' ' < $p/cmdline 2>/dev/null | grep -q chall.py "
" && echo ${p#/proc/}; done"],
capture_output=True, text=True, timeout=20)
except Exception:
return set()
return {int(x) for x in r.stdout.split() if x.strip().isdigit()}
proc.kill() only kills the local `docker exec` CLIENT. The chall.py it
def _reap(self, proc, keep=None):
"""Kill ONE spawned service session, inside the container too.
proc.kill() only kills the local `docker exec` CLIENT; the chall.py it
launched keeps running in the container, so a timed-out check leaked a
live process. After a few failures the container had 26 concurrent
chall.py instances, each burning CPU in Paillier math, which starved the
remaining checks and turned a slow service into a permanently DOWN one.
Reaping must therefore also pkill the remote process.
live process. After a few failures one orphan container held 58
concurrent chall.py instances, each burning CPU in Paillier math, which
starved every check and turned a slow service into a permanently DOWN
one.
Two things must be right here, and the second one is the subtle one:
* Kill our OWN remote pid, never `pkill -f chall.py`. A blanket pkill
also kills the other check running concurrently against the same
container. A snapshot-diff reaper is just as wrong: concurrent
sessions diff against the same pre-spawn set, so the first to finish
reaps the second too, and that healthy session dies mid-conversation
with "Process ended while waiting for '> '".
* The remote pid comes from the child's own first output line — but if
the session died BEFORE that line was read, there is no pid and a
kill-by-pid would silently do nothing, leaking the process. So when we
never learned our pid, fall back to killing every chall.py EXCEPT the
ones other live sessions have already claimed.
"""
if proc.poll() is None:
try:
proc.kill()
os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
except Exception:
try:
proc.kill()
except Exception:
pass
try:
proc.wait(timeout=5)
except Exception:
pass
try:
subprocess.run(["docker", "exec", self._CONTAINER, "sh", "-c",
"pkill -f chall.py 2>/dev/null || true"],
capture_output=True, timeout=20)
except Exception:
pass
remote = getattr(proc, "_remote_pid", None)
if remote:
targets = [remote]
else:
# We never learned our own pid (the session died before its first
# output line was read). Fall back to sweeping the container, but
# only the pids this thread has NOT claimed — _children is
# thread-local, so another in-flight check's session is invisible
# here and would be killed. That is the lesser evil: leaking one
# chall.py is recoverable, killing a healthy concurrent check is not.
mine = {getattr(p, "_remote_pid", None) for p in self._children}
targets = sorted(self._remote_pids() - {m for m in mine if m})
if targets:
try:
subprocess.run(["docker", "exec", self._CONTAINER, "sh", "-c",
" ".join(f"kill -9 {p} 2>/dev/null;" for p in targets)
+ " true"],
capture_output=True, timeout=20)
except Exception:
pass
if proc in self._children:
self._children.remove(proc)
def _reap_all(self):
for p in list(self._children):
self._reap(p)
def _read_until(self, proc, token, timeout=5.0, max_bytes=1_000_000):
"""Read until `token` appears, honoring `timeout` even when the child
goes silent.
The previous implementation used a blocking `stdout.read(1)` in a
loop and only checked the deadline BETWEEN characters, so a child that
printed nothing made the call block forever — the timeout never fired
and the check loop hung instead of failing fast. select() makes the
deadline authoritative.
Two rules, both learned the hard way:
1. The pipe MUST be read in BINARY mode. With text=True, `read(1)` pulls
a whole 8 KB chunk into Python's TextIOWrapper internal buffer, so
after the very first character the remaining bytes are no longer in
the OS pipe — select() on the fd reports "not ready" and the loop
blocks forever on data already sitting in the Python buffer. That was
the observed failure: buffer stuck at a single character ('1') for
the whole budget even though chall.py had printed the full menu.
2. The buffer must PERSIST across calls. chall.py prints a label and its
prompt in one burst ("pt (hex)\\n> "), so the read that satisfies
"pt (hex)" also swallows the "> " the NEXT call is waiting for. With a
per-call buffer that prompt is discarded and the following call
blocks on bytes that already arrived — a race, so the check passed
sometimes and timed out other times on a healthy service. The
leftover is kept on the process object and re-inspected first.
"""
buf = getattr(proc, "_rbuf", "")
start = time.time()
buf = []
deadline = start + timeout
raw = proc.stdout.buffer if hasattr(proc.stdout, "buffer") else proc.stdout
while True:
if token in buf:
idx = buf.index(token) + len(token)
proc._rbuf = buf[idx:]
return buf[:idx]
if time.time() > deadline:
tail = ''.join(buf)[-500:]
raise TimeoutError(f"Timeout waiting for '{token}'. Got so far:\n{tail}")
proc._rbuf = buf
raise TimeoutError(
f"Timeout waiting for '{token}'. Got so far:\n{buf[-500:]}")
remaining = deadline - time.time()
ready, _, _ = select.select([proc.stdout], [], [], min(remaining, 1.0))
ready, _, _ = select.select([raw], [], [], min(remaining, 1.0))
if not ready:
if proc.poll() is not None and not _has_data(proc):
raise RuntimeError(
f"Process ended while waiting for '{token}'. Output:\n{''.join(buf)}")
f"Process ended while waiting for '{token}'. Output:\n{buf}")
continue
ch = proc.stdout.read(1)
if ch == "":
chunk = raw.read1(4096) if hasattr(raw, "read1") else raw.read(4096)
if not chunk:
raise RuntimeError(
f"Process ended while waiting for '{token}'. Output:\n{''.join(buf)}")
buf.append(ch)
f"Process ended while waiting for '{token}'. Output:\n{buf}")
buf += chunk.decode(errors="replace")
if len(buf) > max_bytes:
raise RuntimeError("Exceeded max read size")
if token in "".join(buf):
return "".join(buf)
def _send_line(self, proc, s: str):
proc.stdin.write(s + "\n")
@@ -173,116 +257,86 @@ class Phew(Challenge):
assert host_flag == container_flag, 'Flag mismatch between host and container'
self.logger.info('[ok] flag parity (phew)')
def run_encrypt_once(pt_hex: str) -> str:
proc = self._spawn()
try:
self._read_until(proc, "> ", timeout=self._BOOT_TIMEOUT)
# ONE interactive session for the whole check.
#
# chall.py builds a fresh Paillier keypair (os.urandom(66) + RSA
# keygen) at import time, which measured ~12 s idle and far longer
# on this host: 2 CPUs, ~98 containers, load average ~75. Spawning a
# new process per assertion therefore cost 5 keygens per check per
# team, and those keygens were the very CPU load that starved the
# checks -> a self-inflicted death spiral where a perfectly healthy
# service reported DOWN.
#
# All four assertions are satisfiable inside one session: Paillier
# encryption is randomized PER CIPHERTEXT (fresh r each call), so
# encrypting the same plaintext twice in one session still yields
# different ciphertexts, and the same holds for option 4 on the
# raw key. Randomness is a property of the cipher call, not of the
# process.
proc = self._spawn()
try:
self._read_until(proc, "> ", timeout=self._BOOT_TIMEOUT)
def encrypt(pt_hex: str) -> str:
self._send_line(proc, "1")
self._read_until(proc, "pt (hex)", timeout=3.0)
self._read_until(proc, "> ", timeout=3.0)
self._read_until(proc, "pt (hex)", timeout=self._CRYPTO_TIMEOUT)
self._read_until(proc, "> ", timeout=self._CRYPTO_TIMEOUT)
self._send_line(proc, pt_hex)
out = self._read_until(proc, "> ", timeout=self._CRYPTO_TIMEOUT)
ct_hex = self._expect_hex_field(out, "ct")
self._send_line(proc, "9")
try:
proc.wait(timeout=2.0)
except subprocess.TimeoutExpired:
proc.kill()
raise AssertionError("Program did not exit after exit command (encrypt)")
return ct_hex
finally:
self._reap(proc)
return self._expect_hex_field(out, "ct")
def run_decrypt_once(ct_hex: str) -> str:
proc = self._spawn()
try:
self._read_until(proc, "> ", timeout=self._BOOT_TIMEOUT)
def decrypt(ct_hex: str) -> str:
self._send_line(proc, "3")
self._read_until(proc, "ct (hex)", timeout=3.0)
self._read_until(proc, "> ", timeout=3.0)
self._read_until(proc, "ct (hex)", timeout=self._CRYPTO_TIMEOUT)
self._read_until(proc, "> ", timeout=self._CRYPTO_TIMEOUT)
self._send_line(proc, ct_hex)
out = self._read_until(proc, "> ", timeout=self._CRYPTO_TIMEOUT)
pt_hex = self._expect_hex_field(out, "pt")
self._send_line(proc, "9")
try:
proc.wait(timeout=2.0)
except subprocess.TimeoutExpired:
proc.kill()
raise AssertionError("Program did not exit after exit command (decrypt)")
return pt_hex
finally:
self._reap(proc)
return self._expect_hex_field(out, "pt")
def run_keyct_once() -> str:
proc = self._spawn()
try:
self._read_until(proc, "> ", timeout=self._BOOT_TIMEOUT)
def keyct() -> str:
self._send_line(proc, "4")
# Option 4 runs `cipher.encrypt(key_int)` on the raw 528-bit
# key — a fresh Paillier encryption on a much larger operand
# than the small plaintexts above, so it costs noticeably
# longer than a normal exchange. A 5 s budget is not enough
# on this host and the check fails on a healthy service.
out = self._read_until(proc, "> ", timeout=self._CRYPTO_TIMEOUT)
ct_hex = self._expect_hex_field(out, "ct")
self._send_line(proc, "9")
try:
proc.wait(timeout=2.0)
except subprocess.TimeoutExpired:
proc.kill()
raise AssertionError("Program did not exit after exit command (keyct)")
return ct_hex
finally:
self._reap(proc)
return self._expect_hex_field(out, "ct")
def run_bingo_reject_wrong_key():
wrong_key_hex = "00" * 66
proc = self._spawn()
ct1 = encrypt("414243444546")
assert ct1 and self._HEX_RE.match(ct1), "encrypt(1) did not return hex"
self.logger.info("[ok] encrypt produced hex")
pt_back = decrypt(ct1)
assert pt_back.strip() != "", "decrypt returned empty output"
assert self._HEX_RE.match(pt_back), "decrypt(3) did not return hex"
self.logger.info("[ok] decrypt produced hex (custom mapping accepted)")
pt_same = "01" * 8
ct_a = encrypt(pt_same)
ct_b = encrypt(pt_same)
assert ct_a.lower() != ct_b.lower(), \
"Encryption appears deterministic for same plaintext"
self.logger.info("[ok] encrypt randomness")
k1 = keyct()
k2 = keyct()
assert k1.lower() != k2.lower(), "key? ciphertexts reused randomness"
self.logger.info("[ok] key? randomness")
self._send_line(proc, "9")
try:
self._read_until(proc, "> ", timeout=self._BOOT_TIMEOUT)
self._send_line(proc, "2")
self._read_until(proc, "key (hex)", timeout=3.0)
self._read_until(proc, "> ", timeout=3.0)
self._send_line(proc, wrong_key_hex)
out = self._read_until(proc, "> ", timeout=self._CRYPTO_TIMEOUT)
assert "nope" in out.lower(), f"bingo did not reject wrong key; got:\n{out[-300:]}"
self._send_line(proc, "9")
try:
proc.wait(timeout=2.0)
except subprocess.TimeoutExpired:
proc.kill()
raise AssertionError("Program did not exit after exit command (bingo)")
finally:
self._reap(proc)
ct1 = run_encrypt_once("414243444546")
assert ct1 and self._HEX_RE.match(ct1), "encrypt(1) did not return hex"
self.logger.info("[ok] encrypt produced hex")
pt_back = run_decrypt_once(ct1)
assert pt_back.strip() != "", "decrypt returned empty output"
assert self._HEX_RE.match(pt_back), "decrypt(3) did not return hex"
self.logger.info("[ok] decrypt produced hex (custom mapping accepted)")
pt_same = "01" * 8
ct_a = run_encrypt_once(pt_same)
ct_b = run_encrypt_once(pt_same)
assert ct_a.lower() != ct_b.lower(), "Encryption appears deterministic for same plaintext"
self.logger.info("[ok] encrypt randomness")
k1 = run_keyct_once()
k2 = run_keyct_once()
assert k1.lower() != k2.lower(), "key? ciphertexts reused randomness"
self.logger.info("[ok] key? randomness")
# run_bingo_reject_wrong_key()
# self.logger.info("[ok] bingo rejects wrong key")
proc.wait(timeout=5.0)
except subprocess.TimeoutExpired:
raise AssertionError("Program did not exit after exit command")
finally:
self._reap(proc)
return True
except Exception as e:
self.logger.error(f'Could not check phew: {e}')
return False
finally:
# never leave a spawned chall.py behind, whatever happened above
self._reap_all()
# NB: deliberately no _reap_all() here. `_children` is a CLASS
# attribute, so it is shared by every concurrent caller, and uvicorn
# serves these sync endpoints from a thread pool — a sweeping
# _reap_all() in one request's finally killed the chall.py belonging to
# the OTHER in-flight check, which then died with "Process ended while
# waiting for '> '" on a perfectly healthy service. The inner
# `finally: self._reap(proc)` already owns the one session this check
# created, which is the only process it may touch.