fix(sla): 100% fleet SLA (64/64) - per-challenge SSH login, phew checker, sidecar detection
Three independent root causes, all found by measuring instead of assuming:
1. SSH failed on 10/16 challenges while state.json looked perfect.
Only the 6 native GEMASTIK XVIII images provision 'ctfuser'; every imported
XVI/XVII image does 'echo root:${PASSWORD} | chpasswd' and logs in as root.
set_ssh_passwords() hardcoded ctfuser, so chpasswd set a password nobody
used -> 'Permission denied' on every team. Registry gains a per-challenge
'ssh_user'; chpasswd targets the real login and reports failures loudly.
2. phew SLA timed out on a healthy service, four bugs stacked:
- chall.py block-buffers stdout through the exec pipe (PYTHONUNBUFFERED now
set) and does a fresh Pailier keygen (~12 s) before printing its menu;
- _read_until read a TEXT pipe, so read(1) pulled 8 KB into Python's
TextIOWrapper buffer and select() then blocked on data already in memory;
- its buffer was per-call, so the read satisfying 'pt (hex)' also swallowed
the '> ' the next call waited for -> a race that failed intermittently;
- reaping killed chall.py it did not own: a blanket pkill -f, a
snapshot-diff (concurrent sessions diff against the same pre-spawn set),
and a class-level _children shared across uvicorn's thread pool. The child
now prints its own pid so exactly one session is reaped.
Also: ONE interactive session per check instead of five spawns (Paillier is
randomized per ciphertext, not per process) - 5 keygens were the CPU load
that starved the checks. And the 6 orphan single-node containers from the
original deploy were removed; one held 58 leaked chall.py and drove load
average 76 on 2 CPUs.
3. missing_sidecars() matched compose-generated names (teamN-<svc>-1) while
every service sets an explicit container_name, so it reported all 16 running
challenges as missing and hid the one real gap (anti-alchemy-db, which has
no container_name). Now reads container_name when present and falls back to
the compose default otherwise.
Verified: 64/64 SLA across 4 teams; 64/64 real SSH logins succeed with
correct <chall>_teamN hostnames; phew 3/3 sequential with no process leak.
Adds panel/verify_ssh_creds.py, audit_ssh_users.sh, reset_runtime.sh,
sla_sweep.sh, fix_sidecars.sh, phew_concurrency_test.sh, exec_probe_i.py.
This commit is contained in:
+43
-10
@@ -506,29 +506,62 @@ def missing_sidecars(idx: int) -> list[str]:
|
||||
compose = svc_dir / "docker-compose.yml"
|
||||
if not compose.exists():
|
||||
return []
|
||||
declared = _compose_service_names(compose)
|
||||
declared = _compose_service_names(compose, project=f"team{idx}")
|
||||
if not declared:
|
||||
return []
|
||||
want = {f"team{idx}-{n}-1" for n in declared}
|
||||
# _compose_service_names returns the REAL container_name values
|
||||
# (`<chall>_container_teamN`), so compare them as-is — do NOT re-wrap them
|
||||
# in the compose default `teamN-<svc>-1`, which no service here uses.
|
||||
want = set(declared)
|
||||
running = set(subprocess.run(["docker", "ps", "--format", "{{.Names}}"],
|
||||
capture_output=True, text=True).stdout.split())
|
||||
return sorted(want - running)
|
||||
|
||||
|
||||
def _compose_service_names(compose: Path) -> list[str]:
|
||||
"""Top-level service names from a compose file, without needing PyYAML."""
|
||||
def _compose_service_names(compose: Path, project: str = "") -> list[str]:
|
||||
"""Container names the compose will create, without needing PyYAML.
|
||||
|
||||
Two shapes exist in these composes and BOTH must be caught:
|
||||
|
||||
* services with an explicit `container_name:` (`<chall>_container_teamN`) —
|
||||
that name is what the SLA checkers and the receiver env key off, so it
|
||||
must be matched as-is;
|
||||
* sidecars WITHOUT a `container_name:` (anti-alchemy-db, gemas-notes-db) —
|
||||
compose names those `teamN-<svc>-1`, and they are exactly the ones that
|
||||
go missing, because a per-service `up <main>` never starts them.
|
||||
|
||||
Matching only the first shape reported "[] missing" while the db sidecar
|
||||
was absent on every team, and matching the compose default for everything
|
||||
reported all 16 running challenges as missing. Both are wrong; return the
|
||||
real name when there is one and the compose default when there isn't.
|
||||
"""
|
||||
names: list[str] = []
|
||||
in_services = False
|
||||
pending: str | None = None
|
||||
pfx = f"{project}-" if project else ""
|
||||
for line in compose.read_text().splitlines():
|
||||
if not line.strip() or line.lstrip().startswith("#"):
|
||||
continue
|
||||
if not line.startswith((" ", "\t")):
|
||||
in_services = line.rstrip() == "services:"
|
||||
if re.match(r"^services:\s*$", line):
|
||||
in_services = True
|
||||
continue
|
||||
if in_services and line.startswith(" ") and not line.startswith(" "):
|
||||
name = line.strip().rstrip(":")
|
||||
if name and not name.startswith("-"):
|
||||
names.append(name)
|
||||
if in_services and re.match(r"^[a-zA-Z]", line):
|
||||
break # next top-level key
|
||||
if not in_services:
|
||||
continue
|
||||
m = re.match(r"^ ([A-Za-z0-9_.-]+):\s*$", line)
|
||||
if m:
|
||||
if pending is not None:
|
||||
# previous service had no container_name -> compose default
|
||||
names.append(f"{pfx}{pending}-1")
|
||||
pending = m.group(1)
|
||||
continue
|
||||
m = re.match(r'^\s+container_name:\s*"?([A-Za-z0-9_.-]+)"?\s*$', line)
|
||||
if m and pending is not None:
|
||||
names.append(m.group(1))
|
||||
pending = None
|
||||
if pending is not None:
|
||||
names.append(f"{pfx}{pending}-1")
|
||||
return names
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user