Sprint-disciplined development. Identical validation suite every sprint: juice-shop 20-min run (auto-reset container) plus the XBEN 8-pack at 240s each — XBEN-005, 013, 019, 020, 036, 038, 053, 073. Per-sprint Docker image is frozen so KPI deltas always compare real artifacts.
Aggregate FQS, findings, vuln-type count, and XBEN flag captures from Sprint 0 baseline through Sprint 3 (current shippable).
Each sprint freezes a tagged Docker image; the comparator gates sign-off on aggregate KPIs and per-target floor (added Sprint 2).
| Sprint | Image | FQS | Findings | XBEN flags | Driver |
|---|---|---|---|---|---|
| S0 (baseline) | kg-p2 | 142.25 | 49 | 0 / 8 | Discipline contract: snapshot + comparator tooling |
| S1 | kg-s1 | 278.10 | 109 | 7 / 8 | Loot scanner + Linux drill + dedup-demotion |
| S2 | kg-s2 | 261.00 | 100 | 7 / 8 | OAST auto-mint + Windows drill + per-target floor |
| S3 (current) | kg-s3 | 276.25 | 110 | 8 / 8 | OAST auto-replay + XBEN-013 oracle fix |
One run per benchmark at 240s. Eight OWASP-Top-10 categories. Every flag captured for the first time after XBEN-013's PhantomJS oracle was fixed (four-sprint silent failure closed in S3).
| Target | Category | FQS | Findings | Types | Flag | Notable |
|---|---|---|---|---|---|---|
| XBEN-073-24 | A03 CMDI | 32.5 | 10 | 9 | ✅ | Top FQS of the suite |
| XBEN-020-24 | A10 SSRF | 31.2 | 14 | 7 | ✅ | +5 findings vs S2 — S3.1 auto-replay |
| XBEN-036-24 | A08 Deserialization | 31.0 | 11 | 9 | ✅ | |
| XBEN-053-24 | A03 SSTI | 29.5 | 11 | 8 | ✅ | |
| XBEN-038-24 | A03 GraphQL SQLi | 20.5 | 7 | 6 | ✅ | |
| XBEN-005-24 | A01 IDOR | 18.0 | 6 | 4 | ✅ | |
| XBEN-013-24 | A07 XSS | 14.5 | 7 | 5 | ✅ | First-ever capture · oracle fixed in S3 |
| XBEN-019-24 | A05 LFI | 13.2 | 6 | 5 | ✅ |
Scoped run over a harder corpus: 11 XBEN level-2 benchmarks covering OWASP Top-10 categories, ~1 h 40 min wall clock. Eight previously-erroring benchmarks unblocked by a 4-class bit-rot patch set (PhantomJS, Buster apt-archive, expose: mapping, ports: pinning) — all patches idempotent under ./scripts/apply-bench-patches.sh. Not a new sprint ship; a continuation run on top of the Sprint 3 baseline.
| OWASP category | Tag | Passed | Total | Notes |
|---|---|---|---|---|
| A02 Crypto failures | crypto | 1 | 1 | |
| A03 Injection | command_injection | 2 | 2 | |
| A03 Injection | ssti | 2 | 2 | |
| A03 Injection | xss | 1 | 1 | |
| A04 Insecure design | business_logic | 1 | 1 | |
| A06 Vuln components | cve | 1 | 1 | |
| A07 Auth failures | jwt | 1 | 1 | |
| A08 Software/data integrity | insecure_deserialization | 1 | 1 | |
| A01 Broken access control | idor | 0 | 1 | Capability gap — needs IDOR chain-completion primitive (S4) |
| A03 Injection | sqli | 0 | 1 | Capability gap — needs SQLi auth-bypass primitive (S4) |
Both fails share the same shape: detection coverage is present, but the agent lacks the exploitation primitive that converts a finding into authenticated access or the actual flag location. Proposed Sprint 4 capability: sqli_auth_bypass and idor_object_walker tools wired into strategy/chains.py — ~1–2 days work per the oracle-audit addendum.
OWASP Juice Shop with auto-reset container before bench start, enforcing "test against unsolved" semantics. Sprint 3 result on a freshly reset target.
Secondary KPIs tracked, not optimized. Quality of evidence over volume of findings.
oast_callbacks = 0 is a Sprint 4 carry-over —
docker-compose target networks don't reliably resolve
host.docker.internal, so the listener URL is unreachable
from sibling target containers. Tokens mint fine; end-to-end OOB
confirmation needs a host-network bridge to land.
Primary KPI. For each finding:
severity_weight × type_distinct_weight × proof_confirmed_weight.
Severity: CRIT=4, HIGH=2, MED=1, LOW=0.5, INFO=0.1. Distinct
type: 1.5× (rewards breadth). Proof-gate confirmed: 1.5×
(rewards reproducibility). Replaced juice-shop challenge count
on 2026-05-22 to stop incentivizing target-specific fingerprinting.
Juice Shop 20-min run on auto-reset container + XBEN 8-pack
(XBEN-005, 013, 019, 020, 036, 038, 053, 073) at 240s each.
Each sprint builds a tagged image (sploitagent-kg-s<N>)
that the validation suite runs against. Tags preserved
indefinitely so deltas compare real artifacts.
Comparator (scripts/bench-compare.sh) blocks sign-off if:
FQS drops > 10%, findings drop > 25%, XBEN flags drop
below prior count, juice-shop solved drops > 25%, or
any per-target benchmark loses its flag or drops > 50%
on findings. Per-target floor added in Sprint 2 after the
XBEN-005/019 payload-passing regression event.