Red-teaming science, standards, and specifications for assessors and regulators

TLDR

We believe in the need of an independent, non-profit organization turning the red-teaming of AI control infrastructure from a craft into a science. This includes advancing red-teaming methods and developing open specifications and standards for both assessors and regulators, especially in the eventuality that third-party auditing will become mandatory. Ideally, this organization also executes red-teaming campaigns of production control measures to ground and update methods.

Theory of Change

Problems

1. False assurance: As models capabilities grow, and especially during recursive self-improvement, assurance will increasingly depend on the robustness of control measures and the detection of incidents. If labs and evaluators build safety cases on control assessments that are subtly wrong or incomplete, their reports will state a level of safety that does not exist.
2. Auditing cannot scale and diversify: We have a very limited number of entities that can execute audits, in the case they become mandatory. However, so far, most red-teaming exercises did not produce detailed, repeatable practices. As a result, new assessors have nothing to learn the practice from, so if audits are expected by law, the choice will be between too few qualified auditors and box-checking by unqualified ones.
3. Policy lag: The lack of operationalizations in control-based safety cases plausibly impairs governance efforts, as policy makers cannot define when testing is adequate and what are red-lines. This can slow down regulations at all levels, from proposal to enforcement. Also, it indirectly leaves room to non-committal language in voluntary frameworks.
4. Poor epistemics: With no standards, control measures are badly compared across the industry or over time. As a result, stakeholders are left without a clear picture of which developers are doing best. Also, when a measurement shifts, it is harder to isolate the cause, as tests change over time. However, there are virtuous examples to build on top of, like Control Leaderboard (Redwood Research & Equistamp, 2026).

Gaps

Gaps represents what is missing from the ecosystem. Problems are downstream effects of gaps.

Uncertainties

1. We didn’t model deeply how the lack of standards might impair policy efforts, or how much it weights against other factors (i.e., political will). We believe this is a reasonable intuition and assumption, but we would like external input and feedback.
2. We believe that defining and refining proper standards needs hands-on red-teaming experiences and we would build the in-house expertise for that. However, it’s unclear to us if executing red-teaming campaigns without payment and setting evaluation standards represent a conflict of interest as the case of advocating for the necessity of third-party evaluations and being paid for conducting evaluations. Also, building labs access within short timelines seems hard.
3. The gaps in red-teaming science and evaluation standards seem very solid to us. To our knowledge, the gaps we identified represent open questions and it’s unclear when other, more established, organizations will address them. However, we think it’s likely that it will happen. Therefore, we question the need for a new organization, if not for the execution speed given by the lower operational overhead. If a separate organization is desirable, we would like feedback or help for setting up collaborations with existing organizations, as adoption of standards and specifications need endorsement from practitioners.
4. We wonder whether standard and methodology development can be amplified by building open tooling to automate evaluations that encode these standards and methods.
5. Generally, we would like feedback on this idea, its usefulness in reducing catastrophic risk, and if the problems and gaps identified hold.