Independence

The principles that hold across our work

• The measurement settles the conclusion, and nothing else is admitted into that decision.
• What we claim is bounded by the access we were given, and the bound is stated wherever the claim appears.
• The instrument is published with the result.
• The organisation responsible for a system is heard before we publish, and the reader is given everything they would need in order to disagree with us.
• Corrections are published as prominently as the original.

How the laboratory sustains itself

Quantile Labs is a company, and this work is what sustains it, which puts a plain obligation on us to arrange that work so that it can never bear on a result.

• Scope, method, harm threshold, and terms are agreed before the study is run, and none of them moves with the result.
• Nothing we are owed is contingent on what an evaluation concludes.
• We hold no equity or other financial interest in the organisations whose systems we evaluate.
• We do not sell tooling or remediation to an organisation whose system we have evaluated.
• Training and evaluation of the same organisation are separated in time, and disclosed on the front of any finding.

The work we do not take on

• Work in which the subject would hold a right to prevent publication.
• Work in which what we are owed, or any other interest we hold, would turn on the outcome we report.
• Certification, accreditation, or any other output presented as a mark of approval.
• Evaluations scoped so narrowly that the result would mislead the reader about how the system performs.
• Work in which the test set cannot resolve the populations the harm model identifies.
• Red-teaming commissioned in order to support a marketing claim.