About

Understanding how these systems behave once they are deployed

Loan decisions, clinical triage, identity verification, and benefit eligibility across Africa are increasingly produced with the help of models, and independent evidence about how those models behave in service is scarce.

We are a testing laboratory, so we do not build the systems we evaluate, we do not sell tooling to the organisations we evaluate, and the value of what we produce rests on our figures being reproducible by someone who has no reason to trust us.

Our work

Evaluation: how a named system performs on the populations it meets in service, run to a protocol registered beforehand.

Threat research: what a system is exposed to once it is reachable, from instruction injection to disclosure through outputs and logs.

Training: evaluation and safety capability built with the people who operate these systems.

The commitments that hold across our work

• Every completed evaluation is published, whatever it concludes.
• The protocol and the harm threshold are registered before the study is run.
• Every figure carries an interval, a denominator, and the version it was measured under.
• The access tier is stated on the front of every report.
• The subject is heard before publication, and its reply is published unedited.
• The notebook, the results file, the harness, and the test set are published wherever law and licensing allow.
• Corrections are published as prominently as the original.

These are set out in full on Independence.