Closing the Loop: The DiSSCo Annotation Validation Framework for Trustworthy Data Round-Tripping

What this was about

Wouter Addink, DiSSCo technical director, framed the problem of a curator swamped by thousands of AI annotations that could override records and lose provenance. DiSSCo's planned annotation validation framework runs machine annotation services centrally for all collections and places annotations in layers: an untouched institutional base layer, a community enrichment layer for unvetted annotations, and a curated consensus layer. Annotations pass automated technical validation, then are routed by type and impact: low-impact corrections are auto-promoted, medium-impact ones need a two-agent condition or a trusted agent, and high-impact ones (new identifications, type status, catalogue numbers) need managed expert review. A public consultation was extended to 16 October with an online meeting on 22 October.

Why it matters. Addresses how to absorb large volumes of human and AI annotations without losing institutional control or provenance, a key prerequisite for AI-assisted curation at scale.

In the room

• Scenario: a museum builds an AI annotation service that produces thousands or millions of annotations; directly writing to the CMS loses sovereignty and provenance.
• Machine annotation services run in the DiSSCo infrastructure so they can be applied to all participating collections; anyone with ORCID can annotate.
• Layers of trust: institutional base layer (untouched), community enrichment layer (unvetted), managed review, and curated consensus layer holding the latest digital specimen version.
• Validation flow: submission with persistent identifier and agent ID; automated technical validation (schema, content policies, e.g. no malicious content or exact locations of protected species); automated routing; promotion creating a new version.
• All versions are stored, making the system resilient to errors in automated promotion.
• Comments and quality flags go to the enrichment layer; delete proposals need administrator review.
• Edits are assessed for impact: low (error corrections, measurements) auto-promoted; medium needs two agents agreeing or a trusted agent with a good track record; high (new identification, type status, catalogue number) needs expert review, where experts may also be trusted machines.

Notable moments