How is digitisation doing? Monitoring the increase in digitised collections using MIDS

What this was about

Elspeth Haston reported a MIDS-based analysis of ten years (since 2016) of preserved-specimen records of biological taxa in GBIF, using full-GBIF Darwin Core downloads partitioned with DuckDB into Parquet and scored with a version of the MIDS calculator (2017 excluded for data problems). Progress was real but not steady: record numbers reaching MIDS 1 and 2 grew, yet the percentage stayed fairly flat as total records grew, with a big jump in MIDS 1 after 2016 driven by GBIF enforcing licences and more datasets providing modified dates. Newly arriving records vary a lot, some cohorts decline (one institution's datasets dropped organisation data for two years), enhancement tools have uneven impact, kingdoms differ in data completeness, and big datasets have big effects.

Why it matters. Provides the first longitudinal, standard-based view of how complete published specimen data are and what interventions (e.g. mandatory licences) actually move the needle.

In the room

• GBIF snapshots lacked full specimen metadata, so the team used Darwin Core archives from users who downloaded all of GBIF.
• Big files need big computing resources; one record had over 60,000 coordinates.
• The large MIDS 1 increase after 2016 came from licence enforcement and more modified dates; name and identification are high due to the GBIF backbone.
• People identifiers started appearing, showing the effect of dedicated enhancement work, though without raising overall MIDS levels.
• Recent years show fewer records arriving below MIDS 0 and more at MIDS 1, driven by organisation and modified fields.
• 2016 records fell from MIDS 0 to -1 by 2018 because five datasets from one institution dropped organisation data, corrected by 2019.
• Animalia more often record object type; Plantae lead on media and media IDs (possibly IIIF); collecting numbers more common in plants and fungi.

Notable moments