Artificial Intelligence Amplifies Paleontology's Reproducibility Problem

What this was about

Brooke Long-Fox, a data curation scientist at MorphoBank and data editor for Historical Biology, argued that AI is entering a field that already has reproducibility problems of availability (data not deposited or on personal websites), documentation (no README, undefined characters) and traceability (unclear links from evidence to result). AI can turn ambiguous or incomplete sources into perfectly formatted, schema-valid tables that get DOIs and feed further AI, improving structure without improving scientific validity. Using morphological character matrices, she showed that a binary score can encode expert judgements about homology, preservation and ontogeny, and proposed adding verifiability to AI readiness: tracing assertions to evidence, recording whether they were transcribed, transformed or inferred, and who validated them.

Why it matters. It distinguishes technical validity from scientific validity, a warning relevant to any biodiversity data being mass-extracted or inferred by AI.

In the room

• This is not an argument against AI; she uses AI-assisted curation, but AI can amplify existing reproducibility problems.
• Paleontology is data-intensive; the digital extended specimen links a specimen to images, occurrences, stratigraphy, measurements, matrices and outputs, so errors propagate further.
• Availability problems: data never deposited, manuscripts claiming no data exist, files on personal websites or cloud links, private or incomplete repository records.
• Documentation problems: no README or data dictionary, unexplained abbreviations, 3D models without links to specimens, incomplete character definitions; a PID says where a file is, not what it means.
• Traceability problems: data statements not matching deposits, scores not linked to sources, matrices modified across studies without documentation, unclear versions.
• AI can produce schema-valid, DOI-bearing structured outputs from ambiguous sources (e.g. OCR column shifts, inconsistent translation), which then feed other AI systems.
• Technical validity is not scientific validity: a matrix cell such as character 37 = 0/1 may encode reasoning about homology, absence vs non-preservation, ontogeny, polymorphism or inapplicability.

Notable moments