Multimodality for Knowledge Extraction from Historical Entomology Literature

What this was about

Jana Hoffmann and Sefika Efeoglu of Senckenberg's new Semantic Data Systems and Knowledge Engineering group presented a prototype linking text and illustrations in historical entomological literature, using Maria Sibylla Merian's 1679 'Der Raupen wunderbare Verwandlung' (50 plates, digitised and published on Zenodo, soon in BHL). The workflow uses olmOCR for text, an ontology of concepts, detection of visual entities, relations and attributes in illustrations, annotation of text entities using those visual entities, merging into a multimodal sub-knowledge graph, and verbalisation into evidence-grounded captions, since historical plates often lack captions. Next steps are scaling to all 50 articles, provenance, and expert human-in-the-loop validation.

Why it matters. Illustrations carry complementary morphological, life-stage and interaction evidence that text-only pipelines ignore.

In the room

• Text describes organisms, host plants and feeding interactions; illustrations show morphology, developmental stages and spatial relations.
• Merian volume from the Senckenberg German Entomological Institute library in Müncheberg.
• Generated captions are restricted to what is supported by the graph; freer generation adds hallucinated content.
• Aim to provide a reusable workflow for other historical sources and downstream BHL uses.

Notable moments