IWAC AI Pipelines

With nearly 21,000 items and more than 32 million words, the equivalent of over 300 books, the Islam West Africa Collection has long outgrown what one researcher could read, transcribe and catalogue by hand. Without help, much of it would remain “digital clutter”: gathered, but never truly accessible.

These pipelines put large language models to work on the tasks that would otherwise take months, or never get done. They draw on commercial models from Google, OpenAI and Mistral, and on open-weight models such as Qwen, DeepSeek and Gemma, which can also run on a university’s own servers. The aim is to make a large collection workable, so that researchers can spend their time on interpretation rather than data wrangling. The code is open source and modular, with its prompts kept as plain text files, so other collections can adapt it.

What the pipelines do

• Read scans: extract the text of scanned newspapers and magazines, and correct the errors left by older OCR software
• Read handwriting: transcribe manuscripts in French, Arabic or both
• Listen: transcribe interviews, oral histories and YouTube videos, including in Hausa, and describe what videos show
• Index: find the people, places and organisations in each document and link them to the collection’s index despite variant spellings, and give academic references their subject keywords
• Summarise: write short French and English summaries that make documents easier to find
• Split magazines into their individual articles
• Read press attitudes: rate how each article portrays Islam and Muslims, with a panel of five models

Careful by design

The pipelines write to the live archive, so each one is built to be cautious. It can first report what it would change, saves a copy of every record before editing it, and asks for confirmation before a bulk update. Every AI-generated text and link also records the model that produced it, using a small vocabulary built for the purpose, so readers can always tell machine work from human cataloguing.

A companion web app, AI-NER-Validator, keeps human judgement at the centre of entity extraction. Researchers review each article with the extracted names highlighted, accept or reject each one, add what the model missed and export a clean list.

Limitations

These tools are research aids, not substitutes for scholarly judgement. The models are black boxes, and the same text run twice may not give the same answer. Trained mostly on Western data, they can misread African contexts and names. Their mistakes are also harder to spot: where old OCR produced visible gibberish, an AI error reads as fluent prose. Using them well takes expertise and close knowledge of the sources.