Digital resources in the Social Sciences and Humanities OpenEdition Our platforms OpenEdition Books OpenEdition Journals Hypotheses Calenda Libraries OpenEdition Freemium Follow us

Will Digital Destroy the Humanities?

The short answer is no, but arriving at that answer requires a bit of a journey, and there are quite a few caveats along the way. In the long run, the advent of quantitative methodologies in traditional humanistic disciplines will forever and irreversibly change scholarship, but whether this will be a net loss or gain depends on digital and traditional humanists working together with common values, a shared understanding of the risks, and a willingness to recognize one another’s distinct but interconnected contribution to scholarship.

Celebrating New Technology

This essay was inspired by my experience teaching Measuring Manuscripts, a computational philology course at the European Summer University in Digital Humanities in Besançon, France in July of 2026. In that environment, composed of present and future digital humanists, the mood was exuberant. Finally among allies, we could celebrate our favorite questions and indulge in our favorite methods. Many there shared the experience of moving in more traditional academic circles, where the overall vibe is that we “computer people” have come to spoil the party. At ESU, that feeling was conspicuously absent. In a wide-ranging talk on her experience as a digital humanist beginning in the 1990s, Elisabeth Burr, the founder of the ESU, referred to the digital humanities as a “new epistemology”, a thought-provoking statement, which reflected a shared feeling. We were doing something new and exciting that promises to change our fields for the better. But in our exuberance, we could inadvertently underestimate the risks.

New technologies often replace what came before. In fact, they are routinely designed to do exactly this. Horseless carriages and cordless phones were both (once) named for what they were supposed to eliminate: horses and cords. The iPhone was designed to replace cell phones, iPods, and eventually digital cameras in a single package. In some cases, bad actors are not content to let their ideas win in the marketplace, but instead aim to take by force, as exemplified in the famous antitrust case against General Motors, Firestone, Standard Oil of California, and Phillips Petroleum, which alleged that they had deliberately stifled competition from other modes of transportation. (They were acquitted, but everyone knows what really happened.) Or they impose their views on the public through a claim to authority, without even needing direct control over the process. German architects in the early twentieth century, frothing with modernist zeal, actively tried to reeducate the public out of their love for Gründerzeit ornament by popularizing Entstuckung—literally tearing the facades off of existing buildings—resulting in an antihuman urban landscape scarred by dystopian wastelands of shabby boxes. This is not to say that innovations in transport or architecture are bad things, but destroying what already exists in a manic pursuit of novelty impoverishes and uglifies the whole world to the harm of those who never had any say in the decision. History justifies our worry.

But the worry can easily be misplaced. Such outcomes are not inevitable, and treating them as a given can distract from the real danger. Video did not kill the radio star. Streaming (arguably an updated version of radio) did. Fledgling artists struggle to survive on the royalties from streaming services like Spotify. While MTV as a source for music was only a short-lived fad, audio-only formats were destined to change the industry to one in which only those of independent means can hope to survive, where the working-class hero is no longer something to be. Worrying about the wrong problem is just as dangerous as not worrying at all. The crucial question is: where do the real risks lie?

A Dubious Denial

When speaking publicly on this topic, digital humanists often say that their methods are not meant to replace traditional ones. Perhaps they mean this sincerely, but the claim is ultimately disingenuous. Nearly every computational method I taught in Besançon is meant to replicate, and therefore possibly replace, an existing non-digital one. One of the motivations for creating new methods is to improve upon old ones, and often the only way to sell a new method to the public is to demonstrate its improvement on an existing task. If that improvement is successful, the new method will inevitably supplant the old, because its advantages will outweigh its disadvantages, making it on the whole better. Better might mean more efficient, more reliable, easier to scale, easier to evaluate, or any number of other things, but better is the explicit goal. Importantly, better does not mean free of negative consequences, many of them unintended and unexpected. It means more likely to be selected as the standard approach. The difference is crucial.

Source: Martin Howard via Wikimedia

Typewriters

Here it might be useful to step outside of our immediate surroundings and lean on the power of metaphor. Imagine a PC salesman in the early 1990s who claims that PCs are not intended to replace typewriters. Such a claim would naturally be met with skepticism. Isn’t desktop publishing one of the promises of home computers? How then could they not replace an existing technology that performs the same task less effectively? Of course PCs replaced typewriters, and for many good reasons, but with some tradeoffs. We gained speed and capability: non-sequential editing, free choice of layouts, fonts, and colors, the ability to store and transmit documents digitally. We got to dispense with jams and correcting fluid and bulky machines. We lost the tactile experience of a mechanical keyboard and the pleasant thump of the typebar on paper and the satisfying ping of a real physical carriage return. Almost everyone took the deal. (Including me. This essay was written on a computer.) They weren’t forced to. Anyone can write on a typewriter today if they want, but most people make the same choice. And that was the point. PCs were deliberately designed to replace typewriters, and claiming otherwise would have been dishonest.

Now imagine that same salesman claiming that PCs would replace parks. “After all,” he might argue, “you can visit a virtual park right there on the screen without even needing to put your pants on.” Such a claim smacks of tech-bro hubris, and it would rightfully be met with outright derision. A simulation of a park on a computer screen can’t do what a park does. No reasonable person would claim otherwise. But this example highlights the danger of unintended consequences. What if people start spending all their time at home staring at computer screens? What if cities, seeing a decline in visitors, decide to save money by reducing funding to the parks department? Will competition from a lower-quality substitute diminish the real thing? This is a real danger, not replacement but displacement, about which I will say more later in this essay.

But first, I must state frankly that the humanities are full of typewriters: established methods that can be improved upon, and where the improvement will (at least partially, perhaps completely) replace the old way of doing things. One obvious example is palaeographic dating. Traditionally, palaeographic dating depends purely on expertise and scholarly authority. Though often reliable, the underlying process is a black box. Disagreeing with a date given in an old source simply creates two differing opinions. By contrast, computational approaches to palaeography are replicable—other scholars can rerun the analyses while checking their underlying operations and conclusions. They scale to hundreds of texts and they provide precise measures of their own certainty. The advantage to traditional humanists is undeniable. A scholar studying culture across time benefits from working with an entire corpus of reliably dated texts and doesn’t need to become an expert in palaeography in order to obtain one. Traditional palaeographic methods don’t offer these advantages. Digital text editions are another example. Having hundreds of witnesses aligned in a customizable synoptic view (as in the Kalila wa-Dimna project) fundamentally changes the capability of the philologist. Digital dictionaries are yet another. For Egyptologists, the Thesaurus Linguae Aegyptiae has replaced bulky, incomplete physical dictionaries. Plus, it’s available in English, and it’s free.

Typewriters are not the problem. They don’t replace expertise or humanistic inquiry. In fact, in many cases scientific methods highlight the ongoing need for expertise. A useful case study is found in the Gospel of Jesus’ Wife scandal. In 2012, Karen King, a professor at Harvard Divinity School, announced the discovery of a papyrus fragment with the words: ⲡⲉϫⲉ ⲓ̅ⲥ̅ ⲛⲁⲩ ⲧⲁϩⲓⲙⲉ… “Jesus said to them: ‘my wife…’”. The Coptic is unambiguous, and the news coverage exploded. Numerous Coptologists responded immediately with doubts. Leo Depuydt analyzed the text, identified its resemblance to the Gospel of Thomas, and argued that the likelihood of such similarity occurring by chance is astronomically small. Christian Askeland, comparing another text in the same hand, found that document to be a copy of a modern text edition. The argument raged between the experts, but the truth eventually came out: it’s a forgery. (The full story is covered in Ariel Sabar’s Veritas: A Harvard Professor, a Con Man, and the Gospel of Jesus’s Wife, which is a genuine page-turner.) Science seemed to authenticate the fragment, dating the papyrus to the eighth century CE and validating the ink as plausibly genuine. In this case, and in so many others, the accumulated knowledge of experts was needed to evaluate the seemingly straightforward results of scientific tests. The science didn’t fail. It correctly produced the result it was intended to. But such results, without qualified interpretation, are partial at best. The lesson is clear: quantitative results cannot stand on their own, they must be evaluated, discussed, and understood by scholars.

Parks

PCs can replace typewriters, they can’t replace human writers. They also cannot replace parks. And the humanities are full of parks as well. In a recent conversation with a colleague about this topic, I pointed out that quantitative methods can sometimes provide objective answers with a measurable degree of certainty. She countered that, while this may be true, she isn’t interested in such questions. She prefers questions that don’t have simple, straightforward answers. And rightly so. Some of the most interesting questions cannot be answered definitively, with quantitative tools or any other, and one of the most essential roles of the humanist is to engage with such questions.

There is a contrasting claim, sometimes made by those with an inclination towards scientism, that “if it can’t be measured, it doesn’t exist.” This is nonsense. Plenty of things can’t be measured. Some of the most important aspects of the experience of being human can’t be measured. No amount of MRIs studying love in the brain can ever capture the bitter poignance of Damon’s “Nunc scio quid sit amor” or the quiet yearning in Wesley’s “As you wish.” Quantitative methods will never address the core questions that drive the humanities. They can only provide useful background information that informs those questions. In some sense, they might even be said to help us identify the big questions by reliably excluding those that are easily answerable and therefore less interesting. None of this is to say that measurement isn’t important. In fact, I would emend the absurd: “if it can’t be measured, it doesn’t exist” with the more reasonable (but surely still controversial): “if it can be measured, it should be.” Measurement, and by extension quantitative methods more broadly, provide a robust foundation that undergirds our reasoning. As we extend that foundation, the number of interesting questions grows as well.

Philology is full of such questions. Why does Sinuhe flee when the pharaoh was assassinated? The text never tells us, as its unknown author presumably intended. (In fact, it never even explicitly calls it an assassination. We’ve had to infer that from other sources.) We are supposed to wonder and discuss, as scholars have done for more than a century, just as ancient Egyptian audiences surely did nearly four millennia ago. Computational methods may tell us when the text was written, but they cannot tell us why. These are the parks. A computer cannot replace them. Nothing can.

A Troubling Pattern

And yet, there is a historical pattern in the sciences worth remembering. Humanistic inquiry tends to comprise what is left over after quantitative approaches have done all they can. This relegates the humanities to the remainders, a seemingly shrinking space of scholarship surrounded on all sides by math and science. Many fields began as natural philosophy and became sciences when numbers arrived. Geology is a well-known example. After plate tectonics and radiometric dating, geology became a quantitative discipline, and previous lines of inquiry gradually faded away. Biology was once the purview of naturalists crouching in bushes drawing birds and beetles in notebooks, now the vast majority of biology papers include statistical analyses and would never be published without them. Scholarly disciplines can be and frequently have been taken over by quantitative methodologies. Oversimplifying a bit, the sciences are everything you can do with numbers, and the humanities are everything you can’t. From such examples we might conclude that, as soon as a field gains numeracy, it switches sides.

The field I work in, Egyptology, is fundamentally humanistic. It deals primarily with an ancient culture and its relationship to the human experience. Many of its best questions cannot be quantified. The dilemma is whether those questions will protect the field from crossing the boundary into the realm of science or be swept aside by a tsunami of quantitative methodologies. As a digital humanist, I and my like-minded colleagues have a responsibility to oppose this outcome at every step. Just as parks might suffer neglect when everyone stays home looking at their computers, so too might humanistic questions be relegated to obscurity if my own approach to knowledge creation is carried to a harmful extreme.

But as we all know, funding for scholarship is limited and awarded by competition. As funders promote digital methods, less is left for others. The economics are straightforward. In many ways the shift has already begun. The solution is simple (though not necessarily easy). We, the digital humanists, must insist that our work always support existing humanistic questions or lead to new ones. Presentations of tools, datasets, or methods must continue to be seen as only partial scholarly outputs. Computation for its own sake must never be allowed to dominate the field. Digital and non-digital methods must always be integrated into shared, overlapping modes of inquiry.

There is reason for optimism. Archaeology, fundamentally interested in human societies in the past, has adopted numerous quantitative and scientific tools without abandoning its humanistic core. As a field adjacent to philology and embedded in umbrella fields like Egyptology and classics, it is likely that a similar scenario will unfold naturally. Perhaps what makes a field humanistic is not its methods but its focus on humans.

Source: Joi Ito via Wikimedia

Video Games

Even then, there will be some loss in market share for the traditional humanities. That is, the things I’ve chosen to label typewriters will be replaced. In the Venn diagram of digital and non-digital methods, there is a lemon of intersection occupied by methods that will become digital. Ideally, these things will spawn new humanistic questions, thus taking over some territory but giving back more in exchange. Some questions will remain humanistic, so long as we can manage it—those things that computational methods cannot simulate. The parks must be protected.

But there is a missing piece of this Venn diagram. What of the things computers can do that traditional methods cannot? Abusing the earlier metaphor, we might think of these as video games. These are not simulated parks, which wouldn’t be any fun anyway, because real parks are unbeatable. These are entirely new spaces, which can only exist thanks to computers. In the scholarly realm, no equivalent of video games seems to exist as a category of research. Up till now, digital methods have been focused largely on replicating existing methods. But surely, if there are questions only non-quantitative methods can address, then there are also questions only quantitative methods can address.

There are a few early forerunners worth considering, as they hint at the possibilities that await us. Agent-based modeling, already in use in economics, offers new ways of understanding human behavior in complex interactions. A similar approach has been used to offer new perspectives on complex, otherwise invisible phenomena in prehistoric societies. And it would be negligent to omit the remarkable results from the Vesuvius Challenge, where an ancient papyrus scroll, charred into a lump of coal by the eruption, has been virtually unwrapped and read for the first time in millennia.

At present, the examples of video games are few. What sorts of new and exciting possibilities lie waiting to be discovered? Without much in the way of precedent to start from, how can we even discover such things? This is precisely the sort of question digital humanists need to start thinking about. Until we add something new, we appear only to take away. We need a new approach to finding ideas that goes beyond the recreation of existing methods in computational form.

Will Digital Destroy the Humanities?

My answer at the beginning of this essay was no, but with quite a bit of nuance and caution. In particular, nothing guarantees this outcome. It depends on us. To some extent and in specific cases, new methods can and should replace old ones, and we must address such situations honestly. But replacement in individual cases is not the real danger. Displacement is. The typewriters should go the way of the dodo while leaving the parks unaffected. And the role of expertise should never be handed over to a machine. Digital tools and quantitative measures cannot be relied upon without interpretation. Further, the questions they ask and answer do not cover the full extent of humanistic inquiry. We must not forget this.

We must not become like those greedy robber barons and narcissistic architects who impose their will on others by destroying what has come before. The burden is on the digital humanists to direct their research toward humanistic questions, either by supporting them with new tools that accelerate the work and strengthen the underlying foundation, or by devising new ways of looking that lead to new questions. In both cases, the quantitative results are always partial. Scholarship lies in intelligent interpretation. The traditional humanists will need to trust that we are all on the same team, and that trust must be built through honesty about what we’re here to do and a commitment to keep asking humanistic questions.

Data Survives, Projects Die: Rethinking Sustainability in Digital Humanities

Over the past decade, sustainability has become a central concern across the digital humanities (DH). This debate frequently gravitates toward the question of data formats, metadata schemas, repository infrastructures, and long-term archiving practices. Within the German academic context, the most authoritative guidelines on these matters stem from the Deutsche Forschungsgemeinschaft (DFG). Two documents in particular—the 2019 DFG Handreichung on Standards for the Creation of Linguistic Corpora and the 2021 DFG Checklist for Handling Research Data—have become touchstones for projects seeking technical clarity and compliance with established expectations of research data management.

Yet as comprehensive as these guidelines are in the domain they address, they remain fundamentally limited in scope. Their primary orientation is the data object: its technical quality, its annotation, its interoperability, its metadata description, and its long-term archival stability. What these guidelines do not address—indeed what they do not even acknowledge—is the complex ecosystem in which digital humanities projects actually operate: the people, infrastructures, governance mechanisms, software repositories, linguistic challenges, and socio-technical dependencies that ultimately determine whether a DH project survives beyond its funding cycle, especially DH projects that concentrate on non-Latin scripts. 

The DFG Perspective: High Standards, Narrow Scope

The 2019 DFG Handreichung is a deeply technical document, and rightly so: its objective is to standardize the creation of linguistic corpora. It offers detailed recommendations for recording audio and video data, including the use of uncompressed WAV PCM audio, high-bitrate MPEG or MJPEG2000 video, and high-quality external microphones. It instructs projects to avoid proprietary formats and to maintain archival “master copies” of all recordings. It also insists on structured transcription formats, aligned annotation layers, ISO 24624 compliance, and the use of interoperable XML-based tools such as ELAN, EXMARaLDA, ANVIL, or Praat. For written corpora, it prescribes digitization standards (TIFF or JPEG2000 at a minimum of 300 dpi), careful choice of OCR workflows, documentation of error rates, and the use of TEI P5—with ODD customizations where necessary—for structural encoding. It also outlines the importance of metadata compliance with Dublin Core, OLAC, CMDI, and similar vocabularies, as well as the need for consistent archiving in certified repositories such as CLARIN-D or DARIAH-affiliated centers.

The Checklist published by the DFG in 2021 expands the conversation beyond linguistic corpora to research data management more generally. It describes six categories that applicants must consider: the nature and volume of their data, its documentation and quality assurance mechanisms, storage and archiving during the project, legal requirements, long-term accessibility, and the responsibilities and resources needed to manage data appropriately. The checklist also asks researchers to identify who will curate the data after the project ends, through which infrastructures, and under what access conditions. Its emphasis is on ensuring technical, legal, and procedural clarity so that data can remain FAIR (findable, accessible, interoperable, reusable).

Taken together, these two documents articulate a rigorous and technically sophisticated understanding of data sustainability. They excel at defining how to produce, encode, document, and preserve data according to community standards; how to ensure its future interpretability; how to safeguard privacy and copyright concerns; and how to prepare datasets for long-term deposit in recognized archival institutions.

Yet this is precisely where their limits appear: the DFG documents address the sustainability of data, not the sustainability of projects.

What the DFG Leaves Unsaid: People, Infrastructure, Software, and Scripts

Digital humanities projects do not fail because of improperly formatted WAV files, insufficient sampling rates, or deviations from TEI P5. They fail because the person who created the custom Python scripts leaves the team; because the only server administrator moves to another institution; because the GitHub repository has no documentation; because the project website expires when the hosting contract lapses; because a hand-coded interface breaks after a WordPress or PHP update; or because non-Latin script rendering becomes incompatible with newer browser engines. None of these risks appear in the DFG guidelines, though all of them are structurally common in DH work.

The DFG checklist speaks of “responsibilities” and “resources,” but only in the narrow context of data management. It does not address the reality that DH projects require sustained team capacity, leadership continuity, onboarding practices, long-term staff training, and governance mechanisms that survive personnel turnover. It does not engage with the fact that many DH projects rely on fragile chains of expertise involving student assistants, short-term researchers, or single individuals with highly specialized knowledge. Nor does it articulate models for multi-institutional collaboration, knowledge transfer, or communication across linguistically diverse research environments.

Furthermore, the DFG guidelines assume a broadly Latin-script-centric technological infrastructure. They provide no discussion of Unicode normalization challenges for non-Latin scripts, no guidance on right-to-left (RTL) interface design, no analysis of segmentation or tokenization problems in Arabic or Persian, no comments on the limitations of OCR engines for Arabic-script manuscripts or print culture, and no strategies for dealing with under-resourced NLP ecosystems. Data fragilities associated with Arabic shaping, diacritics, bidirectionality, or grapheme-cluster segmentation simply fall outside the DFG’s scope, even though these issues fundamentally determine the usability and reproducibility of non-Latin script data.

Likewise, the DFG documents say nothing about the maintenance of public-facing web interfaces, even though such interfaces constitute the primary access point for most digital humanities audiences. They do not discuss hosting strategies, domain expiration, HTTPS/SSL dependencies, front-end frameworks, responsive design practices, multilingual navigation, accessibility guidelines, or versioning of code bases. The Web, in the DFG’s perspective, appears only indirectly—as a potential source of “born-digital” texts or as a means of accessing archived corpora—not as a living, evolving surface that requires continuous care.

Software sustainability is another area absent from the guidelines. The DFG documents classify software simply as tools for generating or analyzing data. They do not treat software as a research output that requires its own maintenance strategy, contributor model, or version governance. They do not consider issues such as dependency rot, maintenance burdens, or the need for inclusive open-source practices. Nor do they address the fact that DH projects often rely on customized pipelines, scripts, or APIs whose survival depends on well-documented code, clear licensing, and active maintainers.

Finally, the DFG does not frame sustainability in terms of community or audience engagement. It offers no guidance on how projects should cultivate user communities, maintain multilingual documentation, support culturally diverse stakeholders, or design public interfaces that serve both academic and non-academic visitors. In the context of non-Latin scripts—where communities of heritage, diaspora, or local institutions often play central roles—this absence is especially notable.

The CtG Approach: A Holistic View of Sustainable Digital Humanities

The CtG project takes a fundamentally different starting point: sustainability is not merely a matter of data longevity but of ecosystem longevity. Based on a study of more than 240 projects working with non-Latin script data, the CtG analysis integrates technical, social, organizational, and infrastructural dimensions. Rather than assuming that valid formats, standard-compliant metadata, and archiving plans guarantee survival, CtG examines the fragile places where DH projects often collapse.

One component of CtG’s sustainability analysis addresses team structures and governance models. CtG identifies how leadership styles, communication habits, institutional dependencies, and levels of technical expertise shape the life cycle of a DH project. Issues such as turnover, the departure of key individuals, and mismatches between project scope and staffing capacity emerge as crucial sustainability threats—none of which appear in the DFG materials.

CtG also attends to multilingual and multi-script user interfaces. It evaluates how projects handle Arabic script display, right-to-left navigation, font rendering, translation practices, and the creation of bilingual platforms. These considerations are essential for the accessibility and cultural relevance of non-Latin script DH work, yet they are absent in both DFG documents.

Finally, CtG adopts a broad view of sustainability by integrating publication practices, community engagement, and long-term hosting strategies. The analysis examines how open-access publications are linked to project data, how metadata circulates through discovery systems, how interface design affects usability, and how projects communicate with audiences beyond academia. It identifies where data are stored, how interfaces remain online, and which infrastructures (GitHub Pages, institutional servers, external repositories) determine the fate of a digital project after funding ends.

In essence, while the DFG provides rules for the sustainability of data, CtG provides a model for the sustainability of digital humanities projects as living sociotechnical systems.

Why the DFG and CtG Approaches Must Be Seen Together

It would be a mistake to view the CtG framework as a replacement for the DFG’s technical standards. Rather, the CtG perspective reveals that technical standards—while necessary—are not sufficient. The DFG documents excel at defining how research data should be produced, documented, and archived under high-quality conditions. They ensure that data remain interpretable, reusable, and legally compliant. But they do not answer the question that DH practitioners must confront with increasing urgency: Who will keep the project alive, the website running, the software updated, and the community engaged after the funding ends?

The sustainability of digital humanities in practice depends on the convergence of both dimensions. The DFG’s technical standards ensure that data are robust, but CtG’s ecosystem perspective ensures that the entire project remains functional, accessible, and culturally meaningful. Only by combining these approaches can we build genuinely sustainable DH infrastructures—especially in fields dealing with non-Latin script data, multilingual materials, and global research communities where technological, organizational, and cultural challenges intertwine.

Generative AI and Low-Resource Languages: The Quest for Linguistic Equity

As an expat working in an international environment, I’m used to conducting nearly all my daily and professional tasks in a language that isn’t my own. As a linguist, I find that both fascinating and rewarding, yet it feels unsettling how this one particular language—English—has come to dominate almost every part of my life, distancing me from other languages and cultures that shaped me or that I’ve grown to love. This very blog post, the sources I consulted and questions I googled or asked ChatGPT in order to write it, but also my private searches for recipes or workout videos all happen in English. Most of the content I consume online is in English, despite not living in an English-speaking country. This quiet dominance reflects something larger than personal habit: the global linguistic hegemony of English, which mirrors deep social, political, and economic inequities. It privileges those of us fluent in it while excluding countless others—and even entire communities—from access to knowledge, participation in technology, and influence in the conversations that define our shared future. With the rapid spread of generative AI into nearly every aspect of daily life, this imbalance becomes even more amplified.

Even though there are over 7,000 languages spoken worldwide, the internet is primarily written in English and a small group of other mostly European or Asian languages, such as Spanish and Mandarin. Since generative AI tools are mostly trained on internet data—websites, books, Wikipedia, news articles, forums, code repositories, and social media—access to these tools may be limited for individuals or communities who are not fluent in these languages. To make matters worse, this limited access leads speakers of under-resourced languages to have a smaller digital footprint, making it less likely that their languages are included in web-scraped training data, resulting in a literal downward spiral: without sufficient data to train usable language-based systems, most of the world’s AI applications will under-represent billions of people, further deepening existing economic and political inequities.

Of all the languages spoken worldwide, only about twenty are considered “high-resource.” The rest—so, practically most of the world’s languages—are “low-resource.” This term refers not only to the limited amount of textual data available to train language-based systems effectively, but also to the lack of computational infrastructure needed to model and process these languages: things like keyboards, Unicode support, or even basic digital tools. It also includes the absence of researchers with relevant expertise and the lack of financial or political support for institutions researching and modeling these languages. Together, these factors pose a major challenge for building models that work well with low-resource languages and often lead to their significantly poorer performance.

Large-scale evaluations of generative AI reveal a striking imbalance in how well these systems handle the world’s languages. In the MEGA benchmark (Ahuja et al., 2023), which tested GPT-3.5, GPT-4, and other multilingual models across seventy languages, English and a few other high-resource languages consistently outperformed the rest by a wide margin. GPT-4, for instance, achieved over 96 percent accuracy on an English reasoning task but dropped to around 77 percent for Burmese, while similar disparities appeared for Tamil, Haitian Creole, and other low-resource or non-Latin-script languages. Even when prompts and translation strategies were adjusted, the gap persisted—showing how the dominance of English in training data continues to shape the very boundaries of what AI can understand. This inequity becomes even more pronounced within a single language family. In the SADID evaluation dataset (Abid, 2020), translation models handled Modern Standard Arabic—the formal, well-documented variety—far better than the spoken dialects of Egyptian or Levantine Arabic, where translation quality fell by more than half. These results expose a deeper layer of inequality: not only do low-resource languages struggle to be recognized by AI, but even within a shared linguistic tradition, the everyday voices of millions are systematically left out.

The results of this inequity, which is also known as digital language divide, go far deeper than making it harder for speakers of Indigenous or low-resource languages to navigate Internet as chatbots, translation devices, and voice assistants become a crucial way to do so. It has a direct impact on the quality, reliability, safety and even pricing of the generated content. Since it is very true that LLMs learn their values from their training data, this data should be carefully selected, filtered and curated to make a chatbot helpful, human-sounding, not-racist or non-sexist. Unfortunately, the texts readily available in low-resource languages are often of poor quality, badly translated or of limited use. For years, the main sources of text for many such low-resource languages in Africa were translations of the Bible or missionary websites, such as those from Jehovah’s Witnesses. While these texts are historically and linguistically of a great value, they are not the most useful base for an application that should help you to tutor your child, draft work memos, summarize books, conduct research, manage a calendar, book a vacation, fill out tax forms, surf the web, and so on.

Another issue lies in a vicious circle of models trained on incorrect data producing content of a questionable quality that is than used to train them further resulting in worse, unreliable or even unintelligible content. Since most websites make money through advertisements and subscriptions, which rely on attracting clicks and attention, an enormous portion of the web consists of content with limited literary or informational merit—an endless ocean of junk that exists only because it might be clicked on. For a wider outreach motivated by profit purposes this (poor) content is very often (poorly) machine translated to multiple languages by freely available AI programs of questionable accuracy. This same content is then scraped by AI developers to train their models further. Since there is still lot of high-quality data available for high-resource languages -especially for English given that half of internet websites are written in it – this problem is not that accentuated as in the case of low-resource languages, where data is generally scarce.  

Another issue is that the pricing of models—and even their usage limitations—are based on linguistic features of English. While ChatGPT is free, many other LLMs charge users according to the number of tokens. For morphologically rich languages such as Armenian or Burmese, which require more tokens than English to express the same meaning, the price for text generation will be much higher. The same applies to limitations on the length of prompts or responses. Due to prompt limits, some tasks that require more elaborate instructions might be impossible in languages like Malayalam, whose token usage is 15.69 times higher than that of English. GPT-3, for instance, can return only up to 4,000 tokens for the combined prompt and response. That token count might correspond to a short tweet in one language but a medium-sized blog post in another.

Recent research also shows that large language models are not only less capable in low-resource languages — they are also less safe. Many studies (Yong et al., 2023; Deng et al., 2023; Shen et al., 2024) reveal that models like GPT-4 are three times more likely to produce harmful or unsafe content when prompted in low-resource languages than in English or other high-resource ones. The reason lies in the unequal depth of training and alignment: safety fine-tuning and moderation data exist mostly for English, so the guardrails that prevent toxic or biased outputs in one language simply fail to generalize to others. In addition, moderation systems themselves are often designed for English-like morphology and miss toxic phrasing in languages with richer or more complex structures. The result is a troubling paradox — the very languages most underrepresented in AI are also the ones most exposed to harm when they are finally included.

When AI fails to function well in under-resourced tongues, the consequences go far beyond convenience and safety—they strike at identity. Many speakers of low-visibility languages already live in a digital invisibility, and younger generations may see no reason to learn a language that no app, chatbot, or search engine understands. For languages with rich literary and artistic traditions—like Arabic, Persian, or many Indigenous and African languages—this effect is magnified: if people turn online for music, poetry, storytelling, or cultural content, and those forms aren’t supported in their language, the culture itself becomes harder to access and perpetuate. Languages that seem invisible to AI risk being abandoned—and with them, the songs, poems, and narratives that give them life.

The onus is on AI developers to listen to the actual needs of speakers, not just to generalize “big-tech” solutions for everyone. True linguistic inclusion requires more than adding translation features or tokenizing more languages—it means engaging with the communities who speak them. Speakers know best how their languages live and change, how they are taught, sung, and used in everyday life. Without their participation, even well-intentioned AI efforts risk reproducing the same hierarchies they claim to overcome. Building models that genuinely support linguistic diversity means designing with context, not just for convenience—acknowledging that a language is not only a system of words but a vessel of culture, memory, and identity. Only by centering real voices, not abstract datasets, can AI become a tool that strengthens rather than erases the world’s linguistic richness.

People Behind the Interface: Sustainability as a Social Process in DH


People Sustain Projects—But Who Sustains the People?

In Digital Humanities (DH), sustainability is often discussed in terms of infrastructure: servers, standards, and repositories. But as I began to explore what sustains—or quietly erodes—a DH project over time, it became increasingly clear that infrastructure alone is not enough. Projects are not only built on servers, schemas, and software, but also on people: their communication, their turnover, their sense of purpose, and their evolving relationships with one another and with the work itself.

Recent scholarship, such as Claire Battershill’s The Stories We Tell, invites us to see DH projects as narrative and emotional spaces—held together not just by code and metadata, but by human intention and care. Taking inspiration from this perspective, I was eager to explore sustainability not as a technical checklist, but as a lived experience within a team.

To ground this exploration, I turned to a long-running DH project that combines rigorous philology with innovative digital practice. I spoke with three members of the team—each at a different career stage—to hear how they navigate continuity, change, and the question of what remains once the funding ends.

The following sections share insights from these three vantage points—not as isolated anecdotes, but as interdependent reflections on what it takes to sustain a project not only technically, but relationally.

Insights from the Principal Investigator: Leading for Longevity

The interview with the Principal Investigator (PI) illuminated how the sustainability of a Digital Humanities (DH) project is shaped not only by technical foresight, but also by leadership choices, institutional negotiation, and the evolving social fabric of a research team. From the outset, the project was grounded in a critical response to the underrepresentation of non-Western texts—particularly Arabic wisdom literature—within the prevailing frameworks of world literature. What began as a curricular intervention gradually developed into a long-term DH initiative, made possible by the collaborative funding structures of the German academic landscape.

While the PI brought a strong scholarly vision to the project, the transition into team leadership was initially marked by uncertainty. “I was scared,” they admitted. “I had never led a team before. I was learning on the job.” Over time, however, a leadership model emerged that prioritized intellectual trust and distributed agency. By deliberately avoiding micromanagement, the PI fostered an environment in which team members could internalize the project’s goals and develop their own methodological approaches. Weekly one-on-one meetings—often unstructured and occasionally lasting several hours—provided a consistent framework for communication, mentoring, and mutual learning.

Despite this intentional structure, staff turnover presented recurring challenges. The departure of key team members—whether for academic appointments or professional advancement—was described not only as a logistical concern but also as a form of emotional and epistemic loss. Individuals carried with them deeply embodied forms of project knowledge that could not easily be documented or replaced. The PI acknowledged the uniqueness of each team member’s contributions, noting, “There’s no one like anyone else. You can’t replace a person, only reconfigure the team.”

In response, the PI adopted a practice of promoting from within. Junior researchers were provided opportunities to assume greater responsibility, and in doing so, many exceeded expectations. What initially emerged from necessity evolved into a strategic approach that balanced continuity with capacity building. This form of internal promotion addressed immediate gaps and supported professional development across career stages—an outcome the PI framed as both intellectually and ethically valuable.

Yet human dynamics were only one axis of sustainability. The PI’s concern extended equally to the preservation of digital outputs—particularly in relation to non-Latin scripts and complex textual traditions. Standard editing models, such as TEI/XML, were deemed ill-suited to the structure of Arabic manuscripts. The team therefore opted to develop custom tools that responded more intuitively to the demands of right-to-left script encoding and fluid textuality. While such decisions enhanced usability and philological rigour during the project, they also introduced new risks regarding long-term interoperability and institutional adoption.

“I knew from the beginning,” the PI explained, “you don’t work on an 8th-century tradition and let it disappear after ten years.”

Although sustainability planning was not explicitly required by the funding agency at the time of application, the PI took proactive steps to secure the long-term viability of the project’s outputs. These included negotiating with the university for post-project hosting, advocating for the integration of the edition into broader research infrastructure, and insisting on detailed internal documentation—ranging from GitHub repositories to graduate theses outlining software architecture.

Nevertheless, structural limitations persist. Institutional uncertainty, shifting IT policies, and ongoing budget constraints complicate efforts to formalize preservation pathways. The PI described this situation with a tempered sense of realism: “It’s an uphill battle. But it’s one worth fighting.”

Perhaps most notably, the interview underscored a broad and inclusive definition of sustainability—one that encompasses intellectual, technical, and relational dimensions. The project has cultivated a diverse team in terms of disciplinary background, gender, and academic rank. Credit is shared generously, including co-authorships with student assistants. Mentorship is embedded in daily routines, and collaboration is understood as a mutual investment in both knowledge production and professional growth.

In this sense, the project has come to function not only as a research endeavour but also as a sustained community of practice. Its continuity does not rest solely on software or servers, but on the relationships, values, and adaptive strategies that allow it to evolve in response to change. 

Insights from the Research Associate: Carrying the Technical Legacy

The RA’s journey offers an on-the-ground view of how knowledge transfer, role expansion, and emotional investment intersect to shape project sustainability—often in invisible but critical ways.

Originally hired as a student assistant to convert TEI-encoded XML files and upload them to the platform, they eventually developed a deep familiarity with the technical infrastructure of the database. When a key collaborator—their closest programming partner—left the team, they faced a sudden and overwhelming shift in responsibility. Much of the work had not been documented, which necessitated intense daily meetings to transfer knowledge. While this gave the RA greater control and understanding of the project, it also diverted them from their regular tasks and concentrated essential knowledge in a single person: themselves.

The weight of this experience was not just technical but psychological. While the RA valued the learning process and grew into their expanded role, they now worry about what will happen when they, too, eventually leave. The project’s workflows have grown increasingly complex, making it difficult to break tasks into smaller, trainable components for newcomers. As they explained, “it was also very hard for me to create tasks for a new programmer because it’s now so interconnected.”

The RA’s reflections also touched on the broader infrastructure of digital sustainability. They noted that while the team had successfully published parts of the edition on the University server, true longevity would require a long-term maintenance strategy—something they felt was missing. Without clear institutional plans for post-project preservation, and without standardized workflows across DH projects, the future usability of the data remains uncertain.

Yet, despite these systemic limits, their narrative remain grounded in pragmatic optimism. The RA is currently mentoring a new student on programming tasks, trying to rebuild a more shareable, modular workflow. Their reflections call for a deeper institutional responsibility: “Maybe the university should provide tools or standardized ways to make these projects more sustainable—not just leave it to researchers who have to reinvent everything.”

Insights from the Student Assistant: Fragmentation and Emotional Distance

The perspective of the student assistant (SA) offers a valuable lens into how early-career contributors engage with and interpret sustainability from their unique position within a Digital Humanities project. Over the course of their three years on the team, their responsibilities centred around preparing and uploading XML files to the platform, based on the team’s transcriptions and segmentations of Arabic manuscripts. These contributions, though often seen as peripheral, play a crucial role in the overall functionality and accessibility of the digital edition.

What emerges from the SA’s experience is a sense of structured participation: while their tasks were clearly assigned and executed with autonomy, they remained distinct from the project’s overarching design or decision-making processes. Still, their account demonstrates the ways in which student assistants become deeply woven into the fabric of a project, particularly through interpersonal connections. They noted the strong bonds formed with earlier team members, and how their departure left a noticeable absence—emotionally and professionally. The early phase of their involvement was also marked by team-building activities, which facilitated a strong sense of belonging and familiarity among members. As those activities tapered off, newer team members came to play a smaller role in her day-to-day work, subtly reshaping the social dynamic.

Interestingly, while the concept of sustainability had not been explicitly discussed during team meetings that included them, the interview itself prompted them to reflect on its importance for the first time. When invited to imagine a scenario in which all project data vanished without lasting output or publication, they responded with genuine concern. The SA expressed a strong desire to see the project reach its conclusion and hoped to witness its results publicly realized—suggesting that a deeper sense of investment does exist, even if not always made visible in daily tasks.

Their reaction to the interview underscores an important insight: sustainability awareness often emerges not only from formal training or top-down directives but also through moments of dialogue, reflection, and contextualization. That they were thankful for the interview initiating such reflection illustrates the transformative potential of including all team members in discussions about a project’s long-term goals and outcomes.

Sustaining the Human Infrastructure

Across the three interviews, a clear picture emerged: the long-term sustainability of a DH project is never solely a technical or institutional issue. It is also deeply interpersonal. The PI emphasized vision, delegation, and the emotional labour of letting go; the RA described the burden of undocumented knowledge and the fragility of continuity in the face of staff turnover; the SA highlighted how task specialization and clear roles can support workflow efficiency while also expressing appreciation for a team culture that welcomes new ideas and encourages personal growth.

Taken together, their perspectives reveal a project that is held together not just by platforms or preservation plans, but by a human infrastructure—an evolving network of relationships, expectations, mentorships, and affective investments. When this infrastructure is strong and transparent, transitions become opportunities, not crises. When it is neglected, technical robustness alone cannot guarantee sustainability.

Digital Humanities often prides itself on collaboration, yet too often the emotional and structural conditions of that collaboration remain invisible. If we are serious about sustaining the outputs of our field, we must also sustain the people who produce them—through inclusive planning, reflexive leadership, and honest conversations about what will happen after the funding runs out.

In the end, digital sustainability is not simply a matter of keeping data alive. It is about making sure that the stories, skills, and people behind those data are not forgotten.

The Future of Our Digital Heritage?

Human Heritage in the Digital Era: A Fragile Legacy

In our rapidly advancing digital age, the concept of heritage—the legacy of physical artifacts and intangible attributes of a group or society passed down from previous generations—has taken on new dimensions. Traditionally, heritage encompasses tangible elements like monuments and artifacts, as well as intangible aspects such as traditions, languages, and knowledge. These elements serve as vital links to our past, shaping our identities and guiding our future.

Why Do We Produce Heritage?

Heritage production is an intrinsic human endeavor. It serves as a repository of collective memory, a means to understand our origins, and a foundation upon which we build our identities. By preserving stories, customs, and knowledge, we ensure that future generations have access to the wisdom and experiences of the past.

The Role of Heritage in Human Development

Heritage plays a pivotal role in the evolution of humankind. It fosters social cohesion, provides a sense of belonging, and contributes to economic development. Engagement with cultural heritage has been linked to improved mental health and well-being, highlighting its importance beyond mere historical interest.

The Heritage We Produce Today

In contemporary society, much of our heritage is being created and stored digitally. From social media interactions to digital art, scientific research, and governmental records, our collective memory is increasingly housed in digital formats. This shift presents both opportunities and challenges for preservation.

The Invisible Backbone of Modern Society

Our digital infrastructure functions as the transparent base of modern society. We rely on it for communication, commerce, education, and even the production of food and warmth. Yet, this infrastructure is largely invisible and often taken for granted. In our survival as a people, we rely—often without question—on invisible infrastructures, whether too distant to see or entirely digital. The knowledge and systems that underpin our daily lives are stored digitally, making them susceptible to various risks.

The Fragility of Digital Knowledge

Despite the conveniences of digital storage, the knowledge we depend on is at risk of disappearing. Factors as simple as the introduction of new data formats, hardware obsolescence, or cyber-attacks—and even less probable, yet still possible events like natural disasters—can render digital information inaccessible. While measures like backups and security systems are in place, they are not foolproof. The challenges of digital preservation include data loss, file format challenges, fragility of storage media, rapid technological evolution, and lack of funding.

This leads us to a surprisingly simple question: what has a greater chance of being discovered in 1,000 years—a meticulously stored digital dataset from a decade-long, two-million-euro research project, or a little girl’s diary?

Essential Digital Resources for Students of Arabic Studies

In the past, studying Arabic literature meant spending hours in archives, flipping through bulky dictionaries, and deciphering various calligraphic styles in the manuscripts. While physical books in the library remain invaluable, the rise of digital humanities has given us access to powerful tools that support research, enhance analysis, and provide deep insights into Arabic literary heritage. 

The research process in Arabic studies is not always linear—sometimes, you start with a text and then consult dictionaries, while other times, you might begin with a dictionary to clarify a word before selecting a text. Depending on your focus, you may also need computational tools for text analysis or reference works for historical and literary context. To help students navigate Arabic studies more effectively, this post introduces essential digital resources, including text repositories, dictionaries, digital analysis tools, and academic databases.

1. Text Resources (Poetry & Qur’ānic Texts) → Find and Explore Primary Sources

Before diving into linguistic details, we need access to primary sources—poetry, historical texts, and often the Qur’ān. These digital repositories make it easier to find and work with Arabic texts.

Poetry & Arabic Literary Texts

  • Al-Diwan: A comprehensive database of classical and modern Arabic poetry, allowing users to search by poet, theme, era, and country. It provides valuable insights into the evolution of Arabic poetic forms and styles, making it an essential resource for students of Arabic literature.
  • Al-Maktaba Al-Shāmila: A comprehensive digital library housing thousands of classical Arabic works, including poetry, prose, and Islamic texts. Its powerful search functionality allows users to locate specific texts, keywords, and phrases across a vast collection, making it an essential resource for academic research. The library is accessible both on computers and mobile devices, with categorised browsing options for Islamic sciences, literature, and historical sources, facilitating efficient navigation and exploration.
  • OpenITI: A digital corpus of pre-modern Arabic texts designed for computational research. It enables students to conduct large-scale text mining, linguistic analysis, and comparative studies across thousands of Arabic texts. OpenITI is especially valuable for those interested in digital humanities, allowing for the exploration of stylistic trends, textual variations, and intertextuality in Arabic literature and historical sources.

Qur’ānic Resources

Understanding and analysing the Qur’ān requires access to accurate, well-structured digital resources. Whether you are studying its linguistic features, theological interpretations, or textual variations, these platforms provide essential tools for both beginners and advanced researchers.

  • The Noble Qur’an: A widely used online platform offering translations, tafsīr, and recitations of the Qur’ān in multiple languages. This resource is beneficial for comparative studies and linguistic analysis.
  • Tanzil: A high-quality, verified digital Qur’ānic text that ensures accuracy for academic reference and software development. It provides both Uthmani and Imlaei script versions, allowing students to study different orthographic styles. Its advanced search functionality, inclusion of pause marks, and customisable diacritic options make it a highly flexible tool for Qur’ānic studies, Arabic linguistics, and digital humanities research.
  • Corpus Coranicum:  A comprehensive digital project that compiles Qur’ānic manuscripts, variant readings, and historical texts related to the Qur’ān. It features a searchable database with access to early manuscripts, transliterations, and variant readings, along with a philological commentary examining the historical development of the Qur’ānic text. 
  • Quranic Arabic Annotated Corpus: A linguistically annotated database that provides morphological and syntactic analysis of the Qur’ān, allowing users to explore grammatical structures and lexical patterns. It also features a syntactic treebank, semantic ontology, and detailed word-by-word analysis, which makes it easier to explore grammar, syntax, and Qur’anic linguistic structures.

2. Dictionaries & Lexicons → Understand Word Meanings

Once you have found a text, you may need dictionaries and lexicons to interpret complex words and phrases. These resources not only help with translation but also provide insights into historical meanings, etymology, and linguistic variations.

  • Ejtaal: A searchable collection of Arabic dictionaries, including both Hans Wehr, and Lane’s Lexicon. While navigating between dictionaries can be challenging, the platform allows users to compare definitions across multiple lexicons within a single interface.
  • Lane’s Lexicon: A historical Arabic-English dictionary that provides rich etymological insights. It is particularly useful for research dedicated to classical Arabic literature and historical texts. Two versions of this lexicon are available, allowing you to choose the one that best suits your needs. (link I, link II)
  • Almaany: A modern dictionary offering contextual meanings and quick translations. Its user-friendly interface and extensive database make it an excellent choice for students working with contemporary Arabic texts.

3. Digital Humanities Tools → Computational Approaches to Text Analysis

While digital tools are not always necessary, they provide valuable insights by identifying linguistic patterns, tracking word frequency, and analysing textual structures. Computational methods enable efficient comparative analysis, visualisation of textual relationships, and deeper engagement with Arabic texts. These tools are especially useful for students exploring digital humanities, computational linguistics, and advanced text analysis techniques.

  • Voyant Tools: A web-based text analysis and visualisation tool that allows users to explore word frequency, collocations, and thematic trends in Arabic texts. It provides visualisations such as word clouds, frequency graphs, and keyword-in-context analysis, supporting both quantitative and qualitative research approaches.
  • Farasa: A suite of natural language processing (NLP) tools for Arabic text analysis, developed by the Qatar Computing Research Institute (QCRI). It offers tokenization, part-of-speech tagging, named entity recognition, and sentiment analysis, making it essential for Arabic text processing and linguistic research. The tools are accessible via an online demo page or through Web API services.
  • Arabic Romanization ALA-LC: A tool that converts Arabic script into standardised Latin transliteration, ensuring accuracy and consistency in academic work that involves transliterated Arabic terms. 

4. Reference Works & Academic Databases → Contextualise Research

Once a text is analysed, you may need secondary sources to support your arguments and understand historical, literary, or religious contexts.

  • Encyclopaedia of Islam II: a leading academic resource on Islamic history, culture, and linguistics, offering in-depth articles on historical figures, legal traditions, religious practices, and social structures. It provides authoritative, well-referenced information on Islamic civilization, with critical insights into both historical developments and contemporary interpretations.
  • Encyclopaedia of the Qur’ān: a comprehensive reference work covering Qur’ānic terms, concepts, personalities, place names, cultural history, and exegesis. It also includes essays on key themes, making it an essential resource for those exploring the Qur’ān’s content, historical context, and interpretative traditions.

Both Encyclopaedia of Islam II and Encyclopaedia of the Qur’ān can be accessed through the university network or via the VPN service.

  • Encyclopaedia Iranica: a comprehensive academic resource covering Iranian history, culture, literature, and its intersections with Arabic and Islamic studies. It is valuable for researching cross-cultural influences between Persian and Arabic traditions, providing in-depth articles by experts in the field on historical figures, literary movements, and cultural exchanges that shaped the region. 

Digital resources have transformed Arabic studies, making it easier to access, analyse, and contextualise Arabic texts. By integrating these tools into the workflow, you can enhance your understanding of Arabic literature, historical texts, and the Qur’ān.

Online Academic Projects

A better way of publishing

Academic publishing stands at a crossroads. The traditional system of scholarly journals, born in the age of printing presses and horse-drawn carriages, seems increasingly ill-suited to our digital world. While these venerable institutions have served scholarship well for centuries, they now represent a bottleneck in the free flow of academic knowledge. Digital projects can and should replace most traditional academic publishing, offering a more efficient, accessible, and transparent way to share scholarly work.

The Limits of Academic Publishing

The current academic publishing system is, to put it bluntly, ‌obsolete. Academic journals emerged in 1665, with the nearly simultaneous appearance of the Journal des sçavans on January 5th and the Philosophical Transactions of the Royal Society on March 6th. These publications filled a crucial need in their time, enabling scholars to communicate when no other reliable means existed. Before telegraphs, telephones, radio, television, or the internet, printed journals were indispensable for their ability to spread knowledge across borders and continents.

It’s worth remembering that early academic publishing was often the domain of wealthy hobbyists. Printing has always been an expensive endeavor, and the costs of production and distribution created natural barriers to entry. The Royal Society itself was initially composed largely of gentleman scientists, who could afford to pursue research as a passion rather than a profession. This historical context helps explain some of the traditions and structures that persist in academic publishing today, even though the social and technological landscape has changed dramatically. Today we thoughtlessly ape the behavior of noblemen from centuries past, but without the same goals and limitations that originally motivated it.

To be clear, there are still many cases where printing makes sense. No one is suggesting that we should stop printing books. But few scholars sit down to read academic articles as they would a novel. Instead, we jump from abstract to conclusion and back, skim for specific content and references, and generally dissect the paper as a data source instead of digesting it as a prose narrative. Digital formats are almost always sufficient for this purpose and often better suited to it. Carrying around reams of paper just to have access to the few pages relevant to one’s research is impractical, while an iPad can hold an entire library’s worth of academic papers in a smaller package than even one single print journal. More importantly, the ability to search, annotate, and cross-reference digital texts makes them better tools for research and scholarship than their printed counterparts.

More troublingly, the current system takes more than it gives. Academic publishing has evolved into a $25 billion industry with profit margins around 40%—numbers that would be impressive if not for the fact that most of the labor is provided for free. These profit margins exceed those of tech giants like Apple and Google, despite the publishers contributing relatively little to the actual creation of content. Researchers write articles without compensation (though one could argue they receive indirect benefits through career advancement), peer reviewers donate their time and expertise, and academics often handle editing duties themselves.

The structure of academic publishing is particularly problematic from an economic perspective. Most purchases are made by institutions on an ongoing basis without the opportunity to choose an alternative at any given moment, creating a market with limited to non-existent competition. University libraries, the primary customers for academic publishers, often find themselves locked into expensive subscription packages that eat up increasing portions of their budgets. Competition is crucial for market efficiency, yet decision-makers at institutions lack both the tools and incentives to reduce prices. The result is a system where prices continue to rise while services remain largely unchanged.

The traditional publishing model also inadvertently promotes fraud and questionable research practices. By making prose (with some figures and tables) the main focus, it becomes easier to bury unsupported hypotheses, engage in p-hacking, and in extreme cases, manipulate data. This problem is more widespread than many realize. A concerning number of published papers contain images that show signs of manipulation, statistical analyses that don’t stand up to scrutiny, or methodologies that are impossible to replicate. Oversight, where it exists at all, is thankless and often personally risky work. Short of outright fraud, the system encourages softer forms of academic misconduct, such as CV padding through unnecessary publication splitting, selective reporting of results, and the famous “least publishable unit” approach to research. Goodhart’s law applies to academics as well: “When a metric becomes a target, it ceases to be a good metric.” Gaming the system is rampant—openly practiced by many scholars and encouraged by a volume-driven publishing industry.

The emphasis on publishing in high-impact journals has created perverse incentives that work against good science. Researchers feel pressure to produce dramatic or counterintuitive results, knowing that null findings or incremental advances are less likely to be published. This pressure can lead to corner-cutting, overstatement of results, or worse. The replication crisis in psychology and other fields can be partly attributed to these systemic pressures.

The lack of transparency in traditional academic publishing is another significant concern. Research projects naturally develop over time, with false starts, dead ends, and changes in methodology. Single publications obscure this process, presenting a sanitized version that fails to capture the true nature of scientific inquiry. This “snapshot” approach to research documentation can make it difficult for other researchers to understand the full context of the work and can hide important details about how conclusions were reached.

We should remember that current publishing practices are not synonymous with good scientific practice, though they’re often presented this way. Peer review, now considered a cornerstone of academic publishing, wasn’t the norm until the mid-20th century. The review process and other forms of quality control don’t depend on paper articles or even the traditional article format. Alternative review processes exist and thrive in various contexts—consider the rigorous standards maintained by online communities like r/AskHistorians. Or they might be allowed to rise to prominence in a new structure. For instance, partial review processes developed for interdisciplinary work allow experts from multiple domains to evaluate research as a team effort, even when no single reviewer can address the entire topic alone. 

The history of peer review is itself instructive. Early scientific journals published submissions with minimal review, relying on post-publication discourse for validation. The modern peer review system arose gradually, primarily after World War II, as research became more specialized and the volume of submissions increased. This history reminds us that our current system is not the only possible approach to ensuring research quality.

Perhaps most frustratingly, the current system consumes vast amounts of grant money. Public funds support research that results in academic articles, which are then published by for-profit companies at high markups. In essence, we’re paying premium prices for services that websites can provide almost for free or with minimal advertising support. This represents a significant misallocation of resources that could be better spent on actual research, education, or public outreach.

The inefficiency extends beyond direct costs. Researchers spend countless hours reformatting papers to meet different journals’ submission requirements, navigating complex submission systems, and responding to reviews that sometimes seem more focused on formatting than content. All of this represents time that could be spent on actual research or other scholarly activities.

The Value of Digital Projects

It’s easy to complain about academic publishing, and we often do, but the best way to complain is to make things better. Digital academic projects offer a compelling alternative to traditional publishing. They can function exactly like academic papers when desired—consider ISAW Papers or, for that matter, open-access papers, which are already online digital publications. But they can also do so much more.

Digital projects can incorporate elements impossible in traditional publishing: non-sequential or higher dimensional structures for argumentation, hyperlinked text, multimedia content, interactive visualizations, and raw data repositories. Imagine a research paper where readers can interact with the data, running different analyses or exploring alternative hypotheses, in essence, replicating the results as they read about them. Or consider a historical study where primary sources are directly linked and searchable, allowing readers to verify claims and explore the material themselves.

The ability to update content continuously, while maintaining archives of previous versions, represents a fundamental improvement over traditional publishing. This approach aligns better with how research actually progresses—through iteration, refinement, and occasional correction. When errors are discovered or new data become available, digital publications can be updated quickly while maintaining a transparent record of changes. It also provides a way to “show one’s work” without cranking out small (often vacuous) papers, giving researchers more time to pursue big ideas, which naturally need time to develop.

The technical capabilities of modern web platforms far exceed the requirements for academic publishing. We can create rich, interactive experiences that enhance understanding while maintaining the rigorous standards of academic work. Digital projects can incorporate data repositories, computational notebooks, and interactive visualizations that make research more transparent and reproducible. Tools like Jupyter notebooks allow readers to not just read about methods but actually execute them, promoting reproducibility and deeper understanding.

Perhaps most importantly, digital projects can be made permanently accessible. Static webpages (of which Closing the Gap in Non-Latin-Script Data is one example) can already do everything an academic journal can do, and they require virtually no maintenance expenditure. Platforms like GitHub Pages make hosting scholarly content essentially free. This democratizes access to knowledge, allowing people from disadvantaged backgrounds or without institutional affiliations to access scholarly material.

The cost advantages of digital publishing are substantial. While there are still expenses involved in the scholarly communication process—copyediting, typesetting, server maintenance—these costs are orders of magnitude lower than traditional publishing. Moreover, many of these functions can be automated or streamlined using modern tools and workflows. (ProWritingAid is doing an excellent job of copyediting this blog post in real time as I write it.)

Digital publishing also enables new forms of scholarly communication that weren’t possible before. Researchers can share work in progress, receive feedback from a broader community, and iterate on their ideas in public. This more open approach to scholarship can lead to better research outcomes and more rapid advancement of knowledge.The potential for integration with other digital tools is another significant advantage. Digital publications can be easily indexed by search engines, making them more discoverable. They can incorporate modern reference management tools, making citation and bibliography management more efficient. (Hello Zotero, goodbye pedantic hair splitting over citation formatting.) They can include direct links to data and code repositories and other supplementary materials.

Looking to the Future

The transition to digital academic publishing won’t happen overnight, and some forms of traditional publishing will likely persist where they make sense. However, the bulk of academic communication can and should move to digital platforms. This shift would reduce costs, increase access, improve transparency, and better serve the primary goal of academic publishing: the advancement and dissemination of knowledge. (It may even better serve the secondary goal of academic publishing: the advancement and dissemination of academics.)

Several challenges need to be addressed in this transition. We need robust methods for preserving digital content over the long term. We need new ways of evaluating the impact and quality of digital scholarship. We need to ensure that digital publications are properly credited in academic hiring and promotion decisions. However, none of these challenges are insurmountable, and many organizations are already working on solutions. (There is much more to be said about best practices in digital publication, such as the importance of FAIRness, but this is not the subject of this post.)

The academic community has already begun this transition. The rise of preprint servers like arXiv and bioRxiv, the success of open access journals, and the growing acceptance of alternative forms of scholarly communication all point to a future where digital publishing is the norm rather than the exception. The humanities have been slow to catch up, but this is also changing.

Conclusion

The time has come for digital projects to replace most traditional academic publishing. The current system, born in the age of sailing ships and hand-set type, has served its purpose but now impedes rather than advances the spread of knowledge. Digital platforms offer all the benefits of traditional publishing—peer review, permanent archives, scholarly rigor—while adding capabilities that better serve modern research needs.

This transition represents more than just a change in format. It’s an opportunity to reimagine how scholarly communication works in the digital age. We can create systems that are more open, more efficient, and more effective at spreading knowledge. We can build tools that make research more reproducible and transparent. We can make scholarship more accessible to people around the world.

The technology exists. The benefits are clear. All that remains is for the academic community to embrace this change and begin building the publishing system of the future. The sooner we make this transition, the sooner we can redirect resources from maintaining an obsolete system to advancing the frontiers of knowledge.

HOW TO BE FAIR? 

MANAGING AND ARCHIVING DIGITAL CORPORA – THEORETICAL WORKFLOWS AND PRACTICAL EXAMPLES


The question of how to ensure the long-term FAIRness of our research data can never be discussed too much. The multitude of new technologies and a broad palette of tools and infrastructures make it easier to create more sustainable and interoperable solutions. However, opinions still differ significantly on which ones to implement and how exactly to do so. It is crucial to address these issues within the academic community to develop standardized solutions, and the workshop “Creating, Managing and Archiving Textual Corpora in Under-resourced Languages” was held with exactly this purpose in mind.

The workshop was conceived by DARIAH Working Groups Research Data Management and Multilingual DH, financed by DARIAH-EU Funding Scheme for Working Group Activities 2023-25, and hosted by the University of Hamburg on 28th to 30th August 2024. It brought together a large number of experts on multiple languages and various aspects of digital scholarship, including members of our project Closing the Gap in Non-Latin-Script Data (CtG), which resulted in the elaboration of standardized workflows for building, managing, archiving, and annotating multilingual corpora. Although the focus was on low-resource and endangered languages, these workflows are valuable to any scholar aiming to ensure the FAIRness of their work. The following paragraphs will outline the key aspects of the workflows, along with practical examples from our project.

  • Data source, preparation, and format

It is important to ensure that the research question you have in mind cannot be answered with data already available in digital form. If it cannot, you need to gather the materials and digitize them yourself, using methods such as OCR, HTR, or ASR. A crucial step is to check the ethical and legal considerations regarding your source materials and, if possible, select those that can be shared later, as this will enhance the reusability and impact of your research. Another key decision concerns the format of your data. Avoid using proprietary file formats like MS Office DOCX. From the outset, work with open-source file formats (e.g., TXT, JSON, CSV), which are widely supported and accessible without requiring specific software or licenses. Finally, regardless of the specific purpose or any modifications made to your data, always retain the basic textual data and metadata for archiving and documentation purposes.

CtG conducts meta-research on the field of Multilingual Digital Humanities, addressing the challenges of managing and preserving diverse linguistic data, particularly in non-Latin scripts. The meta-corpus consists of data on various digital projects that focus on Arabic and similar languages. The core principle of the project is to offer all its data in open access for further reuse. To this end, it is ensured that all information included in the corpus—whether sourced online or obtained through interviews—can be shared openly. The data is stored in JSON format, which is open source and easily human-readable, making it understandable to people with varying levels of technical expertise. This approach allows users to easily access, modify, and contribute to the corpus, fostering a collaborative research environment. Moreover, JSON is a widely used format, supported by most systems, ensuring the sustainability and interoperability of the data.

  • Metadata 

Metadata is crucial for ensuring the FAIR principles because it makes data findable and accessible by clearly describing the content and context of your research. Good-quality metadata is also essential for interoperability and reusability, allowing others to understand, interpret, and reuse your data correctly. Be sure to create your metadata in a simple and consistent format, such as JSON or CSV. Ideally, it should encompass the following aspects: provenance, intellectual property, ethical issues, access and reuse (licensing), as well as structural, descriptive, and technical information.

Gathering extensive metadata is of great importance to the scope of our project. Each data entry consists of three metadata sections: 1) record metadata, containing the project-specific UUID, the name of the person who added the entry, as well as the dates of its creation and last modification; 2) project metadata, which includes technical and descriptive information for each added project, from basic details such as title, hosting institutions, project duration, and involved researchers, to detailed records on research objectives, methodologies, technology stack used, and licensing, 3) metadata on the relations of the project, i.e. titles and UUID of related entries. This comprehensive metadata approach is crucial for ensuring that the data remains well-documented, easily searchable, and fully transparent, which facilitates long-term accessibility, reproducibility, and collaborative research.

  • Documentation and versioning 

Develop comprehensive documentation (guidelines) for the entire corpus, individual texts, your annotation processes, research objectives, and the overall research project. Ensure that this document is readily accessible alongside your corpus during the archiving process, and retain a secure copy for future reference. Make sure to document all changes when updating the corpus and preserve previous versions to maintain a complete record of the evolution of your research, facilitating transparency and reproducibility.

The project implements a Git-based file database to manage and version control the corpus data. This approach ensures that all changes are tracked, providing a clear history of modifications and facilitating collaborative efforts. In addition, the project team conducts detailed documentation of all dependencies and technologies used, as well as any changes made to the dataset. The database is hosted on a public GitHub repository, promoting transparency by making all data and changes publicly accessible, which encourages community engagement. This openness not only builds trust but also allows for peer review and validation of the data and methodologies employed.

  • Standardized vocabularies

To prevent confusion regarding information, utilize controlled vocabularies and authority files connected to recognized community repositories. This approach will promote accurate analyses and enhance the clarity of your information.

The project extensively utilizes authority files to link all entities representing institutions, locations, and individuals to identifiers such as VIAF, Wikidata, GND, or Geonames. To ensure optimal searchability and future retrieval of the data, the project team has developed a taxonomy system encompassing all concepts relevant to NLS-specific research. Adhering to the principles of Open Data and Open Science, the taxonomy is grounded in existing controlled vocabularies, including the DHA Taxonomy and TaDiRAH.

  • Licensing 

Make your corpus available in the most open manner possible while respecting any necessary restrictions. Keeping the FAIR principles in mind throughout the process, license your data under Creative Commons (CC), provided that your data providers permit it from legal and ethical standpoints.

Our data sources are twofold: we either gather information that is openly accessible online or directly contact researchers to obtain more detailed insights through interviews, during which we request explicit permission to share the information with the scientific community. This approach enables us to make not only all our workflows but also the entire dataset available under open access for further use on GitHub, licensed under the CC BY 4.0 license.

  • Archiving

Once you have assembled your corpus, metadata, and comprehensive documentation, it’s time to focus on the long-term archiving of your data. You can opt for certified data centers (e.g., CLARIN B-centre), data repositories affiliated with your research institution, or inter-institutional repositories like Zenodo. Institutions like CLARIN offer a robust, distributed network of 70 centers across Europe, providing not only long-term archiving but also tools to ensure the FAIR principles are applied to research data. CLARIN centers, especially those certified as B-centers, host repositories that ensure data sustainability and accessibility for future research projects. Be aware that some institutional repositories may have specific format requirements, which could necessitate migrating your dataset. In such a case, again, steer clear of migrating it into proprietary file formats. Always ensure that you deposit the most recent version of your data and documentation.

Hosting the data on GitHub and employing a Git-based management system offers a lightweight solution that does not require extensive infrastructure or resources. This makes it an ideal choice for projects with limited funding or technical support. The sustainability of this approach is ensured through the use of widely adopted tools and platforms, which are likely to remain supported and updated in the long term. Additionally, regular snapshots via the Web Archive, releases, and backups to Zenodo, along with the decentralized nature of Git repositories, further enhance the reliability and durability of the archived data.

CtG provides an example of radical FAIRness and openness, which, in the case of projects working with more sensitive data, might not be entirely possible. Additionally, the lightweight structure of a file-based database may pose challenges for projects with more complex, relational data. Therefore, it is important that each project from the outset develops a detailed data management plan that maximizes openness and sustainability, given the nature of its data. There are many initiatives and organizations that support researchers in developing FAIR data management strategies. One such resource is the SSH Open Marketplace, a platform where researchers can access and share workflows, as well as create and customize their own workflows for specific research projects. This platform enhances the discoverability and contextualization of research tools, datasets, and workflows, fostering collaboration and knowledge-sharing within the digital humanities community. Researchers are furthermore encouraged to license their images using CC-BY, which allows free use with creator credit, or CC-0, which places the work in the public domain for unrestricted use without attribution. Another valuable resource is the DARIAH Transformations journal, which emphasizes the documentation of methodological and research activities in the arts and humanities. This journal provides a platform for detailed documentation of data gathering, processing, and annotation, ensuring transparency and comprehensive record-keeping. The journal also requires structured metadata to support proper archiving and reusability of data. Its overlay model ensures that research and accompanying documentation are immediately accessible through open repositories, enhancing the reliability and availability of scholarly work. 

More detailed information about the workflows for building, managing, and archiving multilingual corpora, as elaborated during the workshop, can be found here, here, and here.

Optimizing Research Workflows for Students: A Software-Based Approach

Note: This post is primarily directed at students. It is based on a presentation I made recently for the students of Dr. Theodore Beers at his digital humanities seminar, Digital Philological Methods, during the 2024 Summer semester.

___

Preliminary Notes

Optimizing research workflows through efficient software use influences the kind of research questions one decides to work on. For example, one might choose between analyzing the thematic configuration of a single prophetic ḥadīth or tracing the development of a specific theme across a large corpus of ḥadīth material. The former would involve a close reading of the text, perhaps with pencil and paper, yielding results that, though relevant in their own way, can hardly offer generalizations. The latter would cover a larger corpus, reveal thematic trends and their evolution (either as a function of time or genre), and offer a path toward more concrete general statements backed up by a large body of evidence.

Embarking on the second research question—in addition to requiring extensive use of specialized software—entails a structured, organized approach to research and analysis. Such structure cannot afford to be bogged down by inefficient research practices, which encompass various crucial but potentially mundane and cumbersome tasks, including:

  • Building research bibliographies and conducting literature reviews
  • Taking and managing notes 
  • Inserting correct citations, tracking them, and updating them according to various criteria
  • Compiling a correct final Bibliography
  • Consulting one’s own research in the future

If not managed properly, these tasks accrue “research debt.”[1] These are unaddressed aspects of the research process that accumulate over time and become cumbersome to address. This builds resentment and affects quality down the line. The degree of resentment and the amount of chaos a poorly managed system could introduce is a function of the size of the research question—and of bad habits! Students who would like to embark on ambitious projects should address this head-on and early in their career.

This post aims to summarize key aspects of developing good research practices and communicate general ideas about software-optimized research.

These tips are based on my experience. While many of the points mentioned below are common knowledge to experienced researchers, they may be unfamiliar to students, especially those studying in the so-called “Global South”. In these regions, formal instruction on proper research methods is often lacking. I personally faced this challenge and had to teach myself. If students are to have a smooth academic journey and if they are to pursue academia as a career choice, they need to optimize their workflows.

Optimizing Research Workflows

Optimizing one’s research workflows ensures that one frees up important mental bandwidth for more important tasks, without being bogged down by inefficient collateral tasks that waste time and build resentment.

Efficient research requires automating and managing various aspects of the process, including note-taking and citation management. This can be achieved using various tools that can be adopted and incorporated into one’s routine quite easily. If not managed properly, through a system that is well implemented and reproducible, research turns into mundane manual labor rather than fulfilling mental work. This affects the quality of the research. To avoid this, students may keep in mind the following principles:

  • All processes can be optimized for efficiency.
  • One can always save time and focus on what matters without losing control or quality.
  • Simple, reproducible systems are desired.
  • A simple system is designed to automate repetitive tasks and reduce the chaos they introduce. The system itself should not be complex or difficult to use.
  • A simple system is one you set up once and use repeatedly. 
  • A habit of creating systems and workflows is important to ensure quality and keep away resentment—in itself, this process is also fun and fulfilling.

Once a system is in place, students can minimize the number of tasks they need to actively manage. This allows them to iterate more quickly on ideas and their implementations, follow multiple threads, and catch more errors in their thinking. As a result, they can correct mistakes more easily and write more compellingly and confidently. The upfront time investment in designing such a system and building the correct habits to implement it is always justified in the long term.

Below is a brief proposal for a system that addresses some of the collateral tasks mentioned above.

Essential Tools for Efficient Research: A Simple System

By way of example, and to help students get started with optimizing their research workflows, I propose a toolkit of software that can easily be incorporated into their work. In my experience through interacting with fellow students over the years, few (if any) had a system in place to help them in their writing projects.

Software Toolkit:

Here I include only open-source and freely available software, focusing on reference management and note-taking. Zotero, for instance, offers a great solution for managing bibliographies and inserting citations into drafts. As a reference manager, Zotero comes with a browser integration (in the form of an extension that supports many browsers). The integration allows the user to add items easily into their libraries.

With time, Zotero evolves into a personal library. It is important, however, to develop a habit of reviewing every bibliographic entry as soon as it is imported into Zotero. Over time, uncaught errors accumulate. Any neglect in addressing these errors would defeat the purpose of implementing such a system. A correct bibliography serves the student well in their writing, in conjunction with using the official Zotero integration with MS Word or Google Docs which allows for quick insertion and updating of citations using different styles. Furthermore, Zotero doubles as a PDF reader and a repository of notes, helping students manage, in addition to references, both associated PDFs and notes. The notes can be inserted into one’s writing as one does a citation. These notes need to be properly managed as well. The Better Notes Add-on can help with that.

Note-Taking 

Note-taking is a critical part of the thinking process. Richard Feynman once made sure to correct his interviewer, who characterized a set of notebooks as a “record of the day-to-day work,” responding: “I actually did the work on the paper.” According to Feynman, the notes are not the “record, not really, it’s working. You have to work on paper and this is the paper. OK?”[2]

The above highlights the quality of writing—or in our case, note-taking—as an integral aspect of the thought process itself; notes are not a record of work already done—they themselves are the work. Therefore, organizing those notes ensures organized thinking and efficient retrieval of information when writing a first draft, and particularly when revising and editing the final draft. 

There are various ways one can optimize their note-taking habits, such as the Zettelkasten method. Obsidian can be used to digitally implement some of the principles Niklas Luhmann popularized. One key principle is focusing on one idea, thought, argument, or example per note. Obsidian’s linking capabilities can be leveraged to create (using tags and backlinks) a network of interconnected notes and visualize them through the graph feature.

This method, or any method for that matter, can be integrated right into the reading process in Zotero. Using the Better Notes Add-on for Zotero, any notes taken while reading the PDF file of a resource can easily be synced with Obsidian and vise-versa, as both Obsidian use the Markdown file format to store the notes locally on the computer. To get the integration working, you only need to point both Obsidian and Better Notes to the same folder or markdown file.

The implementation of a system that uses the three components above is straightforward.

Implementation

Here’s a brief outline to guide you in setting up an optimized research workflow based on the principles described above. I encourage students to be curious and follow through with each point individually by reading the relevant documentation and watching tutorials shared by numerous researchers on YouTube.

For this context, here is an actionable plan:

  • Set up Zotero:
    • Install Zotero and the browser connectors.
    • Create a folder structure that mirrors your research topics.
    • Develop a habit of immediately reviewing and organizing new entries.
  • Integrate Obsidian:
  • Set up a vault for your research project.
  • Settle on a note-taking method that works for you
  • Understand the linking capabilities of Obsidian and incorporate them into your note-taking habits
  • Integrate Better Notes:
  • Install the Add-on inside Zotero
  • Sync your notes with Obsidian
  • Regular Maintenance and review:
  • Schedule regular reviews of your Zotero library and Obsidian vault to keep them organized and ensure that they adhere to your system.
  • Conduct regular more in-depth reviews, looking for connections between notes and potential research directions.
  • Be consistent, but not obsessive.

By consistently applying these practices, you’ll develop a robust, efficient research workflow that allows you to focus on what truly matters—generating insights and advancing your field of study. 

[1] I employ this term here in a way that is analogous to what software engineers refer as “technical debt.” https://en.wikipedia.org/wiki/Technical_debt

[2] https://www.aip.org/history-programs/niels-bohr-library/oral-histories/5020-5

How to Check Thousands of URLs for Broken Links

Introduction

At the core of “Closing the Gap” is a database of JSON files, each containing information relating to a research project in the Multilingual/Non-Latin-Script Digital Humanities. The files look like the following (a snippet; see the full file if you’re interested):

{
  "schema_version": "0.2.3",
  "record_metadata": {
    "uuid": "d1e6d69b-5e9a-4b4a-85ad-09aac56ed2d9",
    "record_created_on": "2021-11-08",
    "record_created_by": "Kudela, Xenia Monika",
    "last_edited_on": "2022-02-18"
  },
  "project": {
    "title": "Kalila and Dimna – AnonymClassic",
    "abbr": "",
    "type": "project",
    "ref": [],
    "date": [
      {
        "from": "2018-01-01",
        "to": "2022-12-31"
      }
    ],
    "maintained": null,
    "websites": [
      "https://www.geschkult.fu-berlin.de/en/e/kalila-wa-dimna/index.html",
      "https://kalila-and-dimna.fu-berlin.de/"
    ],
...

There are many places in our database schema where URLs are recorded. Projects have websites. We also include information about host institutions, the researchers who lead or are affiliated with each project, etc.—and their website URLs are generally listed.

With our project database already containing over 165 entries, the number of URLs can be expected to be several times larger. This is in fact the case: as will be described below, we have currently over 1,300 unique URLs. A challenge is that these links are scattered across different parts of a bunch of JSON files.

If we want to check the URLs in our database automatically for breakage—which we do—then we need some way of extracting them, along with, obviously, sending HTTP requests to all of them and tracking the responses. That is what this blog post is meant to explain: our usage of a tool called lychee to find and check links.

Sidebar: Why do this?

It is worth asking why we go to the trouble of checking hundreds of URLs and replacing them as they break. After all, one of the harsh realities of the Internet is that no URL will remain valid forever. With 1,300+ links to check (and counting), we find at least some breakage on a weekly basis. It often seems to be the case that, within a few years of the end of the funding period of a given DH project, the host university will allow the project website to go offline. With this in mind, why bother struggling against entropy?

Our answer is that part of the mission of “Closing the Gap” is to contribute to the improvement of standards of practice in the Digital Humanities. Research projects, and the institutions that host them, should strive to assign stable URLs and to maintain their validity for as long as is practical. How long is long enough? This is a matter of some subjectivity, but we think it is safe to say that any DH project that was active within the last decade should have an accessible website. (Relatedly, we advocate the use of static websites and web apps, which are easier to keep online over longer periods—and easier for the Internet Archive to snapshot.)

There have been cases in which we noticed that URLs at a given institution were breaking at high rates, and we were able to notify them and to see an actual improvement in the situation. So there is some method to the madness. We go through all the URLs in our database on a regular basis; find broken links; fix/replace them to the best of our ability; and notify site owners when we see larger problems. This allows us to keep our data relatively clean and to perform a kind of community service among DH researchers.

Extracting URLs

lychee is a modern, performant command-line utility for checking links. It is implemented in Rust, a relatively new and popular programming language that is designed to make it easier for developers to write correct, memory-safe software without sacrificing performance. There are so many excellent CLI tools written in Rust that even non-programmers might benefit from interpreting its use as a suggestion of high quality.

At any rate, the easiest use case for link-checking is an HTML document, since one can at least parse the HTML and look for URLs in <a> elements. lychee does this nicely. It can, in fact, be pointed at a webpage online, wherein it will find all links and check them. We can use as an example the homepage of the Kalīla and Dimna Project at the Freie Universität Berlin (and you will notice a few broken links if you run this command):

lychee https://www.geschkult.fu-berlin.de/en/e/kalila-wa-dimna/

The lychee documentation explains more about the options that can be set, the file formats supported, etc. But, again, a simple approach will not work for the “Closing the Gap” database. We have links in hundreds of JSON files, sometimes deeply nested. There are also many duplicate URLs. What we need is to iterate over the files, extract links from each, and generate a unified list, which can then be checked.

There would no doubt be many different ways of accomplishing this. The approach that we chose is to couple lychee to another CLI tool (also implemented in Rust), fd-find. The following command, when run in the root of our repository, recursively identifies all JSON files:

fd -e json

And the other preparatory command that we need, with lychee, takes a file (JSON or otherwise) and “dumps” a list of all URLs found therein. The list can then be written to an output file, which we will use later to check the links:

lychee --dump [some_file.json] > links_list.md

We can connect the “find” command to the “dump all links” command by using the -x flag in fd-find. That is, we ask for all JSON files, and for the links to be extracted from each, collecting them in a single list:

fd -e json -x lychee --dump > links_list.md

Now, as you can imagine, it will be easy to point lychee’s link-checking function at this list. In the case of the “Closing the Gap” database, the “dump” process yields a list of nearly 2,200 URLs. By the time that duplicates are weeded out and various invalid links are skipped over (see below), we end up with more like 1,325 URLs to check.

Checking Links

This is, in a way, the easy part. Assuming that we’re in the root of the “Closing the Gap” repository and have generated the links_list.md file, we can run the following command (with a few options set, to be explained below):

lychee --max-concurrency 16 -m 3 links_list.md

The option --max-concurrency 16 tells lychee not to attempt to check more than 16 URLs at a time. We set this limit after finding that, by default, lychee would try to work too quickly, generating spurious errors. You can feel free to remove this option or adjust the value to something that works on your machine. Just be on the lookout for link-checking errors caused by the submission of too many HTTP requests at once.

As for the option -m 3, it sets a limit on the number of times that lychee will allow for a link to be redirected. This is somewhat subjective. It is normal for one URL to redirect to another, and even for a chain of several redirects to occur before the web client is given a final, substantive response. At the same time, we have found that a large number of redirects is sometimes indicative of an actual problem. e.g., the website of a DH project may have been taken offline, but the host university set things up so that links to that site are redirected to the university homepage. These are in fact broken links, but they will be considered valid by lychee because they eventually lead to a successful response (albeit for a different resource). We can test for problems like this, to an extent, by limiting redirects.

How much redirection is too much? Again, this is subjective, but for the links in our database, we have found that we can fairly easily set a limit of 3. If we lower this to 2, lychee errors on more innocuous redirects—but we can manage this by updating the URLs in question. In fact, we have been using a limit of 2 internally, since we prefer to err on the side of strictness and to accept the occasional tedium that it produces. A limit of 3 is what we can more comfortably recommend to others.

Dealing with Errors

If you manage a website or a database that contains a substantial number of links, then you will soon encounter “errors that aren’t really errors,” or “errors that aren’t our fault.” Pages sometimes go down temporarily. An HTTP request can fail for any of a huge variety of reasons. And, it seems to us, a growing number of servers are configured to blanket-deny requests from command-line tools, presumably for security reasons and/or to make scraping more difficult. (The same libraries that allow lychee to check hundreds or thousands of URLs in a matter of seconds, could be used by bad actors to launch DDOS attacks.) So there are links that you will find impractical to check programmatically.

There are ways to mitigate this problem—basically, to avoid sending requests to URLs that are guaranteed to fail, and to spare yourself the hassle of being bombarded with error messages that you have no way of fixing. With lychee, you can specify in a configuration file which links it should skip over. Since this post is already more than long enough, we will not go into the details here; but you can read about this in the lychee documentation, and look at the lychee.toml file in the “Closing the Gap” repository. You will see, for example, that we do not bother checking any links to the website of the British Library, since it denies access to command-line tools.

The inevitability of encountering a large number and variety of errors with link-checking also makes it challenging to automate the process. For the time being, we are checking links in the “Closing the Gap” database manually—that is, manually triggering a command, which then runs automatically and reports a list of errors. As long as we make sure to do this once in a while, we can keep up with actual cases of link breakage, while ignoring errors that are not actionable. We do hope to add, at some point, a further degree of automation to this process: lychee can be run in GitHub Actions. Even then, however, we would have the link-checking workflow run at a certain interval (weekly?), simply to generate a list of errors. It would still be up to us to determine which errors we can fix, and to do so.

Conclusions

As has been explained above, our dual modus operandi in checking URLs is to keep our database clean and up to date (to the extent feasible), and to contribute to best practices in the community of Multilingual DH researchers. The Internet will always be a chaotic world, and that’s ok. We should just invest a bit of effort so that the projects on which we work remain findable and accessible over a long enough period that the public can benefit from them.

We encourage you to follow the work of “Closing the Gap in Non-Latin-Script Data” via our GitHub repository, our website, and our page under the website of the Seminar for Semitic and Arabic Studies at the Freie Universität Berlin. This post was written by Dr. Theodore Beers, who sometimes discusses topics relevant to Digital Humanities research on X/Twitter.