It was a tremendous surprise to be nominated for, and to receive, this award, and I would like to thank the Emmett Leahy Award Committee for this honour. I feel both privileged and humbled to join such a distinguished group of recipients.
I have enjoyed the very considerable good fortune that my working life has coincided with the emergence of digital preservation as a new discipline – one which has taken us beyond the known lands of archival and library science, and into strange, exciting and sometimes mysterious realms. It has been an adventure which, for me, has spanned domains as diverse as archaeology, archives, and IT, and the challenges of preserving everything from legislation to Flash games, geophysical survey data to virtual reality models. It has been a tremendous privilege to have the opportunity to play a part in the exploration and charting of such a rich new world, but I have only been one of many. I owe a profound debt to the successive managers who have trusted and empowered me to participate in such a novel and uncertain enterprise, and to the generosity, enthusiasm and wisdom of my fellow explorers - this award is dedicated to them. If ever a venture has depended on collaboration and mutual endeavour, it is this.
When I first began to tackle the challenges of digital preservation in the mid-1990s, it was genuinely possible for one person to read virtually everything written on the subject, to be familiar with all the main initiatives and with those working in the field. It is a measure of success that, today, none of these things is feasible. Digital preservation has come a long way. It has moved from theory to practice, from experiments and pilots to robust, commercial-grade tools and services, from the lab to business-as-usual. It has engaged the interest and participation of governments, commercial suppliers, memory institutions, higher and further education, funding bodies and the private sector. But as much as we can look back with pride and satisfaction on what has already been achieved, there is even more to look forward to. Many big tasks lie ahead – indeed, arguably, some of the greatest remain. I would like to take this opportunity to venture a few brief thoughts on some of these challenges, and our prospects for addressing them.
As a community, digital preservation has benefited hugely from the diversity of its members and their skills, including the various curatorial and information management sectors, IT, and many facets of academia. And this has served it very well in bringing fresh perspectives to bear on difficult problems. But it can also engender a sense of uncertainty as to where it fits, both as a discipline and a set of skills, alongside these more established fields. Should we train new generations of archivists and librarians with digital curation skills, or IT staff to understand archives? Do we need a new profession of digital curators, perhaps analogous to conservators, or do we expect all curators to acquire digital skills?
The answer is, of course, all of the above – digital curation must become part of the core skill set of all information management professionals, whilst enabling those with the inclination to specialise further. But this is not in itself enough – we must also engage and enlist the technologists, the educators, the media and the researchers, and we must embed the idea that good digital information management is a basic prerequisite for good management as a whole. Great progress has been made: digital curation skills are now recognised and taught as a key part of professional information management qualifications, as well as in their own right, while traineeships increasingly offer an entry route for gaining vital work experience. There is also a wide range of excellent vocational training. We must continue this momentum, in particular by expanding the opportunities for traineeships, and using them to attract entrants from as diverse a range of backgrounds and disciplines as possible. There must be an active dialogue between educators and employers, to ensure that new generations of librarians and archivists are equipped with the skills that their first jobs will demand. But we must also do more to embed basic awareness into the training and development of the content creators and the managers of the future, wherever they may be.
Another measure of the maturity of a discipline is its approach to ethics and professional good practice. There are long-established ethical codes and standards in the worlds of traditional conservation and information management, based around key principles such as protecting the integrity and authenticity of the object; a rigorously scientific approach based on investigation, evidence and testing; minimum intervention; reversibility; full documentation; respect for legal frameworks; transparency; professional development; and working cooperatively and collaboratively. While many of the same principles are implicitly accepted and applied in digital preservation, there has to-date been comparatively little explicit study of their application in the digital domain beyond the very high level provisions of documents such as the UNESCO Charter on the Preservation of the Digital Heritage. There are notable exceptions, such as the development of a code of ethics by the Digital Repository of Ireland, but a wider debate and analysis of the ethics of digital preservation would be immensely helpful in consolidating and codifying the growing body of hands-on experience and emerging good practice, as well as helping to address elements of the recent UNESCO Recommendation Concerning the Preservation of, and Access to, Documentary Heritage Including in Digital Form. It would also provide a context in which to frame important ongoing debates on aspects of methodology. For example, there has been much discussion about the significance of validating files against format specifications, and the extent to which repositories should intervene during ingest to amend files which cannot be fully processed by characterisation tools. The potential consequences of doing so, and the attendant ethical questions, were recently illustrated very powerfully by Elvia Arroyo-Ramirez at the 2016 PASIG conference in New York, in relation to the inability of many tools to handle Spanish language diacritics in file names. Such issues highlight the need to sometimes step back from the detail and consider core principles, to re-engage with the 'why' of what we do as much as the 'how'.
If digital preservation is to become ubiquitous, then we need to do much more to make it attractive and accessible in those areas where adoption is currently low, such as the private sector, smaller organizations, and the developing world. We need to lower the barriers to entry and cultivate the desire and determination to surmount them. This means that we must develop sophisticated approaches to defining what good looks like, recognising that this can cover a spectrum of capability, and we need a better suite of tools to help organizations define an appropriate target operating model, and a path to achieve it.
Traditionally, we have looked towards standards and certification schemes to provide that benchmark, but maturity models are emerging as a complementary approach. Maturity models provide a means for organizations to assess their capabilities in a particular area. One of their attractions is that they offer a nuanced and multi-faceted way of thinking about capability, which recognises differences in need within and between organizations. They allow us to define not only a granular series of processes which together constitute a particular capability, but also gradations of capability for each of those processes along a path which begins with awareness of the need to develop it, follows the steps required to acquire it, and ends with its realization. The realization of that capability may itself be at varying levels of complexity and sophistication, ranging from a minimal, baseline capability up to an optimal process which proactively monitors performance and undertakes continuous improvement. Maturity models recognise the fact that there is no one-size-fits-all approach to digital preservation: different levels of capability may be appropriate for different organizations, and a single organization may have different levels of capability for individual processes. They provide a powerful and effective means of assessing both current and desired levels of capability, and of articulating what is required to bridge any gap.
The utility of maturity models is increasingly being recognised: amongst the many excellent examples in this area, the Digital Preservation Capability Maturity Model developed by Charles Dollar and Lori Ashley, and the National Digital Stewardship Alliance's Levels of Digital Preservation have become two of the most widely cited initiatives, while the EU-funded E-ARK project has been doing impressive work to develop a sophisticated maturity model for information governance, and to pilot its use amongst a range of institutions.
In particular, maturity models may help us to address one of the biggest challenges currently facing the digital preservation community, namely how to support the building of capability in the developing world. Good records management is fundamental to achieving transparency and accountability within public administrations and institutions, protecting civil and human rights, tackling poverty and corruption, promoting economic growth, supporting development and improving services. Nowhere is this more critical than in the developing world, and organizations such as the International Records Management Trust have long led the way here, working with governments, national and supranational agencies to establish and embed rigorous recordkeeping systems. In the past this has, of necessity, focused on physical records but electronic records management is now very much to the fore. This is, in turn, driving a growing recognition that digital information systems, and indeed, economies, must be underpinned by sound digital preservation foundations. Reliable digital records and datasets are essential not only to hold governments to account, but also to enable the planning and delivery of policy, and to target and monitor the impact of investment and progress towards development goals.
Pioneering work is already taking place. For example, institutions such as the Eastern and Southern African Management Institute's Centre of Excellence for the Management of Electronic Records are leading the establishment of practical digital preservation capacity. However, having the means to ensure that digital records can be systematically managed and sustained remains a distant prospect for many. To overcome this we need to define a ‘minimum viable service’ for digital preservation, which balances delivering the vital benefits with an attainable level of investment and capability; maturity models offer a very promising basis for defining what good (or good enough) looks like, and how to get there, which would be very well suited to this scenario.
The emergence of Preservation-as-a-Service is also likely to prove significant, by reducing infrastructure and expertise overheads, and driving down the costs of digital preservation. However, its viability in the developing world still depends on a number of factors, including robust connectivity, sustainable long term economic models and an appropriate skills base. Recognition of the need to incorporate digital preservation into international development initiatives, with attendant resourcing, is therefore imperative.
Of course, we can also apply the principles of a maturity model to the discipline of digital preservation itself. We have certainly achieved much success in raising awareness of both the challenges and opportunities and, in recent years, the narrative has moved beyond notions of a ‘digital dark age’, heralded by the forces of format obsolescence and to which digital archivists must respond by choosing between the rival factions of migration or emulation, towards a more nuanced and sophisticated understanding of the risks and opportunities, and the strategies required to respond to them.
We have a good general grasp of the threat models and the mechanisms by which our continued access to authentic digital content may be broken; we have a growing repertoire of techniques to counter them, debated and refined in an open and constructive manner; we can choose between a range of mature, operationally-proven and well-supported commercial and open source solutions; and we can draw upon a substantial body of practical real-world experience, acquired over a number of decades. More broadly, it is increasingly understood that digital preservation is driven as much by opportunities as threats, that it is about articulating the value of digital information as a sustainable resource which underpins economic, scientific, cultural, political and social activity.
However, while digital preservation capability is becoming widely embedded within memory institutions of all sizes, adoption in the private sector is still generally low and, as discussed earlier, there remains a significant capability gap in the developing world. Where we do have capability, it is often not well integrated into the mainstream of organizational activity. Finally, while much has been done to triage, secure and maintain immediate access to digital content, more work remains to develop scalable and effective policies and practice for long term preservation management.
Ultimately, if digital preservation is genuinely to become a utility service, as it must, then it needs to happen wherever the data is. The centralised repository model, in which content is moved to a special place where preservation occurs, may not be the best approach in every case, for example when managing digital continuity for content which needs to sustained for long periods, but not permanently. In recent years, the concept of 'in-place' records management, where centrally-held disposition and management rules are applied across a range of content management systems, has emerged as an alternative to the centralised electronic records management system. I would suggest that we need to consider a similar model for in-place digital preservation. One can envisage a set of common tools and services, managed centrally but applied locally within any storage or content management environment, which provide all the fundamental preservation services, from monitoring and repairing bit-rot, through characterising digital content, to mediating its continued usability and accessibility, and facilitating reuse. The ability to apply preservation services from a single source to varied existing content repositories would significantly expand their reach and impact.
We owe it to ourselves, to our community, and to current and future generations to deliver on these challenges. We must embed the means to sustain digital information - to maintain continuity as systems, organisations and cultures evolve and change - as a basic, achievable and fundamental necessity of any digital infrastructure. This will be hard, but it is vital.
However, I feel more convinced than ever that we can and will succeed. Later this month we will be celebrating the 2016 Digital Preservation Awards, for which I have had the honour of chairing the judging panel. This year has produced the most international field in the history of the awards, with nominations received from ten countries, spanning four continents – one need look no further than this for evidence of the richness, diversity and strength of our community, or to engender confidence in our ability to embed digital preservation as a fundamental enabler to creating an enduring digital society.
Adrian Brown
November 2016