Built to Last: Data Lifeboats for Distributed Digital Heritage

Have you ever clicked a link that shows you a 404 error page? Have you ever lost data or photos when a web service closed down? While the common notion is that digital content is everlasting, the reality is often different. Our increasingly monolithic online platforms, along with the valuable data they host, are not as enduring as we believe. We need new models for preserving our digital culture — decentralized, distributed, and built for longevity.

As the web ages, we have seen that web links and their host platforms can — and do — disappear, taking huge swathes of our digital histories with them. Take, for instance, a 2024 study by Pew Research Centre showing 38% of web pages that existed in 2013 are no longer accessible today. When platforms sunset their services, there are significant losses: Shutterfly closed down Share Site, and Apple terminated its My Photo Stream, resulting in millions of deleted photos for people around the globe. In 2017, Verizon nearly shut down Flickr, one of the most extensive picture collections humans have ever assembled. It was only when SmugMug — a smaller, family-run, photography-first company — stepped in to buy the service from Verizon that Flickr’s collection and community were saved from deletion. 

Flickr was a pioneer of Web 2.0. It made it easy for the people to share their photos with friends and family. After two decades, it has grown to contain tens of billions of images, contributed by millions of people. I was part of the team that brought Flickr into the world in 2004. I was the lead designer until 2007 when I shifted to creating the Flickr Commons program, which focused on making photography collections from the world’s cultural institutions more easily accessible to the public. 

Flickr Commons went live in January 2008 with the Library of Congress as our first partner. Since then, it has brought together almost two million images from over 100 cultural organizations of all sizes — from one person running a tiny local history archive to the collections in our national libraries and archives. On average, Commons accounts contain about 3,000 photos. Almost half a million Flickr members subscribe to new uploads, clocking a huge 4.59 billion views across the program. 

Whether personal memories or cultural heritage, each photo across Flickr and Flickr Commons shows us an “observable moment", John Berger calls photographs: a document of lives lived. Imagine if this invaluable trove of human history were wiped from public view and our collective memory.

How do we make a stand to ensure this will not happen? How can we build robust archival copies immune to the corporate economics, cyberattacks, authoritarian censorship, link rot, and tech failures that hinder web archiving today? How can we, with careful deliberation, extend the life of our digital works, not just in the Flickr ecosystem but across the web? 

Here’s one approach: harness the power of the decentralized web to empower greater autonomy over our digital content and ensure that it endures for decades to come through distributed infrastructure and simple, responsible technology. 

In 2022, I created the nonprofit Flickr Foundation to explore the next phase in the evolution of digital photography, with support from SmugMug’s COO, Ben MacAskill and Filecoin Foundation for the Decentralized Web (FFDW). The organization is a commitment to shaping the next 100 years of web content and ensuring our shared history can persist for generations to come. Our first major project is called Data Lifeboat, a tool to give our digital culture buoyancy in tumultuous digital and economic seas.

Data Lifeboat diverges from many of the prevailing trends in digital preservation over the past decade. So far, the instinct has been aggregation: amassing content and records on platforms like the Digital Public Library of America or Europeana. This is a centralized approach. I believe it risks becoming unsustainable as the volume of digital content expands beyond available resources, and far-reaching platform dependencies make web archiving more cumbersome and less accessible. The systems we have built so far to manage our shared digital heritage are fragile and unsustainable. No single company or organization should have the ability to delete massive pieces of human history, nor bear the sole responsibility for maintaining them. 

With the Data Lifeboat, however, we're developing an archival framework that reimagines how we can preserve large, networked cultural resources, charting a new course away from widespread centralization and consolidation. And through our collaboration with FFDW, we have built a software tool that embraces decentralization at both technical and cultural levels. 

Data Lifeboats are light and flexible, unburdened by platform dependencies. These ‘archival slivers’ have a greater chance of survival. From a technical standpoint, Data Lifeboats enable Flickr members to curate collections and secure their data in a ‘light to store, easy to move’ package: a downloadable, compressed file containing images and their metadata. This implements the digital preservation principle LOCKSS (‘Lots of Copies Keeps Stuff Safe’) by facilitating the physical distribution of self-contained, versatile, and decipherable copies, providing redundancy and resilience should the original Flickr.com ship ever go down. While Data Lifeboats are currently downloaded and stored by their creators, we're developing a Safe Harbor Network — a distributed network of managed "docks" or servers where trusted network members can preserve Data Lifeboats for long-term safekeeping. 

Beyond its technical aspects, the Data Lifeboat is also a conduit to decentralizing culture by dispersing cultural power from central entities, funneling it towards individual users and smaller organizations. The contents of a Data Lifeboat are selected by Flickr users themselves, ensuring that what matters most to people is what gets preserved. This disrupts the hegemony of traditional archival structures, where historical power holders are typically overrepresented in cultural heritage collections. By decentralizing selection and preservation tools, Data Lifeboats support and distribute diverse voices and more inclusive retellings of our collective digital culture.

This approach becomes increasingly vital as we observe platforms' asymmetric power to censor and remove content that conflicts with prevailing political agendas. With internet shutdowns increasing annually across more countries, and the erasure or shadow-banning of critical accounts, the importance of individuals controlling their own data grows more urgent. Decentralized networks, with their inherently distributed structure, offer resilience against single points of failure and censorship. Without controls like these, the security of our collective digital heritage remains at risk. While Flickr’s billions of observable moments are a starting point for this work, our goal is to extend this approach beyond the Flickr community. This is the beginning of a collective effort to underscore the need for more resilient digital systems. 

Right now, what’s on the Internet isn’t forever. Our digital memories and shared cultural heritage are vulnerable to time as much as any physical keepsake. I’ve heard archivists say a piece of paper lasts longer than a website. Our memories matter. It’s time we equip ourselves to make sure they last for the long term. 

George Oates is Co-founder and Executive Director of the Flickr Foundation.