Losing our History

My wife and I spent the Independence Day weekend this year in Washington DC. In addition to watching the fireworks from the base of the Iwo Jima memorial we visited a number of other memorials and museums. But probably the most amazing place we visited was the National Archives. Aside from the U.S. Constitution and Declaration of Independence, the National Archives is in fact an archive of the U.S. government’s correspondent, business and legal transactions some of which are on exhibit. These exhibits include excerpts from the infamous Nixon Watergate tapes to (my person favorite) a letter from a 10-year-old Fidel Castro to President Franklin D. Roosevelt dated November 6, 1940, asking for a “ten dollar bill green American” (maybe Roosevelt should have sent him the 10 bucks – you never know). The fact is that the National Archive is a repository of everything the U.S. Government is involved in. Everything. The good, the bad, the ugly. The greatest achievements, the finest moments and the things we would like to forget. Especially the things we’d like to forget. This is everything from the most visible, substantial and important documents like the U.S. Constitution to mundane interoffice correspondence, which can in the long run be just as important historically.

You might think that the digital age has made the job of the National Archives quite a bit easier. Unfortunately nothing could be further from the truth as this article from the New York Times points out.

Countless federal records are being lost to posterity because federal employees, grappling with a staggering growth in electronic records, do not regularly preserve the documents they create on government computers, send by e-mail and post on the Web. Federal agencies have rushed to embrace the Internet and new information technology, but their record-keeping efforts lag far behind.

Moreover, federal investigators have found widespread violations of federal record-keeping requirements. Many federal officials admit to a haphazard approach to preserving e-mail and other electronic records of their work. Indeed, many say they are unsure what materials they are supposed to preserve.

This confusion is causing alarm among historians, archivists, librarians, Congressional investigators and watchdog groups that want to trace the decision-making process and hold federal officials accountable. With the imminent change in administrations, the concern about lost records has become more acute.

While those conspiracy theory fans among us (okay, I admit it – but the truth is out there) prefer a more tantalizing threat like a shadowy cabal that secretly removes and suppresses information embarrassing or threatening to their members, the reality is much more mundane – and insidious. And it’s a whole lot harder to address.

“The Achilles’ heel of record-keeping is people,” said Jason R. Baron, the director of litigation at the National Archives. “We used to have secretaries. Now each of us with a desktop computer is his or her own record-keeper. That creates some very difficult problems.”

That’s right – it’s those pesky end users. You know, those regular folks who are just trying to get their job done as efficiently as possible. Yeah, those people who we never have the time or budget to provide with decent hardware and software. And forget about education (no money for that in this year’s budget). Oh, and the folks who actually control the purse strings don’t have “keep a public record of the stupid things we do” at the top of their must-fund list. (Yes! I knew I could slide a conspiracy theory in there).

All this is really patriotic, and sufficiently alarmist to get some good hits on Google, but what does it have to do with security, Mr. Security For All?

Actually – everything. Remember the CIA triad: Confidentiality, Integrity and Availability. This issue is fundamental to both Integrity and Availability. From Wikipedia:

  • Integrity – In information security, integrity means that data cannot be modified without authorization. Integrity is violated when an employee (accidentally or with malicious intent) deletes important data files.
  • Availability – For any information system to serve its purpose, the information must be available when it is needed. This means that the computing systems used to store and process the information, the security controls used to protect it, and the communication channels used to access it must be functioning correctly.

I think we can all agree that not saving important information through neglect is the same thing as deleting important data. And when future generations – or a researcher today – can’t get access to an email that is germane to their research because it was never saved violates availability.

So how do we go about mitigating this threat? There is already a program in progress to bring the National Archives more fully into the 21st century, but it is not without it’s all too typical problems.

The National Archives is in the early stages of creating a permanent electronic record-keeping system, seeking help from the San Diego Supercomputer Center at the University of California, and from some of the nation’s best computer scientists.

The electronic archive is behind schedule and over budget. But officials say they hope that the project, being developed with Lockheed Martin, will be able to take in huge quantities of White House records when President Bush leaves office in January.

As a point of reference 32 million White House e-mail messages were preserved as records of the Clinton administration. The National Archives expects to receive hundreds of millions from the Bush White House. And since disputes over White House records have occurred at the end of the last three administrations, we can count on more litigation in January.

So here’s a bold idea: why not take the money that will be flushed down the litigation rat hole and put it towards the electronic record-keeping system? Oh, but wait, that would mean that politicians would have to be subject to the same laws, standards and directives that all government employees are. Or maybe Lockheed Martin could get some help from the IBM Almaden research guys on storing, indexing and accessing insane amounts of information since the Webfountain project went dark. Or underground. (Yes! another conspiracy theory reference).

In any case this is a risk that must be managed – and soon – before we lose what amounts to our civic cultural heritage.