Data With Spherical Indexing & Open Archive for Governments

By: Joshua Konkle

Government, specifically the US Federal, State and Local Government is challenged with a requirement to preserve information for the duration of the Republic (more information on electronic archiving and discovery is at www.estorian.com). The duration of the Republic already spans hundreds of years. The first 175 years stored the Republics business on paper. However, since the 1950's new types of storage media have emerged. That media maintains the historical audio, video and text of the Republic. It is incumbent upon the US Government to make that history available, as needed, to the constituents of the United States of America.

Accessing the history of any department within a branch of government can seem trying at times. For example, the technology used by the Office of the President required end users to decide which emails were necessary for long term preservation, as opposed to storing all of the data, regardless of personal interpretation. Therefore, accessing the unabridged version of the email records for the Republics highest office was hampered by user precision and recall, not technology. Where precision represents the number of correct hits in a return set of specified length; recall represents the number of correct returns relative to the total number of possible correct returns. Specifically, deciding which emails should and shouldn't be kept for long term retention is best left up to software and open records managers.

In order for a manager who is responding to a Federal Open Information Act (FOIA) request to find data, the data must be stored in an open format. Open formats are critical for all agencies within the Government, because they never know when data may be requested. If the data is proprietary and the request is 50 years from now, it's possible the developer or the data-type is long retired or deceased. Therefore, accessing the data may become impossible, unless a computer with the right software is under maintenance at The Smithsonian.

If you are using Estorian LookingGlass you can be assured that your data is accessible. Data accessibility is maintained by LookingGlass Spherical Indexing solution, which is based on open format SQL queries. LookingGlass takes a standard approach to storing the data within an open storage system, such as NetApp NAS or Permabit Enterprise Archiving. The standard data approach, used by most of the vendors supporting an archive, is to take the Microsoft MSG files and store them in collections of ZIP files.

The Microsoft MSG files are the Message file data-type created by Microsoft Outlook. Granted, the data-type may have been created by Microsoft, but it has been in use for over 13 years. Specifically, Microsoft MSG files were introduced when Microsoft released Windows 95 Preview Program (April 1995). Since then, thousands of developers have been manipulating and managing MSG files, it is well known and can be considered a de facto standard. Individual MSG files are a suitable format for email and email meta-data, but billions of immutable MSG files create a performance and storage challenge for all systems.

To address the issue of backing up millions of individual MSG files, LookingGlass incorporates standard zip file support in to the storing of its archived email. The ZIP file format, made popular by Phil Katz (d. 2000) was released into the open format in 1989. Despite Mr. Katz death, the open format creates assurances the data type will continue to be supported within LookingGlass well in to the future.

LookingGlass data storage access is predicated on open access and data ownership by its customers. As citizens of the Republic, we should pledge allegiance to open formats that ensure future generations' access to the present history that defines their conditions. LookingGlass provides assurances to that access through the confluence of Spherical Indexing, MSG file's and ZIP technology. LookingGlass will provide our Federal, State and Local Governments the open access they need to support the open records acts well into the future.

Data Recovery
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 
 • 

» More on Data Recovery
 



Share this article :
Click to see more related articles