It's official: government research shows technology improves the public disclosure process

There’s been a concerted effort in recent years to improve the processes federal government agencies use to respond to Freedom of Information Act (FOIA) records requests. Government agencies such as the National Archives & Records Administration (NARA) and the Department of Justice have researched the use of advanced technologies to alleviate the challenges involved with collecting, searching, reviewing and producing responsive information. They’ve also recommended ways agencies can improve records management processes, which ultimately increase their ability to respond more effectively to FOIA requests. 

The white paper the Quest for a Transparent Government: 50 years of FOIA and Public Records Acts, was written with this same goal in mind: helping agencies consistently respond more quickly, thoroughly, and accurately to Freedom of Information requests.  

In a review of FOIA’s evolution, the white paper discusses challenges introduced and compounded over the years that now require the use of technological tools to overcome. We know that agencies need more advanced technologies to help them respond to FOIA requests because the government’s own research came to the same conclusion.  

Technology Reduces Search & Review Times   

The Office of Information Policy got together with the Department of Justice to conduct a digital-FOIA pilot program in 2013. The program evaluated the use of digital tools as compared to conventional methods to process FOIA requests.  

The “digital approach” used automated and electronic procedures to process all aspects of a request, including search, collection, de-duplication, and initial review. “Conventional” methods meant employees conducted manual searches and sorted through records by hand.  

The Department of Justice said it was most impressed with the speed and accuracy with which the digital approach collected, searched, and de-duplicated records. Results showed: 

• A decrease in the number of times that a search term had to be run. With conventional methods, each search term is individually run for each custodian. In a digital approach, records of multiple custodians could be searched with multiple terms at one time. 
• Improved search times. The digital approach completed a search for records in less than an hour. Conventional methods required multiple work days to complete the search. 
• Decreased time spent on de-duplication. Digital tools drastically reduced the time required to remove duplicate copies of documents from a review set. For the cases using the digital approach, 4.7 million documents were initially collected. Automatic de-duplication removed 1.1 million documents from the set. Once date ranges and search terms were applied, 3,859 documents remained to be reviewed. 

 The numbers speak for themselves. These results clearly demonstrate technology’s superior ability to speed up FOIA response processes.   

Automation & Machine Learning Reduces Burdens 

Their findings echo those of NARA, which was tasked with producing a comprehensive plan for the automated management of email, social media, and other types of digital records, including advanced search techniques.  

NARA found that the greatest recordkeeping burden for employees is appropriately capturing and categorizing records. (Capturing refers to bringing a record under control, whether onsite or in the cloud.) NARA said: 

• The processes and tools agencies currently use to manage electronic records are not adequate to support consistent compliance with the Federal Records Act.  
• Most agencies rely on individual staff to capture and categorize their electronic records - IF they manage electronic records at all. 
• Manual processes will not scale to manage the sheer volume of email, social media, and other electronic records being created. 

NARA looked at the ability of automation and machine learning to categorize emails and records. This approach, NARA said, “has the potential to categorize records from unstructured business processes, including email, with a high degree of sophistication.”  

This level of sophistication is difficult to achieve with other processes, NARA said. The ability to locate and categorize emails and records based on their content rather than sender name or job role is key to performing more optimized searches. NARA pointed out that effectively using automation to categorize records produces three positive effects:  

1. Records are more consistently captured and managed;  
2. Processes can scale up to handle a higher volume of information; and  
3. Staff members have more time available for agency missions.  

Read the White Paper 

The government’s own research shows that the use of digital tools significantly speeds up document review times and helps agency employees effectively and more consistently manage growing amounts of electronic files. For more information on how these fantastic advancements support agency personnel while improving FOIA processes, read the white paper.