In 2010, when we started MuckRock, a non-profit website that helps newsrooms and journalists around the country file and track public records requests, one of our big bets was that if you filed a lot of FOIA requests, you’d get better at it. And if you carefully organized and archived all the data around what worked and what didn’t, computers could help you get even better, uncovering insights into the process that humans might miss.
In the abstract, the idea — practice makes perfect — is not particularly novel, but nobody had ever applied the concept to FOIA, as far as we could tell, or at least not on the scale we were envisioning. Most FOIA studies were composed of a dozen, or a few dozen, requests.
In our first year, we helped file several hundred records requests. Today, we’ve helped our users file more than 36,000 requests, including several hundred in almost every state.
Because we carefully track every step of the process, including the date and method used to file the request as well as the language of the original request, and information on every response from an agency, we have an enormous amount of information to work with. At a FOIA hackathon last summer, a full export of our database was about 112 gigabytes, and it’s roughly doubled since then.
But putting that information to work on behalf of requesters can be surprisingly challenging. It turns out that a lot of the skills used by FOIA and public records veterans for years often have nuance and common sense — and perhaps most importantly, sources — that Siri and other artificial intelligence efforts haven’t caught up to.
Our first major effort was simply trying to build predictive models to determine whether a given response to a request from an agency completed a request, rejected it, asked for more information, etc.
Over the years we’ve found that most responses from agencies fall into one of about eight different buckets. We’ve hand coded thousands of these responses. About two years ago, my co-founder Mitchell Kotler used Scikit-learn, an open-source machine learning library written in Python, to teach computers to do the tedious categorization work that our staff had been doing. As we scaled up from helping our users track hundreds of requests to thousands and then tens of thousands, this approach seemed like the only avenue forward.
In practice, it works something like a spam filter: We’ve given it tens of thousands of examples of agency responses and taught it what to look for. It’s important to know that it’s “looking” in only a very rudimentary sense: Despite being trained on hundreds of thousands of examples, the computer still has no real understanding of what a FOIA request actually is, nor does it even really comprehend language.
Instead, it’s taught to look at a variety of factors and figure out correlations between those factors and the ultimate status. In our cases, those factors included how many pages were attached to the email (40 pages are generally more likely to indicate a successful request versus one page) or the frequency of certain words in proximity to each other (“no responsive records,” for example).
Fortunately for us, the system largely worked. We can now automatically predict, with about 80 percent accuracy, what an incoming communication from an agency is trying to say.
It was also a good illustration for how complex public records requests are when put in practice. While we like to think of it as a fairly cut-and-dried legal process, there’s a lot of nuance: What if an officer provides some documents, but wants you to call and talk about another part of your request? How do you categorize a response if they respond with a cease and desist? Or if they helpfully forward your request on to another department for a final response?
Just like the stories our users report, there are almost infinite shades of gray. Even as we continue improving our automation, many of the responses we get stump me.
We also wanted the requester community to benefit more from what we’ve learned. We’ve put together guides to every state’s public records laws that includes information on how the process works and your options if you get rejected, as well as detailed statistics on average response times, rejection rates and fees. But we knew we could do more with the database we had built up.
We were delighted when Data.World, an Austin-based startup, wanted to help.
Using our API, we pulled out all the information from our requests and put it in a machine-readable format, and the Data.World team analyzed it, building an open-source FOIA predictor application. The application was simple to use: Just paste in your request, pick an agency and get feedback tailored to your request.
Their model was significantly more complex, looking at more factors than the one we used, such as the number of proper nouns, the inclusion of links, sentence length and looking for key elements, such as the mention of fees.
The Data.World team was also able to pick out trends that we had missed: Over the last four years, they found, responses with structured data increased by 417 percent.
There were a few broad strategies they were able to pick out as being generally successful.
According to their analysis, requests mentioning fees were less likely to succeed, while shorter requests tend to get better results.
A lot of what we found was backed up by what we’d seen anecdotally. But state and local trends, such as the increase in responses with structured data, were ones that had flown under the radar.
But their analysis also came to the conclusion that agencies with good track records are more likely to release more records in the future. That generally outweighed other factors, such as how the request was written.
The app was also a good lesson in usability and user expectations.
In machine learning, you often set a segment of data and then test how well your model works against it. The Data.World team found that their predictor was about 80 accurate in guessing whether information would be released.
But users quickly found that they could game the model, or else take advantage of the fact that good agencies were generally likely to release information even if the request wasn’t well written. It wasn’t hard to write a nonsense request that scored well in the system, and people were concerned that well-written requests that scored poorly could discourage requesters.
We discussed the results with the Data.World team and talked through how maybe a better approach is to avoid giving a request a “score.” Instead, we proposed trying to create a virtual request coach that offered concrete suggestions based on what it saw.
While having a single success percentage is deeply satisfying, it was often misleading and not particularly helpful. If you instead took a look at the factors and offered feedback on how to improve the overall request — shorten it here, add a link there — you could maintain the almost instant feedback while not discouraging requesters from filing requests that could very well uncover information in the public interest, even if those are long shots.
Our conclusion was that while algorithms can improve FOIA and public records requests, the request process shouldn’t be algorithmic. There’s a human on the other end, and just like with any other part of the reporting process, judgment is essential in when you should offer carrot and when you should offer stick. Data.World has opened-sourced all their work, and we’re looking into incorporating similar FOIA coaching into our own request system.
Ultimately, big data can help FOIA be more effective, but algorithms won’t be replacing reporters any time soon — just empowering them to do their work better.
Michael Morisy is co-founder of MuckRock, a news and information outlet focusing on access to public records. He was previously a Boston Globe editor and 2014-15 Knight Fellow at Stanford University. Email: @morisy.com On Twitter: @Morisy.
Tagged under: FOI
