Do freedom of information laws increase transparency of government? A replication of a field experiment

Transparency and responsiveness are core values of democratic governments, yet do Freedom of Information Laws - one of the legal basis for such values - actually help to increase these values? This paper reports a replication of a field experiment testing for the responsiveness of public authorities by Worthy et al (2016) in the United Kingdom. We sent 390 information requests to Dutch local government bodies, half of which were framed as official FOIA requests, the other half as informal requests for information. We were able to reproduce the original findings, that is, we found a positive effect of FOIA requests on responsiveness. The overall response rate of local governments was much higher (76%) and the size of the effect was larger than in the original experiment. Furthermore, the strongest effect of FOI was found on proactive disclosure (concordance), something that governments - strictly speaking - are not obliged to do according to the Dutch FOIA. Implications for future replication studies are discussed.

ransparency has become a core principle of good governance (Hood & Heald, 2006;Fenster, 2015) and is expected to generate positive outcomes, such as greater accountability, increased trust, and less corruption (Grimmelikhuijsen & Meijer, 2014). One important potential mechanism to ensure governments become more transparent is Freedom of Information (FOI) legislation, and there are now an estimated one hundred such laws or regulations in force across the world (freedominfo.org, 2018). But how effective is FOI in establishing greater responsiveness and, eventually, transparency? How much better are such laws than more informal, non-legal routes? FOI laws themselves are increasingly being used as a device for experimental research to test for their potential impact on compliance or 'concordance' with the 'spirit' of the law (Grimmelikhuijsen & Meijer, 2014: Richter & Wilson, 2013. First, various experiments have been used to probe how the identity of the sender of a FOI request affects whether and how governments respond (Lagunes, 2006;Michener & Rodriguez, 2015). For instance, Lagunes & Pocasangre (2017) used Mexico's FOI Act to see if implied connections to certain influential families had an effect on responses but found little evidence of partiality. In contrast, Grohs & Knill (2017) used a series of requests to assess gender and ethnic bias in German local government, finding evidence of gender discrimination for Turkish women, but not in general.
Secondly, FOI experiments have investigated whether the content of the request influences responsiveness (e.g. Cuillier, 2010;Ben-Aaron et al., 2017;Spac, Vodab & Zagrapanc, 2018). In the US, Cuillier (2010) experimented with the tone of requests and found that a more adversarial tone led to more responsiveness. In a Slovakian FOI experiment, Spac, Vodab & Zagrapanc (2018) found that a 'moral appeal' had no effect but framing a request as an 'official' FOI request did improve the chances of a response.
Overall, these studies have given us insight into how and whether laws function in terms of responsiveness and potential bias at the micro-level. They have also illustrated some of the limitations, as most of the experiments were not compared with other routes of access, and they have focused only on whether there was a reaction and less on the quality or effect of the information on a broader public. In a recent field experiment with local government in England (U.K.), Worthy, John & Vannoni (2017) found that indeed FOI requests are more effective than asks in terms of the quantity and quality of responses. Their experiment was conducted on a sample of 4,300 parish councils, the lowest local level in England, which were requested to provide the organizational chart of the parish council. Furthermore, authorities were asked to go beyond what is legally required and publish the information publicly, allowing the researchers to assess whether FOI leads to greater pro-active openness (transparency). Overall, the Worthy et al. (2017) field experiment is the first to show that FOI requests-as opposed to informal asks for information-not only increase responsiveness to the individual requester but engender transparency for the broader public.
In this article, we present the results of a close replication of the original field experiment by Worthy et al. (2017) in the Netherlands. Empirical replications are important to build confidence in existing theories and to identify boundary conditions of causal effects. The attention to replication studies in public administration and management research is still very limited, although there has been a recent symposium on replication (Walker, James & Brewer, 2017) and a framework to evaluate such research (Pedersen & Stritch, 2018).
A more specific reason for replication is that the Worthy et al. (2017) experiment was only conducted in the UK (specifically England) and we investigate whether these findings extend to a different political and administrative context (the Netherlands). We use the term "close replication" as an "exact replication" is impossible to achieve (Brandt et al., 2014). This means that we stay as close to the original experimental treatment and procedure as possible. We will use this close replication to assess the external validity and boundary conditions of Worthy et al.'s findings by investigating FOI in a different context (Jilke et al., 2016).
The Netherlands is a good case to assess the external validity of the original findings, as it shares some key characteristics with the UK but at the same time differs in some important respects. First, the Netherlands has a much longer formal tradition with FOI legislation (1980 in the Netherlands compared to 2005 in the UK) and second, the Netherlands has a less adversarial media or political culture willing to turn FOI into a political weapon (Hallin & Mancini, 2004;Grimmelikhuijsen & Kasymova, 2015). Third, the replication inevitably focuses on a different type of local government: the municipal instead of parish level. The Netherlands does not have the latter's lower, primarily village or town level of government. Municipalities operate on a larger scale than parish councils and are therefore expected to be accustomed to larger volumes of FOI requests (with, for example, a larger professional staff trained to deal with them and set procedures). The goal of this replication study, therefore, is to see if the results of the original experiment still hold in spite of these differences in institutional and administrative context. Thus, the central research question is: To what extent can Freedom of Information requests improve transparency of local governments in the Netherlands?

Freedom of Information and Transparency
Freedom of Information (FOI) laws are the legal backbone for creating and safeguarding a basic level transparency. A FOI-type law as has been adopted by most countries across the globe (Relly & Sabharwal, 2009). We define 'transparency' as 'the availability of information about an organization or actor allowing external actors to monitor the internal workings or performance of that organization' (Grimmelikhuijsen & Meijer 2014, p. 139). This definition is generally in line with how transparency is defined in the literature. A systematic literature review on 187 studies on transparency by Cucciniello, Porumbescu & Grimmelikhuijsen (2017) reveals that most definitions address some core components, such the availability of information to 'outsiders', who use openness mechanisms to gain an insight into decision-making processes, operations, budgets or performance of a governmental body.
Most FOI audit studies only measure transparency in a more general sense of responsiveness to an outsider. Some experiments measure simple compliance with the basic right to information enshrined in a FOI law, testing the operation of the system by examining if various public bodies make information available to the requester (e.g. Spac et al., 2018;Cuillier, 2010). The FOI requester is an 'outsider'; yet measuring responsiveness only takes us so far. It is unclear whether a response-without qualifying its content-truly provides an 'insight' into a government organization or if the information was already available by informal routes.
Providing information directly to a single requester is a rather narrow view of transparency. Generally, transparency refers to broader public and pro-active access to information. This can provide insight into how effective the chain of request to response is and offer a snapshot of the strength of openness culture within an organization. So far only the Worthy et al. (2017) experiment included both of these critical elements of transparency: measuring public access via the law as against a non-legal ask, and also measuring the quality of the response to see if the organization goes further than the law requires.
This important difference is captured in the possible responses of governments to FOI requests that Worthy et al. (2017) distinguish: compliance (doing as the law asks to varying degrees) and concordance (behaving in ways that move beyond the law and doing more than it asks, see Richter & Wilson, 2013). Such a distinction resonates with an often-made distinction in the literature between 'passive' and 'active' transparency (Grimmelikhuijsen & Meijer, 2014). Passive transparency aligns with legal compliance: sending information but doing nothing further to disclose information to other people who might be interested. In contrast, active transparency is the simultaneous proactive disclosure of information on websites or other information carriers so that anyone interested can access the information.
Thus, the aim of this study is to probe if FOI fosters active transparency, not just responsiveness or passive transparency. We do this by replicating the Worthy et al. experiment in the Dutch context. Despite some obvious contextual differences between both countries, we have no a priori reason to believe the outcomes will be different. We therefore adopt the two main hypotheses from the original study without modifying them.
Hypothesis 1 is the main hypothesis and is supported by the findings of the UK field experiment. Parish councils were more likely to respond if they were sent a FOI request, as compared to an informal request or ask.
H1: Local authorities will be more responsive to a FOI request than to an informal ask.
Hypothesis 2 is based on the notion that larger authorities also have more capacity to process requests and respond to them in time. This hypothesis is in line with Grimmelikhuijsen & Welch's (2012) finding that organizational capacity partly explains differences in the transparency of local governments. H2 was not supported by the findings of the UK field experiment. Nevertheless, we decided to include this hypothesis to assess if the size of the local authority mattered for responsiveness in the Netherlands, given their larger size.
H2: Local authorities with larger populations will be more responsive than smaller authorities. 1

Experimental setting: Freedom of Information in the Netherlands
As with the earlier UK experiment, we asked municipalities to provide us with information (in the form of an organization chart). Authorities are obliged to give these under the Dutch FOIA (so meeting the requirement for passive disclosure/compliance). However, we also asked them to publish the same information on their website 'in the spirit of the Dutch FOIA' (matching the higher proactive disclosure/concordance requirement).
As with the UK law and elsewhere, the Dutch legislation highlights, through a statutory duty, that public bodies must provide the information to whomever asks for it, and requesters do not have to provide arguments for why they want access to the information. A government organization can deny the disclosure of information based on a limited set of arguments or exemptions summed up in the Dutch FOIA (Wet Openbaarheid van Bestuur). The government organization must then answer the request within a period of four weeks. This period can be extended, but the requester must be informed about this.
Finally, according to Dutch FOIA, in principle any information request can be processed as an official FOI request, including informal requests or asks. Consequently, we cannot exclude the possibility that some informal requests have been classified as official requests. However, that does not affect our general expectation, because specifically mentioning legal provisions is expected to be a stronger signal to municipalities to process the request and to respond.

Design and Ethics
Similar to the original UK experiment, we sent our request on behalf of an NGO called 'De Open Gemeente' [The Open Municipality], a neutral campaign group seeking municipal openness, created through the use of an organization website. Those receiving the request would, we presumed, be likely to be somewhat sympathetic to its aims. This is important to note since different 'types' of requesters provoke varying responses (Worthy, 2013). Moreover, studies simply making requests as named academics have suffered low response rates (Cuillier, 2010).
Our aim was therefore to be as transparent as possible while not endangering the experiment with low response rates. Drawing attention to the fact we are academics 'doing an experiment' would be more likely to lead to either non-compliance or behavior influenced by our position and work (Cuillier, 2010). So the request stated that we were 'part of a research project examining the impact of FOI on openness in local government' (see letter in Appendix A). We ensured transparency by doing the following:  The name of the lead researcher was at the end of the emailed FOI/request with a link to our websites and previous research.  The names of the research team were mentioned on the website with links to our respective staff pages.  We provided contact details and also asked in the request for respondents to contact us should they wish to know more.
 The project summary was also on the website as were details of the experiment after the research was conducted.
Therefore, the NGO approach was chosen to obtain maximum responses and ensure the experiment works. We made it explicit that there is a link between the underpinning research and an NGOstyle organization. Indeed, as researchers we also supported the wider objectives expressed by the organization (making municipalities more open). We believe this approach minimized the risk of non-compliance while being as transparent as possible. Our design was approved by the Ethics Board of the Faculty of Law, Economics and Governance on August 28, 2017, internal reference number 17-002.

Preregistration
Preregistration is generally seen as a good way to increase the credibility of published results (Nosek & Lakens, 2014). We registered our design on October 13, 2017, which was after the data collection started (September 25, 2017), but well before data collection had ended and analysis had begun. The design was registered with the EGAP (Evidence in Governance and Politics) Design Registrations database. The design ID number is 20171012AA (http://egap.org/registration/2883). The preregistered design included a third hypothesis that could only be tested with a follow-up experiment. As of yet, we have not carried out the follow up experiment and thus have not included it in this article.

Sample
We estimated that we needed a sample size of 318 (159 per group) at 80 percent power with an alpha of .05 (two-sided) based on obtaining the same effect size as the Worthy et al. study (an 11.1 percentage point difference). The total size of the municipalities of 390 (in 2017) was thus a sufficient sample. We sampled all municipalities in the Netherlands, which means that we had enough statistical power to give the original study a fair chance of being successfully replicated (Asendorpf et al., 2013). One request was returned to sender by the postal office, which meant that our final sample consisted of 389 municipalities.

Use of Materials
Journal of Behavioral Public Administration,1(2) Although we tried to stay as close as possible to the original materials and procedures, we had to make various adaptions to make the requests fit the Dutch context (see Appendix A for the letter in Dutch and Appendix B for the English translation). First, we had to use regular mail to make the requests instead of e-mail because most municipalities prohibit FOI requests via email. In our written letter we asked for an electronic response to increase the similarity with the original experiment.
The letters were kept as close as possible to the original materials (cf. Brandt et al., 2014). We made one significant change to the control letter to adapt it to the Dutch context. The original control letter in the Worthy et al. (2016) stated 'Please note that this is not an FOI request'. We removed this phrase because according to Dutch law in principle any information request can be processed as an official FOI request, so also the informal request. We were afraid that adding a sentence that this was not an FOI request would actually confuse participants and even prime them in the opposite direction: thinking of the information request as a FOI request because of the mere mention of it.

Measures and Analysis
The outcome variable was pre-registered and consisted of the five ordinal categories (shown below). First of all, we analyzed the difference in responses between the two treatments and control groups Secondly, we also took into consideration the control variables mentioned above by carrying out an ordinal outcome variable regression analysis, which takes into account the ordered rather than interval nature of the response variable and also follows the Worthy et al approach. Note that the coding of the dependent variable is more sophisticated than merely coding for whether or not a response was received. Furthermore, the coding also captures the idea of 'active transparency' (code 4, concordance). Here are the five categories: 0: No reply. Municipality does not respond or only an automated delivery receipt [no compliance]. 1: Reply, but does not provide the organization chart or any detailed information [lesser compliance]. 2: Link or background. A direct link or specific direction how to find the chart only are provided, yet the chart is not attached to the response as requested [partial compliance].
3: Send information. The chart is sent as an attachment, yet the chart is not made public on a website [full compliance]. 4: Make information public. Information was sent and explicitly stated the intent to publish it in the public domain as a result of our request [concordance].
We obtained a list with the names of all municipalities in the Netherlands and randomly assigned each municipality to either control (informal ask) or treatment (legal request). The website random.org was used to carry out the random assignment.
The results are balanced between treatment and control with no significant difference in the following covariates: number of residents, presence of FOI information on website, income, presence of information about previous FOI requests, presence of information about local council meeting minutes and pre-treatment. A regression of the treatment allocation on these covariates shows none of them to be statistically significant. This regression is reproduced in Appendix C, Table CA.

Results
The overall response rate of our study was high: 76.9 percent of all municipalities responded to our request. When we look at the basic measure for FOI effectiveness, the five-point compliance scale, we find some clear differences between the control and treatment group (Table 1).
We find that 89.2 percent of the treated municipalities provide a response, whereas only 64.6 percent of the municipalities in the control group responded; when cross-tabulated against all other responses, this is a significant difference (χ 2 =32.99, p<.001). All significance tests were twotailed. Interestingly, there was no significant difference for lesser compliance and full compliance (for lesser compliance χ 2 = 0.40, p=.527, for full compliance χ 2 =.011, p=.916). However, treated municipalities were more likely to at least partially comply by sending a link to a chart (χ 2 =4.08, p=.043) and to act 'in the spirit of the law', in other words publishing a chart on their website and sending the chart as an attachment (concordance). The difference in the frequencies for concordance in the control and treated groups, when compared to all other responses, is statistically different (χ 2 =16.41, p. <.001). Overall, we find support for the first hypothesis: a FOI request leads to overall higher response rates. In addition, there is a strong effect on concordance as with Worthy et al.'s results, which are reproduced in Table 1.
To assess whether a formal FOI request leads to increased transparency (concordance) we carried out an auxiliary analysis in which we include only those municipalities that were not 'pretreated', in other words, who did not have an organizational chart online at the time of the request. It should be noted that this analysis was not preregistered and thus should be considered exploratory. Fully 50.3 percent of the municipalities had no chart online prior to the experiment. Within this subsample, we find that only 12.3 percent of the municipalities in the control group published the chart online while 32.6 percent in the treatment group did so, which is a highly significant difference (χ 2 =11.82, p =.001). This result corroborates our hypothesis that FOI requests not only increase responsiveness, but more importantly also have the potential to improve transparency to the public at large.

Regression Analysis
To replicate Worthy et al., we carried out regression analysis, reported in Table 2, which also allows for a test of Hypothesis 2, introduces covariates, and controls for the effect of already being treated, which in this case means that the requested information (organizational chart) was already disclosed on a website prior to our request. We use an ordered probit estimator as in the original study. We also produce ordinary least squares (OLS) estimates, reported in Appendix D Table D1, for comparison, though the interpretation of these models is essentially the same irrespective of estimator used. Appendix D also contains probit regressions carried out on each item in the response variable (Tables D2-D5), which shows that only the 'Make Public' outcome, compared to all other response options, is significant when regressed with covariates. Model 1 reports the impact of the treatment alone, as in bivariate analysis. As expected from the descriptive results, there is a large main effect: the coefficient on the treatment represents a factor change in odds of 0.3149 for a unit increase in X (standardised at 0.3003). Model 2 adjusts for the covariates of size, average income of residents, whether there is attention to FOI on the website (e.g. how to do it, what it is), whether there is a repository of recent requests, whether transcripts of council minutes are accessible online, and whether the council has been pre-treated (see measurement above). Please note that the number of cases differs because we had missing observations on some of the covariates.
Only the pre-treatment of the municipality increased the probability of response and the quality of the response, while there remains a treatment effect overall. Model 3 tests whether an interaction of size and the treatment is significant which it is not. There is no variation in response to the treatment by council size, so no support for hypothesis 2. Model 4 tests whether the treatment is affected by the pre-treatment variable, which it is not. Appendix E contains conditional marginal plots for these interactions that show the slopes of the margins for the interactions are as expected (e.g. Figure  E3 for residents), but there is nothing statistically significant to report.

Conclusion: Effect is Replicated, Effect Size is Not
Overall, we find support for H1 (FOI increases transparency) and not for H2 (interaction between size and responsiveness), both of which are in line with the original UK experiment. Also, the strongest effect seems to be on concordance, which is also in line with the original experiment. To compare whether the effect size was replicated, we used the F-statistic of both studies and calculated Cohen's D. The original study (F=109.51, ncontrol=2160 ntreat-ment=2152) had an effect size of D=0.319 ). This replication study has an effect size of D=0.586 (CI95=.383-.789). These are significantly different effect sizes. This means that, although the effect itself is replicated, the effect size is not. There are a variety of potential explanations for this difference in effect size. One contextual variation that might have caused this difference concerns the nature of local jurisdictions. Compared to the original experiment by Worthy et al. (2017), our experiment focused on a more aggregated local and probably more professionalized level of government. Municipalities in the Netherlands are relatively large with a median population of 43,524 residents, compared to a population average of less than 10,000 for parish councils in the UK. The high response rate and effect size could be because, as outlined above, the larger municipalities in the Netherlands may be better equipped to handle requests than the smaller and more local parish councils in the UK, which are mostly run by a part-time clerk. This may have meant that the requests were less likely to be lost, ignored or overlooked in the Netherlands, with more resources and systems in places to deal with them.
A second contextual variation regards the institutional environment. The Netherlands has a less adversarial media and political culture than Anglo-Saxon countries such as the UK. In more adversarial media cultures, FOI can more easily turn into a political weapon that can make local authorities more fearful of releasing information (Grimmelikhuijsen & Kasymova, 2015). Likewise, different 'transparency regimes' might affect responsiveness. Ruijer & Meijer (2016) found that pre-existing institutional differences affect how transparency is approached. In the United States a 'rule-based' regime has emerged, focusing on strict compliance with set procedures. In contrast, the Netherlands has a 'principles-based' transparency regime, focusing less on strict compliance and more with acting 'in the spirit of the law'. We imagine that such institutional differences may also occur between the Netherlands and the UK, resulting in a more responsive approach to FOI requests in the Netherlands.

Directions for a Systematic Replication Agenda on FOI Effectiveness
In order to explain the difference in effect size, more research is needed. This highlights the strength but also the weakness of a single replication study in public administration. The strength is that the validity of findings can be confirmed by replicating an effect in a different political and administrative context. The weakness is that the potential contextual differences-especially in field experiments-are myriad and, if the outcomes of such a replication differ, it is impossible to tell what contextual differences have resulted in a different outcome. In the end, a more systematic replication agenda could solve this issue by building a coherent body of experimental knowledge that can be included in meta-analyses, which can identify potential moderators. We have the following recommendations for such an agenda. First, close replications are needed but in political contexts that differ more widely. Although this replication was carried out in a different context, the political cultures carry broad similarities. Both are Western European countries with strong cultural and economic ties and, to an extent, shared histo-ries. Both the Netherlands and the UK, for example, have a long history of openness and democracy with, at the local level, formalized access to meetings and records going back decades. Are the findings valid in contexts that are more strongly divergent? The Slovakian case would offer one example of a successful experiment in a slightly different environment (Spac et al., 2018). A suggestion for future replications would therefore be to carry out this experiment in other countries that have much less experience with democracy, openness and Freedom of Information regulations. Replicating this experiment in, for instance, a new democracy, a hybrid regime or semi-authoritarian regime could be a 'stress test' for the outer bounds of the external validity of our findings.
Second, we recommend carrying out conceptual replications with different types of requests. Although we now know that FOI helps to increase responsiveness and quality of responses in the UK and the Netherlands for a relatively simple information request, we do not know if or how FOI works with more politically sensitive or problematic requests. One could think of testing requests for spending declarations of politicians, ministerial diaries or certain types of policy analysis. Overall, one would expect governments to be more hesitant to release this kind of information, yet the effectiveness of FOI on disclosing this kind of information is relatively unknown. A further possibility is that other bodies could be requested: elsewhere FOI has been used to examine public institutions as diverse as schools and police departments.
We acknowledge that experimenting with such kinds of requests involves risks as well: first, there is a higher burden for governments to collect and disclose such information, which raises ethical questions about the potential costs of the experiment for 'participants', though this is mitigated by asking for information that is clearly of public and democratic value. Such requests will have to be carefully chosen and crafted as they may also be more likely to come up against legal exemptions.
Overall, the direction of the effect was reproduced in our study, which gives more confidence in the external validity of the initial UK findings (Brandt et al., 2014;Walker et al., 2017). We replicated the increase in response rate and transparency (concordance), yet the magnitude of the effect of FOI requests seems to be more sensitive to contextual varieties.