Speaking truth in power : Scientific evidence as motivation for policy activism

n the United States, federal policies are most commonly enacted by unelected bureaucratic officials through regulation, not legislation (Kerwin, 1994). Absent direct electoral accountability, administrative policymakers often invoke their domain expertise and the technical integrity of scientific data on which regulations are based to establish the perception, among attentive constituencies, of legitimacy of the policymaking process (Weingart, 1999; Bäckstrand, 2004; Weible, 2008; Kinney et al., 2010). In this manner, science can be used as a mechanism to justify policies and garner support from attentive and influential stakeholders such as political elites, interest groups, media outlets, and the general public. While reliable science has been integral to the production of effective policies for decades, it was not until President Reagan signed Executive Order 12291 in 1981 that US federal regulatory agencies became legally required to provide scientific, technical, or economic information with significant regulatory policy proposals (Hahn, Lutter, & Viscusi, 2000). Executive Order 12866, issued by President Clinton in 1993, further requires that each agency “base its decisions on the best reasonably obtainable scientific, technical, economic, and other information concerning the need for, and consequences of, the intended regulation.” In describing the role of regulatory impact analysis (RIA) in policymaking, Radaelli (2005, p. 934) explains that, “The legitimacy of the regulatory process is not based on parliamentary control over the government, but on the credibility of executive agencies.” Regulators recognize the importance of scientific evidence in the credibility of RIA. I S D F I Abstract: Unelected administrative policymakers rely on the domain expertise and technical integrity of scientific information to maintain perceptions of legitimacy. The necessity that regulatory policymakers rely on sound scientific evidence has been formalized at the US federal level through executive order. Yet, the practical impact of scientific evidence on public support and mobilization for policies remains unclear. We investigate whether individual policy activists are more likely to participate in regulatory policymaking when a policy recommendation is substantiated by scientific evidence. We investigate how two separate groups within the public—policy advocates and policy experts—may be affected differentially by scientific evidence. In collaboration with a nationally active policy advocacy group, we conducted a randomized messaging experiment in which members of the group’s e-mail list are sent one of three versions of a policy advocacy message. Results indicate that reference to evidence published in peer reviewed scientific sources increased activism by roughly 1 percentage point among general activists, and decreased activism by 4-5 percentage points among scientific experts.

find extensive use of peer reviewed scientific research in RIAs produced by several agencies across the US federal government.This is not to say that regulators or other regulatory policy actors draw upon science solely for its evidentiary value.Indeed, science use is a political tool, and regulators may draw upon it to advance selected policy objectives (Wagner, 1995;Carrigan & Shapiro, 2017).Regulators' reliance on technical credentials in maintaining a perception of legitimacy of bureaucratic policymaking is well understood.However, despite a voluminous body of research in public administration on the general causes and consequences of advocacy for regulatory policy (e.g., Eckerd, 2014;Jewell & Bero, 2006;Yackee, 2013), there has been little scholarship regarding whether the presentation of scientific evidence affects support of and advocacy for regulatory policy.
Existing research is mixed regarding the degree to which scientific evidence has a persuasive effect on the mass public.Costa, Desmarais, & Hird, (2016) show that the volume of public comments submitted on proposed federal rules is positively associated with the number of scientific citations in the proposed regulation's RIA, but they cannot draw causal conclusions.The public faces considerable uncertainty in evaluating and weighing the scientific justifications presented by policymakers, so it is unclear whether reference to science can serve to persuade the public to support a regulatory policy proposal.A body of recent research indicates that individuals exhibit complex and unpredictable opinion responses when presented with scientific evidence for a policy (e.g., Ding, Maibach, Zhao, Roser-Renouf, & Leiserowitz, 2011;McCright, Dunlap, & Xiao, 2013;Jamieson & Hardy, 2014;Dixon, 2016).Not only does this scholarship challenge the degree to which references to science motivate activism for the general public, it suggests that the influence of scientific research may vary between different groups within the public.
To address this gap, we study the effects of references to independent scientific research on public mobilization for a policy amongst two groups: one group of technical or scientific experts, and one group of policy activists.We conducted a field experiment in coordination with a major policy advocacy organization.The treatment conditions in the experiment were designed as different versions of an activism encouragement email, which was sent by the organization to two different member lists---an advocacy list, and an experts list.In brief, we found that reference to the scientific journal in which supporting evidence was published increased the rate at which members of the advocacy list followed the email's activism prompt but decreased the rate at which members of the experts list followed the email's activism prompt.

Challenges to the Persuasive Efficacy of Scientific Evidence
Effective policy analysis must enable policymakers and citizens to make informed policy decisions.In an increasingly technological society, effective policy analysis often requires the inclusion of scientific information (Brossard, Lewenstein, & Bonney, 2005;McClellan, 2012).Science is expected to establish a basis for effective public policy decisions by satisfying one or more of the following functions: identifying problems, measuring the magnitude and seriousness of problems, reviewing alternative policy options, systematically assessing consequences for each policy option, and evaluating the results of policies (Prewitt, Schwandt, & Straf, 2012).Based on this framework, science has the potential to not only inform debate and critical reflection, but also to guide the public and policymakers in improving policies and addressing global challenges.The efficacy of scientific evidence in the policy debate depends, however, on attentive citizens' reaction to scientific evidence.Recent research on scientific literacy presents potential challenges regarding the effectiveness of scientific arguments as the basis for policy justifications.Citizens' understanding of scientific concepts and processes is generally insufficient (Besley & Tanner, 2011;Zia & Todd, 2010).As a broad designation, "understanding" ranges from a superficial or mechanistic comprehension to a nuanced and conceptual comprehension.In scientific communication, to claim an understanding of scientific information the public must only possess a depth of comprehension that permits participation in public policy discussions of issues (Miller, 2004).This criterion of sufficiency has been operationalized as the capability of reading and comprehending the science section of The New York Times, which results in scientific literacy rates of approximately 20% among Americans (Ding et al., 2011).
According to the literature on scientific literacy, most people are unable or unwilling to understand science at the level required to participate in technical public policy discussions (Roberts, 2004).Individuals tend to employ the least amount of effort necessary to make judgments or decisions (Brossard & Nisbet, 2006;Chaiken, 1980;Ho, Brossard, & Scheufele, 2008).To compensate for a lack of understanding about a policy subject, a low information public frequently relies on heuristics: cognitive shortcuts intended to guide decision-making.
Through heuristics, citizens are able to follow and engage in debates regarding even complex policy alternatives without a full understanding of the relevant scientific evidence (Ho et al., 2010).
Experimental research investigating the effects of heuristics on the public's understanding of scientific information in policymaking has yielded important insights.
For example, Druckman & Bolsen (2011) demonstrate that heuristic factors play a larger role in determining opinions than factual information.In a related finding, Tal & Wansink (2016) show in a series of experiments that "Trivial" (i.e., uninformative) references to scientific information increase the persuasiveness of product advertisements.
There is good reason to expect both nonexperts and technical experts to utilize heuristics in assessing the validity of scientific evidence.In both cases, heuristic clues such as the credentials of researchers, where the work was conducted and published, and how other experts respond to the research, may all be used to form an opinion on scientific evidence (Priest, 2013).These heuristics are useful even to experienced scientists in evaluating the merit of any scientific study.
Still, experts and non-experts are likely to diverge in their amenability to the persuasiveness of scientific evidence.In general, educational information is known to have a greater effect on individuals with lower levels of information in the first place---a phenomenon referred to as the "expertise reversal effect" (Kalyuga, Ayres, Chandler, & Sweller, 2003).Though the expertise reversal effect has mostly been applied to models of learning in an educational context, Meguerdichian, Walker, & Bajaj, (2016) argue that it should also be applied in the context of interprofessional knowledge transfer.Scientific expertise may also lead to further questions that are not answered in the evidence provided to back a policy.For example, what are the funding sources for the study?In one relevant finding related to public policy, McGraw & Pinney (1990) find that domain experts and general political sophisticates differ in terms of both memory and policy evaluation, in reaction to being presented with information about tax policy/law.Critical science literacy permits experts to doubt the scientific underpinnings of certain claims.

Scientific Evidence and Support for Policy
We bring together three strains of research to craft the puzzle we investigate in the current paper.First, the legitimacy of bureaucratic policy is closely linked to the influence of scientific evidence on support for a policy.Second, the general public displays low levels of scientific comprehension, and often relies on heuristics.Third, perception of scientific evidence varies substantially among groups within the general public.
Given that scientists and scientific organizations are considered among the most trusted, honest, competent, and transparent of public authorities (Lang & Hallman, 2005), explicit reference to the source of scientific evidence should be an especially potent heuristic utilized by the general public.Expert sources, in general, are known to exert considerable influence on public opinion, though public attitudes may also reciprocally determine who is considered an expert (Page, Shapiro, & Dempsey, 1987).In turn, public support and willingness to act on behalf of a policy may also increase.
H1: Activists' willingness to act on behalf of a policy will increase with a general reference to science, as compared to no reference to science.H2: Activists' willingness to act on behalf of a policy will increase with a reference to a specific scientific source, as compared to a general reference to science.
However, given the relationship between expertise and a comprehensive understanding of the imperfections of science, we expect to see a lesser impact of scientific evidence on activism among a sample of experts.In characterizing experts as those with a technical or scientific background, this group is likely to exhibit an expertise reversal effect.
We still expect scientific evidence to positively effect experts' willingness to act, as the extant literature does not suggest a negative effect.However, the effect on experts may be smaller than on non-experts.This leads to our final hypothesis.
H3: Experts' willingness to act on behalf of a policy will increase less in response to the presentation of scientific evidence than that of activists.

Data and Methods
In order to identify the causal effects of scientific information on public activism, we conducted a randomized field experiment.The focus of the experimental design is an environmental policy, one recently proposed by the Environmental Protection Agency.The strength of the scientific evidence provided in support of the policy was manipulated, and subjects' willingness to mobilize was assessed in terms of the effect of varying the level of information.

Field Experiment
The field experiment was conducted in April 2016 in collaboration with a reputable national policy advocacy organization that focuses on scientific and technical issues.Participants were drawn from the organization's email list.This follows several recent studies of policy and political activism that use existing e-mail lists as the population in field experiments (see, e.g., Nickerson, 2007;Artz & Cooke, 2007;Malhotra, Michelson, & Valenzuela, 2012;Congdon & Shankar, 2015;Egebark & Ekström, 2016).The e-mail list is separated into two distinct categories: experts and activists."Experts" refers to members of the organization's Science Network, which includes scientists, engineers, health professionals, and other technical experts."Activists" refers to a broader list of supporters.The membership of the experts list is screened by the organization for members' qualifications whereas there is no screening of the activists list.In Table 1 we present descriptive demographic information as provided by the organization.The experts list is more male, older, and much more formally educated than the activists list.Since joining the activist list does not require any form of screening, it is possible for individuals on the experts list to join the activist list.However, in the organization-wide mailing represented in our study, one message was sent to each unique e-mail address such that, if someone joined both lists using the same e-mail address, (s)he would have received only one message.The organization reports that its e-mail list has approximately 460,000 members, with roughly 8% of members opening any given e-mail.
The population of subjects was selected to increase the external validity of the experimental treatment, as list serve members regularly receive policy advocacy emails and requests for action from the organization.As such, we are not concerned that list members recognized the experiment.In addition, list serve members enroll voluntarily, and are thus indicating a heightened interest in issues surrounding science and public policy.They represent an informed segment of the population in terms of the task that regulators face in persuading policy networks regarding the merits of proposed regulations.It is well documented that those who participate via electronic means, such as commenting on regulations.gov,are different than the general public along several dimensions (Wilhelm, 1997;Weber, Loumakis, & Bergman, 2003, Oser, Leighley, & Winneg, 2014).
The experimental email describes a recent proposed revision to the Accidental Release Prevention Requirements of Risk Management Programs by the Environmental Protection Agency (Federal Docket# EPA-HQ-OEM-2015-0725).Objectives of this proposed rule are to modernize and enhance safety at chemical facilities.A simple measure of the public salience of a proposed regulation is the number of comments submitted (Balla & Daniels, 2007).This was a fairly salient rule, having received over 40,000 comments during the commenting period, compared to an average for economically significant regulations on regulations.gov of less than 2,000 (Costa et al. 2016).This salience indicates that we could expect some baseline level of activism in response to the advocacy e-mail.The EPA's proposed revision was supported by the organization with which we collaborated, and the advocacy e-mail encouraged the recipient to support the proposal.The experimental manipulation within the email varies whether and how scientific research is cited as support for the proposed regulation.
In the first condition, a specific scientific journal-the Journal of Epidemiology and Community Health (JECH), a journal in which one of the studies cited in the RIA is published-is referenced.JECH was one of five peer reviewed journals that appeared in the RIA.Since it is a prestigious public health journal, we thought reference to JECH would be the most recognizable authority on the health science evidence on which the regulation was based.The JECH reference emphasizes the legitimacy and quality of the scientific data under the assumption that both public and professional list serve members would recognize the authenticity of the journal title.In the second condition, a specific scientific study is not referenced, but instead there is a general reference to the regulation being supported by "peer reviewed scientific studies."This condition maintains the authenticity of citing peer-reviewed science yet lacks the same degree of precision in establishing legitimacy.
This "peer-review" condition was intended to test the limits of the mobilizing value of referencing scientific evidence in support for a policy.In contrast to the first two conditions, the third condition makes no mention of scientific studies whatsoever.This design served as the control condition, whereby identical evidence was presented, but scientific research is not cited.The experimental manipulation appears at the end of the second paragraph of the e-mail.In the interest of maintaining anonymity of the organization involved in the study, we do not reprint the full e-mail in the current paper.
The details of the policy, journal name and volume number were drawn from the regulatory impact analysis for the proposed regulation.In the RIA, the study published in JECH (Elliott, Wang, Lowe, & Kleindorfer, 2004) is referenced due to the finding that high risk chemical facilities are disproportionately located in poor and minority communities.As such, there is no deception in this experimental design.The e-mail contained four paragraphs.The third paragraph provides data in support of the organization's case for backing the proposal.The experimental manipulations are included in the third paragraph.The sentence added to the e-mail to establish the first treatment condition is given below.The second treatment condition ends at the comma and does not mention any journals in which the supporting studies are published.

The potential impacts of the proposed revisions to the Accidental Release Prevention Requirements of Risk Management Programs have been referenced by peer reviewed scientific studies, in sources such as the Journal of Epidemiology and Community Health.
Within the email, an embedded "Take Action" button is presented alongside a request for members to contact an elected representative.Clicking the Take Action button indicates subjects' willingness to mobilize and was monitored by the organization's e-mail client.The organization also monitored which e-mails were opened.The key outcome of interest in the field experiment is the choice of a subject to click the "Take Action" button, which allows them to send comments to EPA officials.Unfortunately, we do not know if subjects who clicked the Take Action button actually typed in comments to EPA.Due to the intentional nature of URL navigation, and the effectiveness with which it can be passively tracked, URL traffic represents a growing mechanism through which activism on political and public policy issues is measured, in both experimental and observational research (see, e.g., Ryan, 2012;Kronrod, Grinstein, & Wathieu, 2012;Street, Murray, Blitzer, & Patel, 2015;Ryan & Brader, 2015, Platt, Platt, Thiel, & Kardia, 2016).The e-mail platform used by the organization permits us to estimate the rate of mobilization for each e-mail condition. 1 Before presenting our main results, we discuss some limitations related to the use of e-mail tracking in this experiment.We recognize that recipients could have read a portion of the e-mail's content through the preview of the e-mail provided in their e-mail clients (e.g., the "Preview Pane" in Gmail).However, we focus on the effect of the treatment onthose who opened the e-mail since the e-mail treatment was embedded in the third paragraph of the e-mail, which would not be accessible in the preview.We should note a slight imbalance in the rates at which emails were opened across experimental conditions.In the activist group, those assigned to the Journal Name condition opened their emails at a statistically significantly lower rate than did those in the other conditions.In the worst-case scenario for the validity of our results, this could indicate a failure of the randomization---that emails with the Journal Name condition were assigned to subjects who were systematically different than those assigned to the other conditions.Alternatively, it could be the case that the additional text in the Journal Name condition triggered some spam filters, preventing a small proportion of the emails from being opened.We interpret the results as indicating that the Journal Name condition likely triggered some additional email filters, since those in the expert group also opened the Journal Name condition emails at the lowest rate (though not a statistically significantly lower rate than in the other conditions among the expert group).

Analysis and Results
The results of the experiment are presented in Table 2.The experimental manipulation did significantly affect click-through rates in both groups, as indicated by statistically significant chi-squared tests.In addition to the chi-squared tests, we ran pairwise difference in proportions tests within each group.These constitute tests for null average treatment effect differences.Based on pairwise tests, in the expert and activist groups there is a statistically significant difference between the control and journal name conditions (with pvalues of 0.0011 and 0.0002, respectively).The control and peer review conditions are not statistically significantly different in either group.
The direction of the effect of the experimental manipulation differs in the two groups.In the activist group, e-mail recipients in the journal name condition clicked the "Take Action" link at a rate that was 0.8 percentage points higher than those in the control condition.In the expert group, recipients in the journal name condition clicked the "Take Action" link at a rate that was 4 percentage points lower than those in the control condition.These effect sizes are modest, and the large sample sizes in the activist group would lead any systematic difference to be statistically significant, even if not substantively significant in magnitude.However, if reference to science can shift activism rates by 0.5-4 percentage points, when multiplied across the large affiliation bases of advocacy organizations, reference to science could shift the number of advocacy actions by a substantively meaningful magnitude.Of course, there is still lively scholarly debate regarding the degree to which public comments-which are often embedded in mass comment campaigns (Balla et al., ND)-influence regulatory policy.Yackee & Yackee (2006) find that regulators respond primarily to comments from businesses, whereas Shapiro (2008) finds that large volumes of public comments can influence regulatory policymakers.
To formally test whether the effect of the journal name condition in the activist group was statistically significantly different from the effect of the journal name condition in the expert group, we computed a non-parametric bootstrap (Efron & Gong, 1983) distribution of the difference in differences in the journal name versus the control condition (see Rodrigues & Vergnat, 2016 for an example of this approach).We took 1,000 nonparametric bootstrap samples (i.e., simple random samples with replacement) from each group, and re-calculated the difference in click rates across experimental conditions within each bootstrap sample.The sample sizes in the activist/control, activist/journal name, expert/control, and expert/journal name are 33,031, 29,183, 2,039, and 1,919, respectively.
Figure 1 provides a visualization of the bootstrap sample of differences in differences across groups.All of the 1,000 bootstrap estimates are positive, indicating that the differences in differences across groups is statistically significantly positive----the effect of the journal name condition is statistically significantly higher in the activist group than in the expert group.
Our results indicate that reference to scientific evidence has a causal effect on the tendency for individuals who identify with a population of policy activists to become active in a policymaking Notes: Each cell gives the proportion of subjects, among those who opened the e-mail, who clicked the "Take Action" link.Raw counts that go into the proportions are given in parentheses.Chi-squared tests are 14.551 and 11.077 for the activist and expert groups, respectively.These test values indicate that differences across experimental conditions in both groups are statistically significant at the 0.001 level.There are differences in group sizes (i.e., denominators) across conditions within rows since we only included individuals who opened the e-mail.

Figure 1 Difference in Difference Estimate
Notes: Distribution of the difference between the difference in click rates between "Journal Name" and "Control" groups across the Activist and Expert samples.This distribution was derived by first taking 1,000 bootstrap samples from both the Activist and Expert samples.Within each bootstrap sample, we calculated the difference between the Journal Name and Control groups.We then took 1,000 differences between the differences in the Activist sample and the differences in the Expert sample.All differences-in-differences were positive, indicating that the "Journal Name" condition had a statistically significantly higher treatment effect in the Activist sample, than in the Expert sample.
instance.This relationship is, however, nuanced.We expected to find that reference to scientific evidence would have a greater positive effect on activists' participation rates than on experts.However, among the experts list, we found that reference to scientific evidence in the email caused a reduction in the rate at which the email recipients clicked the "Take Action"link, which was an unexpected result.The other result worth reflecting upon is the null result regarding the effect of the peer review condition.This resultmay reflect a simple lack of mobilizing value of a generic reference to scientific evidence in a policy advocacy argument, or it may result from a baseline assumption among group members that policies backed by the group are also backed by scientific evidence.
We speculate regarding a couple of possible mechanisms that could explain the negative effect of referencing scientific evidence among the recipients in the expert list.First, it could be that the evidence presented-in the form of a journal name-served as a sufficient cue to those in the advocacy group that the proposal was backed by scientific research.However, it is possible that those in the expert group subjected the evidence presented to a greater degree of scrutiny and found the presentation of a simple journal name to be insufficient, and possibly indicative of a lack of comprehensive scientific backing.As noted above, scientists' training positions them to be able to question scientific claims.The fact that this reference was embedded in an advocacy email, not an objective piece of scientific work, could have raised additional doubts regarding the veracity of the reference.The second possible mechanism may be that the reference to a scientific study distracted those in the expert group, perhaps causing some to seek out the study or other scientific evidence and fail to follow the organization's Take Action prompt.This second mechanism highlights a limitation of using clicks on the Take Action link to measure activism.This measurement approach is not perfect.By measuring activism as clicking on the Take Action link, we almost certainly have false negatives in our measurement.

Conclusion
Since they do not benefit from an electoral mandate, regulators must rely on their specialized expertise in establishing the perception of legitimacy regarding the ways in which they use their regulatory policymaking discretion.One potential avenue through which regulatory policymakers can ground their proposals in technical expertise is to draw upon scientific evidence.We drew upon the large sample sizes offered by a major policy advocacy organization's mailing list to identify subtle effects of references to scientific research in motivating attentive citizens to take action.Among subscribers to the much larger "activists" list, we found that reference to the journal in which scientific evidence was published caused a substantively modest but highly statistically significant increase in advocacy actions, relative to the control condition.In contrast, we found that reference to the journal resulted in a statistically significant decrease in advocacy actions within the much smaller "experts" list.Our results indicate that drawing upon scientific evidence in support of a policy proposal can affect the willingness of attentive citizens to advocate on behalf of that policy.
This study opens potential avenues for future research.One future direction deserving investigation regards the different results across experimental populations.Future work should examine whether the effects of scientific evidence on support for a policy depends on the level of expertise of the prospective supporter.Another future direction regards the type of information presented in reference to the scientific research.We found an effect of journal name, but did not explore other relevant characteristics, such as title, author names, and author affiliations.Lastly, the results we present are specific to a single policy and members of a single organization's mailing list.Future research should investigate the degree to which our results are robust to variation in policy and target population.A study of the effect of scientific evidence on support for policy that is focused on the general population would be of great interest.
1.This experiment was approved by the internal review board (IRB) at Pennsylvania State University.