Testing behavioral interventions designed to improve on-time SNAP recertification

very year in the United States, many families benefitting from social welfare programs fail to recertify on-time even though they remain eligible and want to continue to receive benefits. The Supplemental Nutrition Assistance Program (SNAP) is no exception. Given that it is hard to measure whether people leave SNAP intentionally or unintentionally, the United States Department of Agriculture (USDA) typically proxies the rate of “unintentional drops” from the program by measuring churn, defined as a SNAP-participating household exiting and re-entering the program within a four-month period. A 2014 USDA report (Mills, Vericker, Koball, Lippold, Wheaton, & Elkin, 2014) estimated annual administrative churn rates ranging from 17 to 28 percent across the SNAP caseload in six states. Administrative churn imposes costs on both the SNAP household and on the SNAP agency processing the case, which must re-enroll the applicant rather than recertify, a more time-consuming task. According to Mills et al. (2014), the value of foregone benefits for recipients ranged from $2.2 million in Idaho to $108.2 million in Florida. The same report also showed that re-enrolling a client can take staff two to three times longer than recertification, increasing total administrative costs for the program by one to four percent. This paper contributes to the behavioral public administration and social policy literature by reporting results from a series of randomized controlled trials (RCTs) testing behavioral interventions designed to inE V Abstract: We report results from a series of randomized controlled trials (RCTs) run between January 2019 and February 2020 testing behavioral interventions designed to increase the likelihood that SNAP recipients recertify on-time in Hennepin County, the most populous county in Minnesota. Given the different levels of governance and the abundance of qualifying rules and processes that low-income households must negotiate to obtain and retain SNAP food assistance benefits, many households may fail to recertify for SNAP. Administrative burden includes the difficulties created by having to learn deadlines and which forms constitute the proper paperwork necessary to recertify. In our main intervention, we test a three-armed study (n=23,756), comparing the efficacy of the Hennepin County SNAP recertification auto-dialer communication, a behaviorally-informed text message, and a third arm with both the Hennepin county autodialer and our text message, against a control group that did not receive any reminder at the beginning of the recertification month but did receive other standard written communications. Results from this trial show that the autodialer is not an effective reminder. However, the interventions with text messages are effective in improving recertification rates around five percent (p<0.01) over no additional message and around two percentage points (p<0.05) over the Hennepin County autodialer message currently being used. Text messaging appears to be particularly effective for SNAP recipients under the age of 60 with low to moderate levels of education.

crease the likelihood that SNAP recipients recertify on-time in Hennepin County, the most populous county in Minnesota. The SNAP program in Minnesota is administered at the county level; and, depending on the characteristics of the household members, participants may be enrolled in the program and recertified according to different regulatory processes and timelines. Our study focused on the caseload, not on Temporary Assistance for Needy Families (TANF). Most of our study group have their SNAP benefits sizes redetermined at six months and have a full recertification for eligibility after 12 months. Given the different levels of governance and the abundance of qualifying rules and processes that low-income households have to negotiate to stay on public programs and make ends meet, many households may fail to recertify due to the administrative burden of doing so (Herd & Moynihan, 2018).
We ran a dynamic sequence of RCTs between January 2019 and February 2020. During this time, the Minnesota Department of Human Services (MDHS) sent SNAP recipients in Hennepin County a letter and packet of forms to complete, through the U.S. Postal Service, about 45 days before their recertification date informing them of the deadline and the necessary processes to recertify. This letter and the accompanying forms are required by federal law. SNAP recipients in Hennepin County must also complete an interview, which can take place by telephone. A typical client facing recertification in Hennepin County and who does not recertify would receive: 1) a separate letter from Hennepin County Human Services and Public Health Department (HCHS&PHD) informing them of the option to complete the required interview over the telephone about 40 days prior to the recertification deadline; 2) an automated phone call (i.e., autodialer) to remind recipients to fill out their paperwork and complete the interview about a month before the recertification deadline; 3) a letter informing clients of a missed interview 15 days before the end of the month; and 4) a letter notifying clients that their benefits will be terminated 14 days before the end of the month. As we explain below, our communication intervention focuses on changes to the autodialer reminder about a month before the recertification deadline. All other communications remain the same.
Throughout 2019, only about 60 percent of the SNAP caseload completed the 12-month recertification process on-time, around 20 percent of the caseload failed to recertify on-time but requalified for benefits in the following month, 5 percent failed to recertify on-time but requalified between 30 and 90 days later, and 15 percent of the caseload failed to recertify and left the program for at least three months. Our intervention targeted the 20 percent of the caseload who failed to recertify on-time but who returned to the caseload quickly, under the assumption that most of these clients exited the program because of administrative misunderstandings and other difficulties and not because of changing attachment to the labor force.
The main intervention we tested was a three-armed study, comparing the efficacy of the Hennepin County SNAP recertification autodialer communication, a behaviorally-informed text message, and a third arm with both the Hennepin county autodialer and our text message, against a control group that did not receive any reminder at the beginning of the recertification month but did receive the other standard written communications. Results from this trial show that the autodialer is not an effective reminder. However, the interventions with text messages are effective in improving recertification rates around five percent (p<0.01) over no additional message and around two percentage points (p<0.05) over the Hennepin County autodialer message currently being used by HCHS&PHD.

Behavioral Science Literature
For decades, behavioral scientists have made a number of contributions to the social science literature showing that behaviors are often inconsistent with the rational model that many economists assume. These discoveries contributed to the development of an alternative unified model of human behavior that relies on dual-intuitive and reasoning-cognitive systems. In short, behavior and thought are dependent on the attention and success of more effortful reasoning against more automatic impressions and biases (Kahneman, 2003;Tversky & Kahneman, 1974). Thaler & Sunstein (2009) argued that policymakers can improve individual and social welfare by making small modifications to programs designed to account for such biases. Over time, several behavioral science principles have been shown to nudge people toward more socially optimal outcomes (Benartzi, Beshears, Milkman, Sunstein, Thaler, Shankar, & Galing, 2017). Below we briefly describe the principles of loss aversion, the endowment effect, reciprocity, and social norming, which informed our experimental treatments.
In their seminal paper on prospect theory, Kahneman & Tversky (1979) first describe loss aversion, the notion that people experience a much greater reduction in satisfaction for a loss than they receive in satisfaction for an equivalent gain. In other words, people tend to dislike losses much more than they like gains. Economists distinguish reciprocity from cooperation in human interactions. Cooperation occurs because parties expect to interact again, and they understand that there are benefits to be gained by working together. Reciprocity, on the other hand, is a positive response to friendly behavior even when no material gains are expected (Fehr & Gachter, 2000). For example, Cialdini (1993) showed that people are much more likely to purchase an item in a store when provided free samples, even though that was not required to receive the sample. When people perceive another party as doing them a courtesy, they are more predisposed to respond favorably. Social scientists have also found that people are strongly influenced by the behavior of other individuals frequently modeling their behavior, a principle called social norming. In fact, merely suggesting to an individual that most people tend to make a particular choice when given options, or that they behave in particular ways, can lead to conformity (Cialdini, 2003). For instance, individuals who were told that their neighbors consumed less energy than they did reduced consumption more than those who were provided energy savings advice (Nolan, Schultz, Caldini, Goldstein, & Griskevicus, 2008;Allcott, 2011).

Administrative Burden Literature
Administrative burden is the cost associated with interacting with government (Burden, Canon, Mayer, & Moynihan, 2012;Heinrich, 2018;Herd & Moynihan, 2018;Moynihan, Herd, & Harvey, 2015). Administrative burden can be divided into three separate categories: learning costs, compliance costs, and psychological costs (Herd & Moynihan, 2018;Moynihan et al., 2015). Herd & Moynihan (2018) explain that learning costs occur as individuals seek information necessary to participate in government services. Compliance costs accumulate as participants attempt to follow the requirements for participation, including the time necessary to fill out forms and complete interviews, and psychological costs include many factors, such as stigma, feelings of dependence, as well as the stress and frustration that may occur when working with a bureaucracy. Herd & Moynihan (2018) argue that social welfare programs, including SNAP, targeting low-income populations tend to have the largest administrative burdens for those eligible. For example, there are multiple compliance costs associated with recertification such as filling out paperwork, completing interviews, and missing work or needing childcare to remain certified that have been shown to reduce participation (Currie 2004;Kabbani & Wilde, 2003;Ratcliffe, McKernan, & Finegold, 2008). Recent work has also demonstrated that providing messages about eligibility and assistance can increase SNAP participation considerably (Finkelstein & Notowidigdo, 2019).
Our intervention attempts to reduce the learning costs for those who want to stay on the program, but are potentially dropped because the recipient does not apply on time. Thus, while SNAP re-enrollment may produce compliance and psychological costs, we focus on reducing administrative burden by making it easier to understand the requirements for recertification.

Data and Method
We used Hennepin County administrative data on individual SNAP cases and created interventions approximately 28 days before the recertification deadline. For example, we offered the treatment in early January 2019 for the cases that were expected to recertify before the beginning of February 2019. 1 Our outcome measure is an indicator set to one if the recipient recertified on time. To conduct the experiment, we provided randomized case identification numbers at the beginning of each month to Hennepin County administrators to generate treatment and control groups and to send the appropriate messaging based on assignment.
We describe several different RCTs that evolved dynamically between January 2019 and February 2020. 2 Our original plan was to A/B test several behaviorally-informed autodialer messages for the first six months of the study period, comparing them to the original Hennepin County autodialer reminder. After testing this messaging for several months, we would then introduce a behaviorally-informed text message as a comparison in a later trial.
However, given the null results from the first RCT, we eventually realized that the autodialer may not be effective at all. 3 After consulting with our partners at Hennepin County, we modified our protocol and developed a new RCT that asked if the original Hennepin County autodialer message was beneficial compared to no autodialer. We ran that RCT in June and July of 2019 and added it to our texting intervention that was started in August 2019. In this last RCT, we randomly assigned recertifying individuals to one of three conditions, which we compared to a group that did not receive an intervention. The three experimental conditions were 1) those that received the original Hennepin County autodialer message, 2) a group that received a behaviorally informed text message, and 3) a group that received both the autodialer and a text message. We included the final group to test if the return to texting was best thought of as a substitute to the auto-dialer or if the extra message provided a boost even to those who also received the auto-dialer response. Additionally, this twopronged information strategy yields information about the possible cognitive overload these low-income clients might face by receiving too much information (Mullainathan & Shafir, 2013;Shah, Mullainathan, & Shafir, 2012). 4 The text for the original auto-dialer used by HCHS&PHD is shown in Figure 1.

Figure 1 Hennepin Autodialer Message
"Hello. This is a courtesy call from Hennepin County Human Services and Public Health Department concerning important renewal paperwork your household received in the mail. Please complete the form and return it with any required supporting documents to Hennepin County. Please remember you will be required to complete an interview after we receive your paperwork. To complete the interview or if you have questions about this paperwork, please call 612-596-1300, Monday thru Friday, 9am to 4pm. Thank you." In Figure 2, we provide the autodialer message with behavioral insights that we used to improve on-time recertification. The content introducing the sender as well as the contact information concluding the message is largely the same as the Hennepin message. However, the treatment contains several important differences. In the second sentence, marked by bracket a, we remind the recipient that we sent them the renewal forms attempting to play on the reciprocity principle. The third sentence, marked by bracket b, has two behaviorally informed concepts embedded. First, we use social norming, telling the recipient that most recipients respond immediately. We also add the phrase "to keep their benefits" to prompt a loss aversion response, i.e., reminding the recipient that these are their benefits, and they do not want to lose them.

Figure 2 Behaviorally-Informed Autodialer Treatment Message
Treatment 1: "Hello, this is the Department of Human Services and Public Health in Hennepin County. [We recently mailed you renewal forms.] a [Most people mail or bring in their completed forms and supporting documents as soon as possible to keep their benefits.] b An interview is also required. Please call us at 612-596-1300, 9am to 4pm, Monday thru Friday, to complete your interview or if we can help."

Figure 3 Text Message
Hi, this is Hennepin County Human Services. We will text reminders about your benefits. We want you to keep your food benefits. Fill out the forms we mailed and call us@ 612-596-1300 for an interview or questions.
In Figure 3, we report the content of the text message used. Because the text message is constrained to be 160 characters, clearly there is less content. Most of the message is informational, but the second and third sentences use reciprocity and loss aversion. In addition, Hennepin County does mention that it is their food benefits that are in jeopardy, which may play some role in any observed treatment effects.
We run OLS regression models to estimate the average treatment effect for each of our RCTs. While SNAP recipients were assigned to the treatment and control groups randomly, we include several covariates in the regression models to reduce unexplained variation and increase precision for the treatment effect estimates (Stock & Watson, 2003). These controls include several sets of indicator variables: race (African American and other race with white being the omitted category), the recipient's highest level of education completed (less than high school, high school, some college education, with college graduate being the omitted category), a set of dummy variables from the month the data were collected, and another for the individual's marital status (married, married and living apart, divorced, never married with residual marital status categories being omitted). The models also include an indicator variable for female, if the individual's primary language at home was English, and for U.S. citizens. Finally, we add continuous measures for the individual's age, age squared, and family size. Online appendix Table A1 provides descriptive statistics as well as balancing tests to determine the success of randomization. Of the 54 differences between the treatment arms and the control group, one was significant at the 0.01 level and one at the 0.05 level, suggesting that randomization was successful.
For both of our RCTs, we report results for all SNAP recipients. We also investigate several subgroups. First, we ask about heterogeneous treatment effects by education level: recipients with less than a high school education, high school graduates, those with some college education, and college graduates (Panel B). We also estimate average treatment effects for those aged 60 and older and those younger than 60 since eligibility criteria and recertification periods differ for those over age 60. (Panel C).

Results
Results from the first RCT asking if the behaviorally-informed auto-dialer messages changed on-time recertification rates show that those who received the behaviorally-informed autodialer message were 1.2 percentage points more likely to recertify on time compared to the group that received the original message (See online appendix Table A2 for details). Compared to the mean recertification rates for the sample, 62.5 percent, this is a 1.9 percent increase; however, the point-estimate is not statistically significant at conventional levels. Overall, with one exception, we consistently find positive estimates for recipients of the behaviorally-informed message regardless of the subgroup, yet none are significant at conventional levels, suggesting that the new autodialer was not an improvement over the original message.
In Table 1, we report results from the RCT with three intervention groups (autodialer, text, and autodialer plus text) against the comparison group who did not receive an autodialer or text reminder (See online appendix Table A3 for the full set of results including the controls). Panel A provides results using the full sample. The RCT results suggest that an autodialer message compared to no reminder at all increases the likelihood of recertifying on time by 1.2 percentage points. Given the baseline recertification rates is 58.5 percent, this is around a 2 percent increase in the likelihood of on-time recertifying. However, this result is not statistically significant at conventional levels. In fact, across all subgroups (Panels B through C), the auto-dialer is never a statistically significant improvement over no reminder at all. This suggests that the use of an autodialer is not effective for any group on SNAP. We focus on the other two treatment arms for the remainder of the paper.
In Panel A, both the groups that received the texting and the texting plus calling interventions are statistically significantly more likely to recertify on time relative to the group that received no reminder by three percentage points or approximately 5 percent. Because the Hennepin County autodialer is the status quo recertification message, we also tested if the texting and texting plus calling interventions improved on-time recertification compared to the current Hennepin message. Results from these tests show that the texting (p = 0.085) and calling plus texting (p = 0.049) are both an improvement over the Hennepin message.
In Panel B, we report results by educational attainment. For those with less than a high school education, we only find statistically significant point estimates for those who received texts plus calls. The point estimate is 4.6 percentage points or about a 7.5 percent increase in the likelihood of on-time recertification. None of the other arms are significant, and tests comparing the coefficient estimates for the two texting groups to the Hennepin County autodialer are not significant.
Interestingly, for high school graduates and those with some college education, texting is the only significant intervention. Among high school graduates, texting increases on-time recertification by around 3.6 percentage points or 6.3 percent relative to no intervention. Among recipients with some college education, texting increases on-time recertification by 8.1 percentage points or 14.8 percent. For both, the high school group (p = 0.053) and the some-college education group (p = 0.028), we also find statistically significant differences between the Hennepin autodialer and the texting result suggesting this option improved on-time recertification relative to the status quo message. Finally, among the college graduates, we do not see any statistically significant differences between the no intervention group and the three treatment arms.
The second two panels provide more insight into which groups are most affected by the interventions. Among those over 60, none of the interventions appears to help with recertification. However, this group tends to recertify on time at higher rates than the younger group (70.4% vs. 59.4%). For those less than 60, texting improves recertification by 3.8 percentage points or about 7.2 percent and calling and texting by 3.7 percentage points or 7 percent over no messaging. Further, both texting (p = 0.036) and texting plus calling (p = 0.023) are an improvement over the Hennepin County autodialer.
We also explored the sensitivity of our intervention to English language proficiency and US Citizenship status in analyses not shown. In both cases we consistently found that the intervention was not effective for non-English language speakers and those lacking citizenship status but that texting improved on-time recertification both alone and in combination with the autodialer for English speakers and U.S. citizens. Source: Administrative SNAP data from Hennepin County, MN Notes: * p < 0.10; ** p < 0.05; *** p < 0.01; a represents a statistically significant difference (α = 0.10) between the reported coefficient and the coefficient for the Hennepin County autodialer; b represents a statistically significant difference (α = 0.05) between the reported coefficient and the coefficient for the Hennepin County autodialer; Standard errors in parentheses; Controls include sex, age, age squared, education, race, family size, U.S. citizenship, English as primary language at home, marital status, and month of recertification.

Discussion/Conclusion
Collectively, our results show that the Hennepin County autodialer was not an effective reminder to recertify for SNAP. However, texting improves recertification rates between five and 15 percent, depending on the group analyzed, over no messaging and frequently represents a large improvement over the Hennepin County autodialer, the current practice in Hennepin County. It is difficult to tell if the texting plus calling treatment is a new result or simply the result of the texting alone since in most cases, the effect size is similar. What is clear is that the texting appears to be most effective for those with less than a college degree, young recipients, US. citizens, and those who primarily speak English at home. What may be most surprising is that a single text is effective even when recipients receive several notifications through the U.S. Postal Service. This result is consistent with other work that shows the importance of text messages and that repeatedly contacting social welfare participants can be effective (Bhargava and Manoli 2015). This message was sent approximately one month before recertification and after two mail notices and before two more reminders. Given the inherent difficulty in complying with the SNAP recertification process, texting appears to convey information in a manner that is more easily acted upon, at least relative to a telephone message, and likely reduces the administrative burden of recertification. Given the size of the effect from such a small intervention, public administrators might consider more frequent use of this communication strategy in other applications designed to reduce administrative burden. For example, future work in this area might compare texting alone to mailed information for SNAP recertification.

Notes
1. We focus on on-time recertification rather than other measures of delay in SNAP recertification, such as churn, because our behavioral interventions should nudge people within a short time frame and is not likely to have much impact beyond that point. If true, we expect to observe higher levels of on-time recertification, but also that some clients who might have missed the deadline and churned, do not. 2. We terminated data collection in February 2020 for this experiment to avoid the potentially unusual SNAP caseloads that resulted from the COVID-19 pandemic. Given the study period, a few cases were duplicated over the year. We only include the first instance of each case in our results. 3. We started a second autodialer message trial in May 2019 before we had completed data from the first trial.
As we report below, we found no evidence of any difference in the original Hennepin County message and the new message. We then halted the trial with the second message in May 2019 to prepare for the non-autodialer versus auto-dialer trial. 4. For the second RCT, we tested no autodialer versus the Hennepin County autodialer exclusively in June and July 2019. We folded these observations into the three-treatment RCT when reporting results. In addition, the vendor had a coding error for two months (August and September of 2019) in which participants randomized to receive only text messages, received both texts and the autodialer. We adjusted the allocation ratios in subsequent months to compensate for this issue. All of our models include month fixed effects to account for any differences in the types of individuals who may recertify differentially across the calendar.