Good luck : a field experiment applying behavioural science to improve attendance at high-impact recruitment events

Finding a job, especially in a recovering economy, is challenging and success is reliant upon effective job-search activity. Jobseekers Allowance (JSA) welfare benefit claimants in the United Kingdom have many competing options available to them in terms of how they direct their efforts in looking for work. Often it is hard to determine which is most productive. Unsurprisingly, Jobcentres – the organisations that support JSA claimants during their unemployment – themselves have very strong links to the labour market. For example, they are often invited to run recruitment events in direct partnership with large employers seeking to hire in bulk. At Bedford Jobcentre, we observe that, despite the relatively high likelihood of gaining work from attending such events, jobseeker attendance rates are still low and, instead, we can only assume that jobseekers may be taking part in less productive work search activities. This paper reports the results of a randomised control trial designed to test the effectiveness of mobile phone text messaging in compelling jobseekers in the Bedford area to attend such events. Tailored text messages are found to significantly increase the likelihood of attendance. We find text messages to be particularly effective when they evoke a sense of reciprocity in the recipient. Electronic version: Address for correspondence CMPO 2 Priory Road, Bristol BS8 1TX www.bristol.ac.uk/cmpo/ cmpo-admin@bris.ac.uk Tel +44(0) 117 33 10799 1 We are grateful to Daniel Butler, Sarah Baxter and Philip Hall at the Department for Work and Pensions for their assistance in the implementation of this trial and for quality assurance of the analysis. We are also grateful to Ariella Kristal for excellent research assistance, and to our colleagues in the Behavioural Insights Team for comments on an earlier draft, particularly to Sam Nguyen. I've booked you a place. Good luck: a field experiment applying behavioural science to improve attendance at high-impact recruitment events


Introduction:
Unemployment is among the largest problems facing most western societies.In the aftermath of the 'great recession', the financial crisis starting in 2008, unemployment across the developing world has been slow to recover, with much of the developed world yet to reach pre-crash levels.
This problem is costly to the unemployed individuals and to society as a whole, with governments paying unemployment benefits to more people and for longer, and productivity falling as a result of hysteresis.As well as these short term consequences, the longer term impacts may be even greater: Gregg (2001) and Macmillan (2011), find considerable evidence of employment "scarring"; a phenomenon whereby people (particularly young people) who experience a spell of unemployment due to a crisis are more likely to experience similar spells later in life, even when other characteristics are controlled for.
There are many models of employment developed by economists.As a basis for this experiment, we assume that there is validity in models positing that getting a job depends on job-search activity; crudely put, the more effort you put into trying to get a job, the more likely you are to get one (all else being equal).This suggests that job search activity is a component of the production function of employment and that more productive work search (for example, attending specific recruitment events where some participants will almost certainly gain employment) is key to finding work faster.
In this paper, we present the results of a randomised control trial designed to test the effectiveness of mobile phone text messaging in compelling jobseekers registered at Bedford Jobcentre to attend specific recruitment events.Such events are run by Jobcentre Plus (in Bedford and elsewhere) in partnership with large employers seeking to hire several new members of staff at once.As such, the chances of successfully finding work for a jobseeker are relatively high when attending these events.

Literature review:
There is a strong research precedent suggesting that text messages can provide a high impact, low cost, channel through which to galvanise action.Text messages have been trialled in a variety of domains such as repayment of court fines (Haynes et al., 2013), loan repayment (Morten et al., 2013) and saving energy (Gleerup et al., 2010).Text messages were already used as one of the communication channels deployed by Bedford Jobcentre Plus to promote recruitment events of this nature.This experiment aimed to identify evidence-based hypotheses as to what kinds of messages might be more effective and to test the efficacy of these messages in the field.

Using personalisation
Drawing on a well-established body of research into the power of using a first name, Haynes et al. (2013) demonstrate, through their work with the UK Courts and Tribunals Service, that personalisation of text messages in this manner has a significant impact on the efficacy of text messages in soliciting fine payment; 41% increase in average repayment amounts per fine compared to a standard message.

Evoking the Jobseeker/ Advisor altercast
Evidence suggests that behaviour can be influenced by an individual's perception of their own identity and that this perception is malleable.Altercasting, introduced by Weinstein and Deutschberger in the 1960s, provides an important conceptual contribution to this literature, exploring the mechanism by which social dynamics form when people or groups assume social roles relative to those of others and based on external information.This can be thought of as a form of identity priming on the part of the messenger on the recipient.For example, Bryan et al's (2011) study of voter turnout, tests a construct known as "manded altercasting" (see Pratkanis, A. R., 2007), by showing that appealing to the participant as a "voter", rather than simply asking them to vote, significantly increases turnout.
There are many archetypal altercasts, which have been shown to achieve different ends in different situations (see Pratkanis, 2007 for a detailed review).We are specifically interested in "tact altercasts", which evoke a specific social dynamic between parties by positioning one party in a particular role (in this case, we hypothesise that the advisor as an official figure will prompt the jobseeker receiving the message to characterise themselves in terms of an obedient counter-role, or alter) in order to shape the behaviour of the other party or parties.
Although altercasting has been widely studied, we are unaware of experiments that explicitly test the theory using text messages, the medium chosen for this experiment.However, Morten et al (2012) explore the effect of drawing attention to the relationship between loan recipient and loan officer on soliciting repayments via text message.Of the conditions they trial, "messages that mention the loan officer's name significantly, substantially, and robustly improve repayment rates relative to messages that mention the client's name and/or to the no-message control group" (pg 9).Given the nature of the particular altercast between loan officer and borrower, it is reasonable to conclude that reciprocity is a large component of this dynamic.Indeed, Morten et al (ibid) posit that, as the effect is only observable in borrowers who have already met their loan officer, it is precisely "social obligation/reciprocity" (pg 9) that explains the difference in results.In this experiment, we build on these findings by attempting to test the difference between elements of the advisor/ jobseeker altercast (which may or may not include implicit reciprocity) and an appeal to the construct of reciprocity as an explicit condition.

Reciprocity
Aside from the loan repayments example previously cited, there is a large literature exploring the use of reciprocity, that is to say unconditional gift giving by one party to another, as a means to encourage a desired behaviour (see Falk and Fischbacher, 2006).Both theoretical models and experimental results show that calling on social conventions of reciprocity is disproportionately effective when trying to get individuals to comply with a request.For example, an experiment into interventions that impact charitable donations by employees at a large bank showed that 11% of people given the small gift of a bag of sweets donated a day's salary compared to just 5% whose donation was solicited by the traditional awareness raising campaign tactics (Behavioural Insights Team, 2013).
Field studies of reciprocity conducted to date (BIT, 2013;Falk, 2007;Alpizar et al, 2008), aim to elicit reciprocity by the giving of physical gifts, such as sweets, fridge magnets and postcards.In this experiment, we consider whether a soft form of gift-giving (the exertion of a small quantity of effort to book an appointment), can be similarly effective at motivating behaviour.

Experiment Design:
We use a randomised control trial design to determine the effect of variations on a text message on attendance rates at specific recruitment events held by Bedford Jobcentre Plus.
Randomisation is conducted at the individual participant level.All participants (n = 1,224) were drawn from the pool of active JSA claimants at Bedford Jobcentre Plus.
Participants were randomly assigned to receive one of the following four (coded 0 -3) possible text message formulations: The study investigates the relationship between message variant and attendance rate at specific recruitment events.These events took place between May and December 2013 and attendance was measured based on whether or not the jobseeker showed up at any time during the recruitment session to which they were invited.
As the sessions were for specific jobs, it was possible to identify suitable participants using information in the Jobcentre Plus database on their skills and suitability for the type of work (as denoted by their Standard Occupational Classification (SOC) code).Jobseekers identified in this way were then texted in the order that their records were returned by the system.The maximum number of text recipients per recruitment event was determined based on Jobcentre capacity, staff availability and an expected attendance rate (estimated at 50%, although this was acknowledged to be a highly cautious estimate).
Owing to resource constraints, the same Jobcentre Plus staff member was responsible for sending the text messages and running the recruitment sessions.In order to mitigate the risks to the trial, the following procedure was used to ensure that the member of staff remained blind to the treatment: 1. Once participants are identified using SOC codes, their information (first name, last name, advisor name, phone number, national insurance number) is entered into a data template designed by the Behavioural Insights Team. 2. The Jobcentre Plus staff member responsible for data entry then runs a code written by the Behavioural Insights Team (by pressing a button), which handles the randomisation and data preparation as follows: i.
Assigns each participant to one of the four trials arms at random.ii.
Uses a conditional message formula to generate the text to be sent based on personal details and trial arm assignment.iii.
Replaces the National Insurance number with a unique identifier (so that the information can be shared with the Behavioural Insights Team without the need for personal data sharing).iv.
Creates two visible sheets: one with the text message and phone numbers only (plus instructions on the day and time the messages should be set to send at, to control for variations between sessions), which can be pasted into the text messaging machine; one with the names of all participants, their unique ID number and blank columns to record attendance plus any comments.v.
Creates a hidden (password protected) sheet that only the Behavioural Insights Team can access, which contains unique identification numbers plus the text condition for each participant.
The text messages are then sent using the first of the sheets and the attendance of participants is recorded on the day(s) of the session(s) using the second.At the end of each recruitment episode, the jobcentre staff send the attendance sheet and the hidden sheet (which is contained in the same document and remains uneditable without the password) to the Behavioural Insights Team along with a delivery report for all SMS text messages.The hidden data is decrypted and the unique identifiers are then used to match attendance to text condition.As an assurance exercise, the delivery records are checked to ensure that failure is consistent across conditions (indicating that the randomisation has been performed effectively) and that a sufficient number of texts were delivered.

Results:
Data were analysed in Stata.Our data contain 1,224 observations of participants who were eligible to receive text messages as part of this trial.For each participant we observe their treatment assignment, the name of their job centre advisor, and the details of the job they are invited to apply for, including the job title, the employer, and the number of posts available.As well as the text message they were sent, we observe whether or not they received the text message (i.e.whether the phone number held by DWP for them is correct) In total, roughly 25% of the text messages (307 participants) failed to be received.Finally, our data contain information about the session to which they were assigned, and whether or not they attended that session.The table below reports the results of basic regression analysis making use of a linear probability model.All models estimated above make use of binary outcome measures, and are estimated by a linear probability model.These results do not change with the use of a logistic regression.
Column 1 in the table above shows Intention To Treat Estimates for the full sample -that is, for all participants whom the job centre attempted to reach in each treatment group.We identify a strong positive impact on attendance in the final, reciprocity, treatment.Table three reports average levels of attendance by treatment, and 95% confidence intervals around each.From this it is clear that treatment 3 performs significantly better than any of the other treatments.Returning our attentions to table 2, the second column shows a complier average causal effect (CACE) model, where the sample is partitioned only to consider those 917 participants who actually received a text message.As might have been predicted, this serves to enhance the size of the effect, to a 20 percentage point increase in attendance in the most successful treatment group.
One major concern is that our randomisation may not have been successful, and so participants assigned to our treatments may have been more likely to attend than those in the control group even in the absence of treatment.To test this, in column 3 we conduct a "Defier Average Causal Effect" (DACE) model, which estimates whether treatment assignment had any effect on those individuals who did not receive a text message, owing to delivery failure.Since these individuals were not, in fact, treated, we expect no significant impact on this group if randomisation has been successful.We find that this is the case, with no significant impact of treatment assignment on outcomes (attendance) for this untreated group.Although treated groups are more likely to respond than untreated groups, this effect is highly insignificant (ps>0.371),and the ordering of these differences is not the same as among the ITT or CACE analyses.We are therefore content that randomisation has been successful and that the differences observed between groups is a result of our treatments.
Finally, column (4) contains a further balance check to determine the success of randomisation.In this specification, the dependent variable is whether or not a text message failed to be receivedthat is, whether treatment assignment (which should be exogenously applied) predicts failure.Here, we find that all treatment groups are slightly less likely to fail than the control group.For all but treatment 2, this difference is insignificant.In treatment 2, the difference is significant (p=0.046),however the effect is not sufficient to lead us to question our primary results.Again, we note that use of a logistic regression model does not substantively alter our results.

Discussion:
The objective of this study was to investigate the effects of variations on a text message on attendance rates.Our results show that the "reciprocity condition" (T3: "... I've booked you a place.Good luck...") is significantly more effective than any other message in attracting participants, leading to an attendance rate of 26.8% compared to 10.5% in the control.Although the effect is not of the same magnitude, we also find that using the Jobseeker's first name and the name of their Advisor is powerful in prompting attendance.These findings are in line with our initial hypotheses.
Whilst many other experiments use the medium of text message to effect a particular action, such as attendance, we have not encountered studies that directly foreshadow what we investigate in this paper.The authors believe that further study could be fruitful to ensure maximum effect is being achieved.For example, one limitation of the current study is that we are not able to disentangle the effect of reciprocity (as evoked by "I've booked you a place") from the effect of introducing the idea of luck ("good luck!").The literature suggests that job seekers with an internal locus of control conduct more work search activity than those who believe that their future outcomes are determined by external factors (e.g.Caliendo et al, 2010).It strikes the authors that evoking the concept of luck may alter the locus of control of jobseekers who feel they do not govern their own future, thereby making them more likely to attend than they may otherwise have been.

Table 1 : Message variants used in the experiment Treatment number Message text (formulation and example) and behavioural rationale
Formulation:[number]new [type of job] are now available at [company].Come to Bedford Jobcentre on [date and time] and ask for [staff member name] to find out more.Example: 8 new Picker Packer jobs are now available at Pro FS.Come to Bedford Jobcentre on Monday 10 June between 10am and 4pm and ask for Sarah to find out more.Hi [jobseeker name].[number]new[type of job] are now available at [company].Come to Bedford Jobcentre on [date and time] and ask for [staff member name] to find out more.Example: Hi Elspeth, 8 new Picker Packer jobs are now available at Pro FS.Come to Bedford Jobcentre on Monday 10 June between 10am and 4pm and ask for Sarah to find out more.Hi [jobseeker name].[number]new[type of job] are now available at [company].Come to Bedford Jobcentre on [date and time] and ask for [staff member name] to find out more.[Advisor name] Example: Hi Elspeth, 8 new Picker Packer jobs are now available at Pro FS.Come to Bedford Jobcentre on Monday 10 June between 10am and 4pm and ask for Sarah to find out more.MichaelLogic: Sending the message from the advisor adds an additional layer of personalisation and is likely to encourage the claimant to act according to the conventions of the social bond with their advisor (altercasting).

:
Hi [jobseeker name].[number] new [type of job] are now available at [company].Come to Bedford Jobcentre on [date and time] and ask for [staff member name] to find out more.I've booked you a place.Good luck, [Advisor name] Example: Hi Elspeth, 8 new Picker Packer jobs are now available at Pro FS.Come to Bedford Jobcentre on Monday 10 June between 10am and 4pm and ask for Sarah to find out more.I've booked you a place.Good luck, Michael Logic: Stating that a place has been reserved and wishing the claimant luck should reinforce the social bond and introduce the concept of reciprocity, which is a powerful behavioural motivator.