Smarter Resolutions programme: Randomised Control Trial A (Workstream 1: post-notification)

Background

In 2021, the Ministry of Justice (MoJ) and the Department for Business, Energy and Industrial Strategy (BEIS) were awarded funds over a 2-year period (2021 to 2023) from the HM Treasury (HMT) Shared Outcomes Fund, to work with the Advisory, Conciliation and Arbitration Service (Acas) to test a series of early interventions in the employment dispute resolution system (the Smarter Resolutions programme).

The Smarter Resolutions programme has 4 operational workstreams that focus on different parts of the Early Conciliation (EC) user journey. Early conciliation is a process through which Acas talks to both claimant and the employer about a dispute, offering the chance to come to an agreement without having to go to an employment tribunal.

This protocol details how we will assess the impact of one part of the 4 workstreams: Workstream 1: Post-notification content. We will use a Randomised Control Trial (RCT) approach.

Population

The population for the trial is all claimants and representatives with an employment dispute who have notified Acas and agreed to participate in EC between 25 July and 19 August 2022.

Intervention

The post-notification content will provide more information and context to help claimants and representatives have a better understanding of:

• next steps in the EC process
• timelines
• what will be required from them after they agree to EC

It will sign-post people towards other appropriate resources, including new video content – ensuring they can access the right information at the right time. This should lead to a better understanding of the process, increased satisfaction with the process and more claims settled during EC.

RCT A Objectives

The aim of RCT A is to test whether sign-posting claimants and representatives to this newly developed content leads to improvements in the EC process, including a better understanding of the process, increased satisfaction with the process and more claims settled during EC.

Method and design

This is an "intend to treat" trial, that uses clustered randomisation. Rather than randomly allocating individual cases to the 2 arms of the trial, conciliators are allocated randomly. All cases that the conciliator progresses will be in the control or intervention group, depending on which group the conciliator is in. The trial will run between 25 July and 19 September 2022. We estimate 3,400 cases will be assigned to each arm of the trial.

Control conciliators will deliver business as usual (BAU). This involves providing an introductory email outlining next steps at the start of EC (all static text). The intervention conciliators will provide discretionary content and emails, sign-posting people towards new videos and guidance pages on the Acas website.

Outcomes will be measured through management information collated through Dynamics 365, Acas's Case Management System (CMS). This holds all information about a claim as it progresses, and an online survey of claimants or their representatives 6 weeks after EC begins (4 to 6 weeks is the standard time allowed for EC).

The primary outcome we are measuring is whether more cases are being settled during the 6 weeks of EC or not, with fewer cases escalating to ET.

A process evaluation will complement the trial. We will conduct qualitative interviews with 20 individuals in the intervention group to explore the way in which the new content was used and their perceptions of how it impacted on their EC experience and decision-making. We will also conduct 10 qualitative interviews with conciliators to discuss their experiences of using the new content. Through this analysis, we will test the intervention implementation, mechanisms for change and assumptions.

An evaluation report will be published in early Summer 2023, bringing together evidence from all 4 workstreams.

About Acas

Acas provides free and impartial information and advice to employers and employees, on all aspects of workplace relations and employment law.

Employment tribunals make decisions about employment disputes. Nearly all legal cases about employment are heard in employment tribunals. This includes cases about things like unfair dismissal, redundancy and discrimination. There are also many other sorts of claim that can be brought, such as non-payment of wages. Acas has a longstanding statutory duty to promote the resolution of claims to the employment tribunal (ET) in order to avoid recourse to a full hearing, by means of its conciliation service. In April 2014 a new legal requirement was introduced that means individuals must notify Acas of their intention to make an ET claim.

Since this change in the law, Acas offers the opportunity for both parties involved in the dispute (the individual and the employer) to resolve the issue through a new voluntary EC process. EC notifications can be made by an individual against their employer or on behalf of a group against their employer.

The EC process has been designed so that Acas can provide an impartial conciliation service with the aim of parties reaching a settlement and avoiding the potentially time-consuming, costly and stressful employment tribunal process. Cases progressing through EC are assigned an impartial conciliator who will:

• explain the conciliation process
• discuss the issues with both sides
• give an overview of the law
• discuss how employment tribunals have considered similar cases
• help both sides explore the strengths and weaknesses of the case
• discuss possible options without making any recommendations
• remain independent of the dispute – they will not take sides or tell either party what to do
• outline the employment tribunal process

If the offer of EC is refused or fails to reach agreement, the individual can still pursue an ET claim.

Background to the Smarter Resolutions programme

In 2021, the Ministry of Justice (MoJ) and the Department for Business, Energy and Industrial Strategy (BEIS) were awarded funds over a 2-year period (2021 to 2023) from the HMT Shared Outcomes Fund, to work with Acas to test a series of early interventions in the employment dispute resolution system ('the Smarter Resolutions' programme).

The Shared Outcomes Fund incentivises and funds innovative pilots, with departments working collaboratively to improve cross-governmental outcomes. It places a high value on robust, high quality impact evaluation and sharing results. More information about the Shared Outcomes Fund is available in Section 6.

The Smarter Resolutions programme has 4 operational workstreams that focus on different parts of the EC service, which is Acas’s service for resolving individual disputes. Each workstream explores how early interventions can help Acas to improve the service offered to customers, to resolve disputes more quickly and effectively, therefore preventing the escalation of issues, improving the experience of individuals and reducing the flow of cases into the ET system. It is expected that this will deliver cost savings across government and arm’s length bodies, such as Acas. The 4 operational workstreams are summarised below.

The overall programme vision is that 'people in a workplace dispute have a clearer understanding of how they can reach a quicker, more cost-effective resolution at the earliest point possible'.

The overall outcomes are that:

• parties spend less time in the ET system
• fewer cases require judicial time (including final determination)
• conciliators are focused on resolution and not administration
• parties engage in discussion with conciliators which is focused on resolution

The programme consists of 4 workstreams. These are:

• workstream 1: content strategy
• workstream 2: assisted notification
• workstream 3: optimised distribution
• workstream 4: large group claims

Workstream 1 is developing several new forms of content, to provide potential claimants or their representatives with information that helps them better understand the dispute resolution process as it applies to them at various points in their journey. The underlying principle is that users need to see 'the right content delivered at the right time'. The content aims to ensure claimants or their representatives have a holistic understanding of the conciliation journey, in order for them to make informed choices about their next steps and have realistic expectations about the dispute resolution process.

Workstream 1 content is wide-ranging and will be embedded across 4 points of the user journey. The 4 stages in order are:

• "Start to find help" content, delivered across the Acas website, to aid users when they are at the early information-gathering phase of their dispute journey. This includes the 'emotional support' page, 'time limits' page and 'wages' video. These are hosted on different Acas pages and visitors to the pages may or may not come across them; it is up to the discretion of the conciliator at the point a case has been allocated to one whether to actively signpost the claimant to any of these pages.
• "Start the form" content, located on the subdomain that hosts the EC notification form – the tell.acas.org pages. This content is designed for claimants to engage with at the point of initially arriving on the form with the intention of notifying Acas about their ET claim.
• "Complete the form" content on the notification form itself and content sent to users after their notification has been received, outlining what to expect next. This involves an automated 'you've notified' email once the form has been submitted and an automated 'introduction to claimant or respondent' email once a case has been allocated to a conciliator, which includes a link to an 'intro to conciliation' video.
• "Conciliate" content, also known as 'post-notification content', provided by the conciliator to users once they have reached the stage of EC. Conciliators will have access to a series of materials they can use at their discretion including template emails and links to wider resources. The wider resources include the 'emotional support' page, 'time limits' page and 'wages' video, mentioned above.

The content was developed during late 2021 and early 2022 by an external content strategist. The theory behind the intervention is that by providing people with the right information, at the right time, they will reach better resolutions. Following user-testing the content was ready to be rolled out from July 2022, beginning with the "conciliate" post-notification content. It is hoped the videos will increase engagement with the full range of materials and this represents the main difference between the control and treatment arms of the trial (people in the treatment arm will be signposted towards the videos).

Randomised Control Trial (RCT) A will only evaluate the "conciliate" post-notification content (provided after the notification form is complete and the user has confirmed they want to progress to EC).

The reason why this trial only focusses on post-notification content is that trialling pre-notification would be challenging. Keeping pre-notification control and treatment groups completely separate is not possible, as people may review on different devices, disable browser cookies or review the content several times before making a claim. Wider workstream content, including pre-notification content will therefore assessed as part of a separate RCT trial (RCT B). However, for the reasons noted above there will be some limitations to the evaluation of the wider workstream 1 content.

RCT B will be a combined evaluation of Workstream 2 (a new notification form) and pre-notification elements of Workstream 1. A separate trial protocol will be published for RCT B.

The diagram below provides more context as to where RCT A sits in the user journey.

RCT A starts at post-notification in the customer journey. Conciliators are randomised to treatment and control, and cases are allocated as usual to conciliators. The treatment conciliators use new content at their discretion and the control conciliators continue with business as usual. The RCT ends after conciliation ends.

The logic model, showing the key outcomes which can be measured for the Workstream 1 content, is summarised below. As noted, RCT A focusses solely on the "conciliate" post-notification content.

The logic model for RCT A is a visual representation of the rationale for the post-notification content changes, the assumptions underpinning the model, and portrays how conciliators trained in using new content is expected to lead to 4 short-term outcomes for claimants, that:

• they understand the EC timetable
• they can access the right information at the right time
• they feel less stressed by the EC process
• conciliation is off to a good start

Together, those are expected to lead to claimants being better informed about and satisfied with the EC process. Ultimately, claimants have a better application experience and the service settles more cases in the EC window.

Please note the mechanisms refer to mechanisms of change. These are the point of connection between specific activities and outcomes that Acas believes will bring about the outcomes. The hypothesis is that these will bring about specific outcomes and the absence of them, or different mechanisms, will not bring about the intended outcomes. In the case of RCT A, the mechanism for change are conciliators issuing informative content, proactively and in a targeted way in response to the needs of their cases, at their discretion.

Objectives

The primary research question to be addressed in the impact evaluation for Workstream 1 is:

1. What is the difference between the number of cases settled in the six-week EC window (outcome) and cases randomised to new content (intervention), compared to cases randomised to control (business as usual; control) conditions?

The secondary research questions are:

1. What is the difference in the experience with the overall process (outcome) between claimants and representatives randomised to new content (intervention), and claimants and representatives randomised to control (business as usual; control) conditions?

2. What is the difference in the efficiency of the EC process (outcome) between cases randomised to new content (intervention), and cases randomised to control (business as usual; control) conditions?

3. What is the difference between perceived knowledge about the whole journey and process reported by claimants and representatives that are randomised to new content (intervention), and cases randomised to control (business as usual; control) conditions?

4. What is the difference between understanding of the timetable after being assigned a conciliator reported by claimants and representatives that are randomised to new content (intervention), and cases randomised to control (business as usual; control) conditions?

5. What is the difference between feeling that they have access to the right information at the right time reported by claimants and representatives that are randomised to new content (intervention), and cases randomised to control (business as usual; control) conditions?

6. What is the difference between perceived levels of stressed during the EC process after being assigned a conciliator reported by claimants and representatives that are randomised to new content (intervention), and cases randomised to control (business as usual; control) conditions?

7. What is the difference between agreement that conciliation got off to a good start after being assigned a conciliator reported by claimants and representatives that are randomised to new content (intervention), and cases randomised to control (business as usual; control) conditions?

The trial will be complemented by a parallel-running, process evaluation. This is discussed in more detail in Section 5.

Trial design

Randomised Control Trial (RCT A) focusses on measuring the impact of conciliators signposting some of the new content from Workstream 1 (through new templated emails). These will be provided by conciliators on a discretionary basis after notification (the "conciliate" post-notification content).

Acas will randomly divide all conciliators into an intervention and control group (more details on how they will be divided is provided later in this section). After notification, cases will be assigned through normal business processes to the 2 random groups.

For the duration of RCT A, only the videos and discretionary materials for conciliators will be live (meaning other elements of the Workstream 1, pre-notification content will not be live during RCT A).

The different elements of Workstream 1 content, the ways in which they will be evaluated and the structure of RCT A are summarised in the figure below.

The Workstream 1 content explored through qualitative research is the more streamlined content throughout the website. The content explored in RCT B is reworking the tell.acas.org pages and the additional content embedded in the notification form. The content explored in RCT A is the materials for use by conciliators. RCT A involves customers submitting a legacy notification form with the normal case allocation to an intervention and control group. The intervention group uses the new content and the control group does not.

The impact study design is an "intend to treat" trial, as conciliators will use the new content at their discretion when they feel it will best help claimants and representatives. Claimants and representatives will be allocated to intervention and control at random on a 1:1 basis.

All outcomes will be recorded by Dynamics 365 (Acas's Case Management System) and an online survey 6 weeks after the EC window opens (these are mapped in the evaluation framework).

Timings

RCT A will run for a period of 4 weeks between 25 July and 19 August 2022. This is the longest it could be run within the parameters of the Smarter Resolutions programme implementation plan and the evaluation timescales. It also allows for sequential evaluations of other programme elements (discussed in accompanying trial protocols).

Participants

The participants in this trial are claimants and representatives who have agreed to take part in EC. There will be no exclusions during the trial period.

Interventions

As noted, there are 4 key points in the claimant journey at which Workstream content 1 will be embedded. This section focusses only on the intervention content provided post-notification, by conciliators, the impact of which is being evaluated through RCT A.

Mechanisms

This new post notification resource aims to improve the information and context to claimants and representatives, to enable a better understanding of the next steps, the timelines and what will be required from them. It will sign-post people towards other appropriate resources, including new video content – ensuring they can access the right information at the right time.

Conciliators in the treatment group will be trained to use the new content at their discretion during the trial. They will use the materials when they feel it is appropriate to do so. Therefore, some people in the treatment group might not receive the content or engage with it. The extent to which conciliators share the content will be explored further in the qualitative interviews. It is in this context that the impact study design is an "intend to treat" trial.

Conciliators in the control group can in theory access the new materials but will be explicitly asked not to do so and Acas will monitor usage of the new content to ensure no contamination between the control and treatment group.

The table below summarises the content available to conciliators to provide to claimants and compares with that available to the control group.

Table 1: How the control and treatment content differ

Stage in the user journey	Control	Treatment
Allocation to conciliator	Legal introductory email, manually issued by conciliators	New, automatically issued introductory email with a video link included (2 mins long)   Video explains what happens during Acas early conciliation and the role of the conciliator. It also explains what the claimant needs to do next.
Discretionary emails which can be sent at any point during EC	Not available	Tailored follow-up emails depending on needs:   "Following our conversation please find linked below information which I hope you find useful". Emotional support page Time limits page Link to wages video (2 mins long) Video explains the difference between wages and expenses, and when an employer can legally take money out of wages.
End of conciliation	Legacy auto early conciliation certificate	New auto early conciliation certificate – with a video link included (1 min 40 seconds).   Video explains what the early conciliation certificate is and how it can be used.

Outcomes

It is expected that the Workstream 1 post-notification intervention will lead to the following short-term outcomes:

• more claimants/representatives having a better understanding what the timetable is after initial contact with a conciliator
• more claimants/representatives having access to the right information at the right time
• fewer claimants/representatives feeling stressed and worried after initial contact with a conciliator
• more claimants/representatives feeling that conciliation has gotten off to a good start

Mid-term outcomes are expected to lead to:

• an improved understanding among claimants/representatives of the full journey and what the key steps in the EC process will be
• a more efficient EC process
• more claimants/representatives satisfied with the EC process

This should lead to the longer-term impacts of:

• more cases are settled during the 6 weeks of EC and fewer reach ET. This is the primary outcome for RCT A
• an improved overall application experience for claimants/representatives

Evaluation framework

The impact will be measured through the following outcomes and indicators.

Table 2: Evaluation framework

Audience	Outcome	Indicator	Source	Timings
Claimants/representative	Understand EC timetable	% of claimants understanding what timetable is after initial contact with conciliator	follow-up survey	6-week follow up
Claimants/representative	Access right information at the right time	% agreeing they had access to the right information at the right time	follow-up survey	6-week follow up
Claimants/representative	Feel less stressed by the EC process	% feeling stressed or worried after initial contact with conciliator	follow-up survey	6-week follow up
Claimants/representative	Conciliation off to a good start	% agreeing the process got off to a good start after initial contact with conciliator	follow-up survey	6-week follow up
Claimants/representative	Better informed about the EC process	% of claimants believing service reflected expectations % with remaining questions about EC process (and nature of questions % of claimants agree award or outcome matched initial expectations	follow-up survey	6-week follow up
Claimants/representative	Satisfied with the EC process	% satisfied with overall EC process	follow-up survey	6-week follow up
Service	More efficient conciliation process	Average time between initial conciliator conversation and outcome or resolution	CMS	After 6 weeks
Service	More claims are settled at EC	% of cases settled at EC stage % of cases reaching ET	CMS	After 6 weeks

Data will be drawn from 2 sources:

• Dynamics 365, Acas's Case Management System (CMS) – for the 2 service outcomes
• an online survey of claimants – 6 weeks after EC begins – for the remainder of the outcomes

We will be able to link the trial arm to both the CMS and online survey data.

Sample size

In this section, we estimate the precision of inferences that we expect to achieve with different sample sizes. The sample size calculation requires assumptions that typically cannot be rigorously tested until the data has been collected. However, using assumptions based on past experience is a crucial part of design, establishing the level of confidence that could be associated with empirical work.

The sampling frame for RCT A are the customers who submit a legacy notification form. Over a 4-week trial, this will equate to around 6,800 cases, of which:

• 3,400 will be placed in the control group
• 3,400 will be placed in the intervention group

According to Ipsos MORI (2020) in the evaluation of Acas Individual Conciliation 2019, conciliation had taken place in around half of all cases, and this is the assumed control-group rate for calculations on whether sample sizes are sufficient.

As explained later, the trial uses a clustered randomisation. Rather than randomly allocating individual cases to the 2 arms of the trial, conciliators are allocated randomly and then all cases that the conciliator progresses will be in the control or intervention group, depending on which group the conciliator is in.

This means the trial is clustered, with a cluster of cases associated with each conciliator. The trial – as it provides evidence of outcomes such as the proportion starting a claim – would test any changes in a cluster-randomised design with a 2-sample proportion test. Power analyses are useful in giving a sense of the size of the effects one could reasonably expect to demonstrate with a study of a given size. Our assumptions are significance level of 5%, power of 80%, balance or equal-allocation design (same number of individuals in the control and experimental group).

The clusters are defined at the conciliator level, who will be equally allocated to the treated and control group, which will give 182 clusters in each group. The number of cases per cluster will be around 18 for the treated and control.

We assume an intraclass correlation of 0.5. The test also assumes that the experimental-group proportion is larger than the specified control-group value. We used the large-sample Pearson's and likelihood-ratio tests commonly used to test hypotheses about 2 independent proportions. Additionally, we calculate the Fisher's exact test commonly used to compare the 2 proportions in small samples. We have performed all the calculations with the 3 tests and the results are consistent.

The analysis defines the primary outcome as an increase in cases settled during the 6 weeks of EC and fewer reaching ET. Table 3 presents the effect size for the given sample sizes using the control group proportion of 50%; delta gives the difference between proportions (the minimum detectable effect). Thus, if we run RCT A for 6 weeks to obtain 80% power for a 95% confidence interval, the smallest difference that can be detected is 0.106 corresponding to a proportion of 0.606 in the experimental group. This means the smallest difference that can be detected between the 2 groups is 10.6 percentage points.

Table 3: Power calculations – experiment 1

Length	4 weeks
Sample	6,552
Number of clusters	182
Cluster size	18
Proportion control	0.50
Proportion experiment	0.6059
MDE (delta)	0.1050

2-sample proportion test: Pearson's chi-squared test

Claimant/representative survey

The claimant/representative survey relies on people agreeing to take part in the research and completing the survey. After the 6-week period is reached we will send an email invite and up to 3 reminders to maximise response to the survey. This will be followed by a period of telephone calling to further boost responses. No incentives are being provided for participation in the survey.

We are anticipating an overall response rate of 33% (25% achieved from initial email invites / reminders and a further 8% achieved through a telephone reminder). Our assumptions are based on a survey conducted in 2021 evaluating Acas's digital advice (46% response rate) and a previous evaluation of EC users (33% response rate). The survey response rate assumptions are set out in the table below.

Table 4: Survey response rate assumptions

	Control	Intervention
Starting sample	3,400	3,400
Online response rate	25%	25%
Achieved online responses	850	850
Sample for telephone reminder	2,550	2,550
Telephone response rate	10%	10%
Achieved telephone interviews	255	255
Total achieved sample	1,105	1,105
Overall response rate	33%	33%

An achieved sample of 1,105 in both control and intervention groups means we will need to see a difference of between 2.5 and 4.2 percentage points between the 2 groups (at the 95% confidence level) for the difference to be statistically significant.

As cases are allocated on a BAU system and conciliators are randomly allocated to the 2 arms of the trial, the non-response to the survey should be the same in both arms and should not introduce bias. However, we will investigate the need for non-response weights after a review of overall response.

We will also check that the mode of completion and date since EC began do not have an impact on responses (some people may complete the survey between 6 and 9 weeks after notifying Acas).

Randomisation

The randomisation will be conducted at the conciliator level. All 332 conciliators that work on individual cases will be randomly divided into 2 subsamples. All members of the first subsample will then consider the cases allocated to them to be assigned to the treatment and all the cases of the second subsample will be assigned to the control group.

We are randomising at conciliator level because splitting by case would mean conciliators would need to switch between handling cases in 2 different ways. Splitting by case also would create a risk that conciliators would want to provide the treatment content, regardless of case allocation to treatment or control. Other options considered, such as randomisation as cases were initiated online, also proved impractical. For example, from first landing on the tell.acas site, it is not possible to be routed to treatment or control pathways with a certainty that an individual – perhaps restarting a claim – would not be allocated to the alternative group.

Randomisation of conciliators means this design is clustered. 3 steps have been taken to increase the validity of inference, such as integrating baseline balance across the 2 treatment groups of conciliators:

• A randomisation tool has been developed that – as well as listing the conciliators in scope for the trial – includes the location, grade and employment type (full-time/part-time) of conciliators. As the treatment relies on conciliator discretion to send materials it is important that that the 2 groups are evenly matched in terms of their background experience. The tool randomises the selection from groups of similar conciliators, so that the baseline characteristics were balanced on these variables.
• A performance measure at conciliator level is provided by Acas for some of the conciliators, reflecting the proportion of cases in which a conciliator reaches a COT3 settlement. Some conciliators had moved to a different role but were still appearing in the conciliator staff list, and others work in teams and so individual performance is not captured; performance is measured at the team level for these individuals. A higher rate indicates better performance. The performance tool ensures the availability of performance evaluations are known for the same number of conciliators across treatment and control. This was used to check the randomisation process (rather than to define the groups of similar conciliators), allowing the average score calculated for treatment and control to qualitatively assure balance at a broad level.
• We are interested in the average treatment effect of the intervention, considering that we know which customers will receive the new discretionary content and which will receive the legacy content. As the trial progresses, we will compile evidence about the content claimants see to observe outcomes, and to check the claimants and representatives supported by conciliators in the control group have not accessed content available to those support by conciliators in the treatment group. For example, collecting data about who calls the helpline that is only mentioned in the new content and seeing if those in the control group have accessed this.

Attrition is the absence of data from participants, and it can be a threat to the internal validity of the experiment. Attrition of data for both the control and treatment group reduces the power of the model in obtaining statistically significant results. Attrition bias can arise if there are differences in the attrition rates for the control and treatment group - for example, if conciliators from either group left their roles. The trial period is relatively short, which helps to minimise this risk, however conciliator leavers should be balanced by introducing new joiners into the same treatment or control group as the conciliator is being replaced. Those providing cover for conciliators during annual leave will only use the legacy content.

Sequence

This is the first and only stage in the RCT. There will not have been any prior allocation stage for this RCT.

Case allocation

From an operational point of view, the allocation team (who allocate EC cases to conciliators) are primed to use their normal, 'business as usual' approach to case allocation. This is not completely random – for instance, in some cases where a conciliator already holds a related case, a subsequent case that is received will go to the same conciliator. However, the allocation team will not know which conciliators are in which group, so randomisation will underpin the allocation of cases.

Around 10% of cases are allocated to 3 "blended" teams. These are teams of around a dozen conciliators that handle cases jointly, so that those making the claim may interact with any team member and the team have ways of working that allow different conciliators to progress a case. The cases allocated to the blended teams are in jurisdictions that can be handled quickly, and where the information available at allocation is relatively complete. While not randomised, the trial will allocate 2 smaller teams to the treatment and one to the control, meaning that around 60% of the cases allocated to the blended teams will be in the treatment group. The 3 teams are judged by Acas to be similar operationally and in the cases they take.

Implementation

Before the trial, we will check:

• sample selection and recruitment
• randomisation
• secure data capture procedures
• appropriate ethical framework

We use the National Statistics Data Ethics guidance as an overarching framework. There is a public good to the study, in that evaluating the effectiveness of improvements to the conciliation processes will assure better service delivery and value for money. That benefit will be to a wide set of the UK population, in that the Acas remit of all employees and employers does not target a sub-population. Data handling and processing will be undertaken so that the data subject's identity (whether person or organisation) is protected, information is kept confidential and secure, and the issue of consent is considered appropriately.

The development of this RCT has carefully assessed the risks and limits of new technologies. In particular, several decisions to de-scope aspects have been taken, such as using an RCT at an early stage in the process testing knowledge sharing only. This recognised that at later stages the conciliation process limited any experimental design. Data use will be consistent with personal data legislation. The data used is personal but not highly sensitive, focusing mostly on the process of conciliation in terms of its efficiency and participant views. The publication of the RCT protocol, alongside other research dissemination, will ensure that views of the public are considered proportionally in light of the data used and the perceived benefits of the research.

During the trial, we will check effective compliance and attrition-reduction procedures.

The study is grounded in the administrative data associated with Acas processes. Overall, administration data linked to surveys provides a robust evidence base for impact analysis because attrition is not present in the administrative data and the linked dataset provides many opportunities to identify and correct for attrition in the surveys, where considered a problem.

Blinding

The allocation rule will be blind for the claimant and representative. There is a risk that independent participants might interact and become aware – if they are in the control – that there is a treatment. We judge this risk to be small, although it is likely that there will be a relatively small number of claims involving representatives or large employers, which are involved in work across several claims and could experience the treated and control arms of the trial. Case allocation does note cases being related, with allocation often being to a conciliator with a related case, potentially mitigating this risk somewhat.

As noted above, the trial will also be blind for case allocators.

Conciliators will know which arm of the trial they are in, as the treatment arm relies on them being briefed to use the new materials. Technically, it will be possible for conciliators in the control group to find the new materials, but Acas will be explicitly asking them not to do so.

Statistical methods

In the context of an experimental estimation, whether to include control variables or covariates depends on the success of the randomisation.

If the randomisation is successful, those receiving the treatment differ from those that did not only in the fact of receiving the treatment. Randomisation should mean all other factors (both observed and unobserved) are identical. This means the first analysis of results would be a simple difference-in-means analysis, asking whether the treated reached better outcomes than the those in the control.

Some remaining imbalance in observed and unobserved variables is possible even after randomisation: the cases may not prove to be in exactly the same mix of jurisdictions, type of claimant, and employer type across the 2 arms of the trial. It is then possible that the control variables explain a lot of the outcome, and their inclusion could reduce residual variance and raise precision (reducing standard errors) of estimated treatment effects. In the case of RCT A, the possible covariates are: status of the customer (claimant or representative), region where the company is located, sector of the company, the length of service to the company, age of the individual.
 
There are 3 principal roles of covariates:

1. Make analyses more informative, so the results will be more precise if the covariates are sufficiently strongly correlated with the outcomes.
2. If the randomisation was compromised, adjusting for covariate differences may remove biases.
3. Randomisation only works perfectly on average. We propose to present estimates of the treatment effect side-by-side with and without the control variables.

Risks

Risks	Mitigating actions
Wider content and videos can be accessed by people through the control arm	The evaluation is seeking to assess the impact conciliators have in sign-posting people towards the materials using new bespoke materials, so people see “the right information at the right time”. The theory is that people being signposted towards the videos will find them more useful that those who come across them by chance in the control arm, and this will be assessed in the evaluation.
Conciliators are not blind to the trial – risk that conciliators in the control group use the new materials/signpost towards content	Conciliators in the control arm will be briefed not to use it. In addition, Acas can monitor use of the content and so Acas will intervene where any control arm conciliators are found to be using the new content.
Inconsistency in terms of how conciliators use the discretionary content	All conciliators will be briefed and encouraged to use the content, but this is at their discretion. The usage of the new content can be monitored.
Case allocation is not completely random	The allocation team are primed to use their normal approach to case allocation. It is important to note that this is not completely random – for instance, in some cases where a conciliator already holds a related case, a subsequent case that is received will go to the same conciliator. However, the allocation team will not know which conciliators are in which group, so randomisation will underpin the allocation of cases.
It is possible response rates will be lower than expected to the follow-up survey	Assumed response rates are based on previous published surveys by Acas following a similar method. Several steps will be taken to improve response rates, including reminder emails and calls, and setting up a helpline for any participant queries. If response rates are lower than anticipated, we will need to see a larger difference between control arm / treatment arm for us to attribute it to the new content.

We will also monitor for a mix of participants by region, claimant / representative, reason for dispute and sector.

We will recruit individuals for the qualitative work through a permission to recontact question at the end of the follow-up claimant/representative survey.

Method

All interviews will be conducted using video-conferencing software and last around 40 minutes each. Interviews will be conducted by the IFF Research team.

All interviews will be recorded and transcribed securely (with express permission). For participating, interviewees will be provided with an incentive of £40.

We will use journey mapping techniques and ask individuals to view the videos/emails (or show screenshots) to prompt them to recall their experiences.

Analysis

The analytical approach for the qualitative research will be iterative and inductive – building upwards from the views of participants, incorporating elements of 'grounded theory' analysis, for example the thematic review and continual analysis of hypothesis from participants' dialogue and researchers' impressions of the discussion (for example, pauses and tone).

Analysis will begin informally during fieldwork itself. Our research team will work closely together throughout the fieldwork period, feeding back headline findings to each other as discussions are conducted, and continually updating our approach and thinking as we amass data.

All interviews will be written up in detail, including verbatim quotes, in an analytical framework in Excel. The framework will be structured around the logic model and research questions, and include key sample data, to allow for comparison of findings by different characteristics.

The data will then be analysed to search for themes and trends, both present and absent. Once qualitative analysis is complete, we will then compare those findings with other evidence to challenge and address gaps. Director-led analysis sessions will bring this thinking together, encourage challenge of assumptions and identify areas for further, targeted analysis.

In addition, we will conduct 10 in-depth interviews with conciliators to discuss their experience of using the new Workstream 1 content.

Sample and recruitment

These will all be conducted with conciliators who have used the new content, as they will be able to talk about their experiences both pre- and post- its introduction. All conciliators will be informed of this strand of the research and will be invited to participate by IFF Research.

The 10 interviews will incorporate a mix of conciliators, in terms of years' experience and specialisms. This will allow us to monitor how materials were shared and used differently with claimants depending on their experience and background, or the types of cases on which they work.

Method

All interviews will be conducted using video-conferencing software and last around 40 minutes each. Interviews will be conducted by the IFF Research team.

All interviews will be recorded and transcribed securely (with express permission). No incentives will be provided. All feedback will be confidential and not attributable to any conciliator.

Analysis

All interviews will be written up in detail, including verbatim quotes, in an analytical framework in Excel. The framework will be structured around direct experiences of implementing the content:

• how the content was implemented – in particularly given the discretionary nature – and how often and when it was shared?
• testing assumptions in the logic model – activities and mechanisms for change
• reception from claimants
• how content could be improved and rolled out further?

The data will then be analysed to search for themes and trends, both present and absent.

Once qualitative analysis is complete, we will then compare those findings with other evidence to challenge and address gaps. Director-led analysis sessions will bring this thinking together, encourage challenge of assumptions and identify areas for further, targeted analysis.

The intervention implementation, mechanisms for change and assumptions will be tested through this analysis.

Funding

The pilot and evaluation are being funded through HM Treasury's Shared Outcomes Fund, which was announced at Spending Review 2020.

The Shared Outcomes Fund has been designed to fund pilot projects to test innovative ways of working across the public sector, with an emphasis on thorough plans for evaluation.

Spending Review 2020 announced £200 million of funding, made available for a second round of the Shared Outcomes Fund between 2021 to 2022 and 2023 to 2024.

The funding for this pilot is £5 million.

The stakeholders are:

• Ministry of Justice
• Department for Business, Energy and Industrial Strategy
• Advisory, Conciliation and Arbitration Service (Acas)