Smarter Resolutions programme: evaluating Workstream 3 (optimised distribution) and Workstream 4 (large group claims)

Background

The Advisory, Conciliation and Arbitration Service (Acas) provides free and impartial information and advice to employers and employees on all aspects of workplace relations and employment law. 

Employment tribunals make decisions about employment disputes. The voluntary early conciliation (EC) process has been designed so that Acas can provide an impartial conciliation service. This is with the aim of parties reaching a settlement and avoiding the potentially time consuming, costly and stressful employment tribunal process.

In 2021, the Ministry of Justice (MoJ) and the Department for Business, Energy and Industrial Strategy (BEIS) were awarded funds over a two-year period (2021 to 2023) from the HM Treasury (HMT) Shared Outcomes Fund. The funds were awarded to work with Acas to test a series of early interventions in the employment dispute resolution system (the 'Smarter Resolutions' programme). The Smarter Resolutions programme is comprised of 4 operational workstreams that focus on improving different parts of the early conciliation process (see background to the Smarter Resolutions programme). The 4 workstreams are:

• Workstream 1: content strategy
• Workstream 2: assisted notification
• Workstream 3: optimised distribution
• Workstream 4: large group claims

Each workstream explores how early interventions can help Acas to improve the service offered to customers, to resolve disputes more quickly and effectively, therefore preventing the escalation of issues, improving the experience of individuals and reducing the flow of cases into the employment tribunal system. It is expected that this will deliver cost savings across government and arm's length bodies such as Acas. 

This protocol details how we will assess the impact of 2 parts of the 4 workstreams, Workstream 3 and Workstream 4, using pre- and post-implementation comparison approaches and qualitative interviews.

A randomised control trial (RCT A), which has no impact on this evaluation, has tested elements of Workstream 1. Likewise, randomised control trial B – which immediately followed randomised control trial A – is currently underway to evaluate part of Workstream 1 and all of Workstream 2. Results for both randomised control trials are not yet available. The evaluation of Workstream 3 will happen after randomised control trial B is completed. An evaluation of Workstream 4 will happen alongside randomised control trial B as delivery of the workstream has taken place, and Workstream 1 or Workstream 2 outcomes have little bearing on Workstream 4.

Workstream 3: optimised distribution

Intervention

Workstream 3 saw the development of the following interventions:

• Improved capture of and system for logging conciliator availability, location and work profiles (for example, expertise, preference, reasonable adjustments), including integration of systems capturing these pieces of information. This is to reduce the time burden on the conciliation allocation team (CAT) members who previously had to look across multiple data sources, and rely on strong communication strategies with conciliators to determine the best allocation.
• Development of an algorithm which automatically organises and distributes cases to the right conciliator or team according to this information. Most case distribution was done manually by conciliation allocation team members. The new algorithm should also take care of administrative issue like finding, merging, and deleting duplicated claims and filtering cases with respondents that have ceased trading. Previously, these instances relied on human processing and identification.

Objectives

The aims of the Workstream 3 evaluation are to:

• test whether changes to and integration of the systems for capturing service information allows Acas to monitor conciliator resource more effectively and accurately 
• test whether an automated algorithm for categorising cases allows more cases to go to the most suitable conciliator more quickly

Participants

The participants in this evaluation are:

• claimants and representatives who notify Acas they wish to use early conciliation within the given three-week post-implementation window (service user survey, qualitative interviews). These cases will be statistically matched to pre-implementation cases for comparison.
• conciliation allocation team members and conciliators (qualitative discussions)
• employer representatives where claims related to them have been raised within the given window (qualitative discussions)

Method and design

The impact study is a quasi-experimental design using cases that fall into the three-week period following 19 January , known as the 'post-implementation' period. These will be statistically matched to cases of similar characteristics in the 'pre-implementation period' (that is before any changes were made to processes and systems as part of the Smarter Resolution programme). These cases will be drawn from the period January 2022 to May 2022. The estimated sample size for the treated cases is around 3,000 cases, with another 3,000 in the matched pre-implementation group.

Outcomes are recorded by Dynamics 365 (Acas's case management system) and an online survey with post-implementation service users 6 weeks after claimants notify. 

The primary outcomes are:

1. Percentage of cases allocated to an individual conciliator without manual intervention from the conciliation allocation team.
2. Time elapsed from notification to initial allocation to an individual conciliator.
3. Percentage of cases 'informally resolved' within the six-week early conciliation period.

Qualitative interviews will complement the impact evaluation. We will conduct qualitative discussions with 6 service users (claimants and claimant representatives) and 6 employer representatives in the post-implementation group to explore their experience of early conciliation. We will also conduct 2 focus groups with conciliators and conciliation allocation team members to discuss their experiences and perceived impacts of the new system and processes. The intervention implementation, mechanisms for change and assumptions will be tested through this analysis. 

Workstream 4: large group claims

Intervention

In October 2021, Acas created a dedicated group claims team (GCT) to deal with larger cases comprising claims involving 10 or more claimants. Previously, group claims were handled by individual conciliators and then 'blended teams'. A blended team consisted of civil service Grade 10 staff who worked on cases that could usually be dealt with quicker than others (known as 'FAST' cases).

Workstream 4 saw the development of the following interventions aimed at improving the work of the group claims team:

• Auto-allocation to the group claims team (any group cases where there are more than 10 claimants should be automatically assigned to the group cases team and an email is sent to the claimant representative). Previously, all allocation was done manually by conciliation allocation team members.
• Changes to the group notification form, to enable claimant representatives to add additional information and give permission for conciliators to speak to respondent(s) first. Previously, conciliators had to seek this permission in first contact with the claimant, reducing the timeframe in which conciliators could engage with respondents (employers or employer representatives). 
• A respondent allocation report which helps the group claims team spot cases with 'same facts same respondent'. Previously, conciliation allocation team members were often trying to identify cases that were in the early conciliation support officer (ECSO) queue that had 'same respondent and same facts', but had to manually open each case to look at the jurisdictions.

Objectives

The aims of the Workstream 4 evaluation are to:

• test whether greater automation increases the time available for resolution
• test whether greater automation allows for consistent involvement of early conciliation individuals or a team in large group cases with the same respondents

Participants

The participants in this evaluation are:

• employee representatives who notify Acas they wish to use early conciliation, and who have their case allocated to and handled by the new group claims team, within the given three-week post-implementation window (qualitative interviews)
• group claims team conciliators and managers – conciliators will be a mix of those who understood and experienced the previous processes for identifying and handling large group claims (in other words, exposed to pre-implementation), and those who are new to the processes
• employee representatives responding to large group claims, who have pre-and post-implementation exposure to Acas large group claim processes (qualitative interviews)

Method and design

The impact study is a quasi-experimental design comparing cases that fall into the pre-implementation period and post-implementation period. The pre-implementation period is January to April 2022, prior to the auto-allocation of group cases, the initial new group notification form and the respondent allocations report being in place. The post-implementation period is October to December 2022, after the implementation of these interventions.

Outcomes are recorded by Acas's case management system, and qualitative interviews and focus groups with Acas staff and employee representatives.

The primary outcomes are:

1. Large cases picked up more quickly.
2. More effective and efficient handling of large cases (for example less work done by other conciliators before cases are transferred to the group claims team).
3. More large cases settled in conciliation.

Qualitative interviews will complement the impact evaluation. We will conduct 2 interviews with the group claims team managers, a focus group with group claims team conciliators and 20 in-depth interviews with employee representatives. The intervention implementation, mechanisms for change and assumptions will be tested through this analysis.

About Acas

Acas provides free and impartial information and advice to employers and employees on all aspects of workplace relations and employment law. 

Employment tribunals make decisions about employment disputes. Nearly all legal cases about employment are heard in employment tribunals. This includes cases about unfair dismissal, redundancy and discrimination. There are also many other sorts of claim that can be brought, such as non-payment of wages. Acas has a longstanding statutory duty to promote the resolution of claims before the employment tribunal in order to avoid recourse to a full hearing, by means of its conciliation service. In April 2014, a new legal requirement was introduced that means individuals must notify Acas of their intention to make an employment tribunal claim. Since this change in the law, Acas offers the opportunity for both parties involved in the dispute (that is the individual and the employer) to resolve the issue through a new voluntary early conciliation process. Early conciliation notifications can be made by an individual against their employer or on behalf of a group against their employer.

The early conciliation process has been designed so that Acas can provide an impartial conciliation service with the aim of parties reaching a settlement and avoiding the potentially time consuming, costly and stressful employment tribunal process. Cases progressing through early conciliation will be assigned an impartial conciliator who may:

• explain the conciliation process
• discuss the issues with both sides
• give an overview of the law
• discuss how employment tribunals have considered similar cases
• help both sides explore the strengths and weaknesses of the case
• discuss possible options without making any recommendations
• remain independent of the dispute – they will not take sides or tell either party what to do 
• outline the employment tribunal process
• record a binding settlement of the legal dispute in the form of a conciliation agreement (COT3) so that the case does not need to proceed to a hearing

If the offer of early conciliation is refused or fails to reach agreement, the individual can still pursue an employment tribunal claim. In this case, the respondent usually has to reply to a claimant's claim in writing within 28 days of getting the claim form. Once they've replied, the tribunal will decide whether there will be a full hearing. These cases are normally held at the employment tribunal office closest to where the claimant lives. All tribunal hearings will be heard by a judge and the respondent, and depending on the type of case, there may be 2 other tribunal members.

Background to the Smarter Resolutions programme

In 2021, the Ministry of Justice (MoJ) and the Department for Business, Energy and Industrial Strategy (BEIS) were awarded funds over a two-year period (2021 to 2023) from the HM Treasury (HMT) Shared Outcomes Fund. This was in order to work with Acas to test a series of early interventions in the employment dispute resolution system (the 'Smarter Resolutions' programme).

The Shared Outcomes Fund incentivises and funds innovative pilots, with departments working collaboratively to improve cross-governmental outcomes. It places a high value on robust, high quality impact evaluation and sharing results. 

The Smarter Resolutions programme is comprised of 4 operational workstreams that focus on different parts of the early conciliation service, which is Acas's service for resolving individual disputes. Each workstream explores how early interventions can help Acas to improve the service offered to customers, in order to:

• resolve disputes more quickly and effectively, therefore preventing the escalation of issues
• improve the experience of individuals
• reduce the flow of cases into the employment tribunal system 

It is expected that this will deliver cost savings across government and arm's length bodies such as Acas. 

The overall programme vision is that people involved in a workplace dispute have a clearer understanding of how they can reach a quicker, more cost-effective resolution at the earliest point possible. The overall outcomes are:

• parties spend less time in the employment tribunal system
• fewer cases requiring judicial time
• conciliators focused on resolution not administration
• parties engage in discussion with conciliators which is focused on resolution

The 4 workstreams and workstream visions are:  

• Workstream 1: content strategy. Users grasp basic principles of employment law related to their dispute, the most appropriate way to resolve it, and have clear expectations of where Acas can help.
• Workstream 2: assisted notification. Users are guided to supply information that conciliators need to provide the best possible service at the earliest point.
• Workstream 3: optimised distribution. Cases are directed to the most appropriate and available conciliator to quickly and efficiently resolve their dispute.
• Workstream 4: large group claims. Users who belong to the same workplace dispute are quickly brought to the attention of a team with the expertise, tools and resources to resolve efficiently.

This evaluation plan relates to assessing the impact of Workstream 3 and Workstream 4.

Assessing the impact of Workstreams 3 and 4

Under Workstream 3, Acas has developed a new algorithm for distributing cases to conciliators, replacing a manual process. This is so that cases are directed to the most appropriate and available conciliator to quickly and efficiently resolve the dispute. 

Underpinning this optimised distribution are improvements to both processes and tools for capturing data, including:

• improved capture of and system for logging conciliator availability, location and work profiles (for example, expertise, preference, reasonable adjustments)
• improved quality of data coming through from the notification form (an expected output from Workstream 2's improvements to this element)
• development of an algorithm which automatically organises and distributes cases to the right conciliator or team according to this information 
• the algorithm also takes care of administrative issues like finding, merging, and deleting duplicated claims and filtering cases with respondents that have ceased trading 

Changes made should reduce the administration work for conciliation allocation team members. This is the team which uses the jurisdictions and the associated track (fast, standard and open) to make sure conciliators receive cases in line with their training or experience. The changes should also reduce the need for reallocation so conciliation allocation team members can focus on more complex and exceptional cases.

Workstream 3 has been in development since late 2021, with continuous discovery and insight pieces throughout 2022. The algorithm went live on 19 January 2023. That said, insight work into how cases are processed manually, case information used for distribution, and systems for monitoring conciliator leave and work profiles have been impacted by learnings from this workstream since early May 2022.

Workstream 4 focuses on an optimised service for:

• identifying users who belong to the same workplace dispute
• quickly bringing them to the attention of the dedicated group claims team (GCT) with the expertise, tools and resourcing to resolve a case efficiently

These claims account for a relatively small number of cases but the number of claimants involved in each case means they affect quite a large number of individuals. 

All activities under Workstream 4 included in this evaluation have now been completed. These include:

• Auto-allocation to the team rather than a manual process. Any group cases where there are more than 10 claimants should be automatically assigned to the group cases team and an email is sent to the claimant representative. This went live in May 2022.
• Changes to the group notification form, to enable claimant representatives to add additional information and give permission for conciliators to speak to respondent(s) first. This started in December 2021, with the form going live in July 2022.
• A respondent allocation report which helps the group claims team spot cases with 'same facts same respondent'. This was introduced to the team in August 2022. 

Further work on the group form is planned for early 2023 and it is not possible to include in this trial due to timescales.

Workstream 3 logic and intended outcomes

When a notification form is submitted, a case can go through 2 stages before it reaches a conciliator. Firstly, the early conciliation support officer (ECSO) team ensures that jurisdictions (aligned with the nature of a dispute, for example wages or unfair dismissal) are on the case and are correct. The conciliation allocation team (CAT) use the jurisdictions and the associated track (fast, standard and open) to make sure conciliators receive cases in line with their training or experience.

Acas design teams and the service designer (Wyser) worked together in the discovery phase to determine current weaknesses in systems, processes and procedures related to case allocation. The findings showed little integration between key systems, such as Dynamics (Acas's customer relationship management system or CRM) and the CAT tool, which resulted in the need for more manual input. They also showed that:

• information around capacity and allocation was not always being captured
• the allocation process more broadly required a lot of manual intervention and human decision-making (due to missing jurisdiction and other information, errors in the system not being picked up, and linked cases not being recognised)

One intended outcome of Workstream 2 (assisted notification) is improvements to the quality and completeness of information provided in the notification form, thus improving accuracy of 'jurisdiction'. Workstream 3 seeks to address the above weaknesses specifically around case allocation and, overall, improve the efficiency and effectiveness of case allocation and resolution.

Intervention

The main features of the Workstream 3 intervention that will be evaluated are as follows:

• Improved capture of and system for logging conciliator availability, location and work profiles (for example, expertise, preference, reasonable adjustments). This includes the integration of systems capturing these pieces of information. Information related to conciliator workloads and availability was not collected consistently, and awareness and subsequent logging of it relied heavily on communication policy between conciliators and the conciliation allocation team. Furthermore, this information and information on conciliator location and work profiles were all stored in different data sources. This meant that the conciliation allocation team team had to look across all of these to determine the best allocation. 
• Development of an algorithm which automatically organises and distributes cases to the right conciliator or team according to this information. Previous to this, most case distribution was done manually by conciliation allocation team members. There was some basic automation whereby cases that were suitable for allocation to a 'blended team' were sent there; this automation will be discontinued shortly as Acas moves away from blended teams.
• The new algorithm should also take care of administrative issues like finding, merging, and deleting duplicated claims and filtering cases with respondents that have ceased trading. These instances relied on human processing and identification.

Figure 3.1 summarises the differences in allocation processes pre- and post-implementation. The goal here is a high-quality notification experience, with clear communications, and the ability to quickly provide the person making the notification with a named conciliator.

Currently, it takes 2 weeks to allocate to individual conciliators. The process is as follows.

1. Claimant notifies
2. Early conciliation support officer (ECSO) checks and prepares case
3. Route to conciliation allocation team queue
4. Allocation
5. Conciliator email sent
6. Conciliator contacts 

The new system and processes should reduce this process to 2 minutes, as follows: 

1. Claimant notifies
2. Automation checks and allocates
3. Introductory email sent
4. Conciliator contacts

The model in figure 3.2 captures our understanding of the workstream. This includes our assumptions of how the programme will operate and how claimants and representatives will experience it. The model illustrates the mechanisms for change and how activities are translated into impacts.

The rationale for Workstream 3 is that:

• if the systems for capturing information about conciliators work profiles (location, expertise, preference, reasonable adjustment), current caseloads and availability, are integrated, Acas will be able to monitor conciliator resource effectively and accurately
• if an automated algorithm for categorising cases, which also considers conciliator resource, is created, most cases can be automatically allocated and go directly from notification to the most suitable conciliator, requiring less administrative input from conciliation allocation team members

The assumptions underpinning the Workstream 3 logic model are:

• the new notification form and information coming through it will provide data accurate and complete enough for automatic allocation of most cases
• the algorithm will work effectively with all information available to assign to the most suitable conciliator
• information on conciliator work profiles, caseloads and availability are 'live' (in other words, up to date with all relevant information)
• conciliators will respond to their allocations at (or at least) the same speed as usual (in other words, the time saved from notification to allocation is not lost with lags in first contact)
• most claims close within 6 weeks after notification is submitted

To ensure the 'jurisdiction-to-track-to-conciliator' allocation works effectively, Acas needs correct, up-to-date information on all conciliators. As such, the first activity undertaken in Workstream 3 was a mapping exercise, creating work and case profiles for all conciliators. Other activities included:

• developing or enhancing processes and tools for capturing conciliator availability and work profiles (resulting mechanism – an effective system for monitoring conciliator resource, availability, and work profile)
• development of an algorithm to increase automation of allocations, and bringing these together (resulting mechanism – cases organised optimally via an improved engine)
• development of an improved automated allocation system (resulting mechanism – claim is automatically allocated to the most suitable conciliator or team) 

The key resulting mechanism underpinning Workstream 3 is that a claim is automatically allocated to the most suitable (available, correct grade, location, and so on) conciliator or team.

Note: The mechanisms refer to mechanisms of change. These are the point of connection between specific activities and outcomes that Acas believes will bring about the outcomes. The hypothesis is that these will bring about specific outcomes, and that the absence of them, or different mechanisms, will not bring about the intended outcomes.

The above outputs are expected to lead to short-term outcomes, at the point of case allocation. These short-term outcomes are:

• that most cases go direct from notification to conciliator or team
• a reduced time gap from notification to conciliator
• a reduction in the volume of cases distributed through conciliation allocation team
• reduced reallocation

We hypothesise that if these short-term outcomes are achieved, claimants and representatives will benefit in other ways, within 6 weeks of submitting a claim. These mid-term outcomes are:

• a reduced time gap between notification and conciliator contacting the service user
• claims resolved more quickly and efficiently
• a higher proportion of cases 'informally resolved' at early conciliation, as there is more time within the early conciliation window to focus on resolution as opposed to administration
• improved user experience (claimants, respondents and representatives)
• allocations perceived as 'fairer' by conciliators
• conciliation allocation team are able to focus on more complex or exceptional cases

Ultimately, the efficiencies introduced through Workstream 3 should result in:

• a reduction in administration costs (of conciliation allocation team)
• an improved early conciliation settlement or resolution rate
• a resulting improved employment tribunal return rate and improved related time and costs of cases at employment tribunal

Workstream 3 objectives

The primary research questions to be addressed in the Workstream 3 evaluation are:

1. What is the difference in the proportion of cases allocated to an individual conciliator without manual intervention from conciliation allocation team (outcome) post-implementation of Workstream 3 (intervention), compared with cases pre-implementation (business as usual: control)?
2. What is the difference in the time elapsed from notification to initial allocation to an individual conciliator post-implementation of Workstream 3 (intervention), compared with cases pre-implementation (business as usual: control)?
3. What is the difference in the average number of cases 'informally resolved' within the six-week early conciliation period (outcome), post-implementation of Workstream 3 (intervention), compared with cases pre-implementation (business as usual: control)?

The secondary research questions are:

1. What is the difference in the proportion of cases requiring reallocation (outcome) post-implementation of Workstream 3 (intervention), compared with cases pre-implementation (business as usual: control)? 
2. To what extent do Acas staff (conciliators and conciliation allocation team members) feel current and potential conciliator resource availability is being better estimated or aligned with allocations, and claims are being resolved more quickly and efficiently?
3. What is the difference in the time elapsed from notification to conciliator contacting service user and first conciliator conversation post-implementation of Workstream 3 (intervention), compared with cases pre-implementation (business as usual: control)? 
4. What is the difference in the average perceived satisfaction with the process of working with a conciliator (outcome) post-implementation of Workstream 3 (intervention), compared with cases pre-implementation (business as usual: control)? 
5. What is the difference in the proportion of cases conciliation allocation team are focused on that are 'complex or exceptional' post-implementation of Workstream 3 (intervention) compared to cases pre-implementation (business as usual: control)? 
6. What, if any, administrative cost savings (between notification and conciliation) are there post-implementation of Workstream 3 (intervention) compared to cases pre-implementation (business as usual: control)?
7. To what extent do Acas staff feel case allocations are fairer following Workstream 3 implementation?

Barriers to a randomised control trial

Ideally the impact of Workstream 3 would be tested through a randomised control trial (RCT) (similarly to Workstreams 1 and 2). 

However, various challenges to having a randomised control trial approach have arisen since the evaluation inception. 

Firstly, it would require developing and running 2 administrative systems (legacy and new allocation method). Running 2 systems would require significant development and testing work which is not possible in the timeframe available. Implementing 2 administrative systems more quickly, without fully testing them, would place too much risk on the delivery of the service. 

In addition, the insight work completed in the discovery and alpha phases has already resulted in better practice relating to manual distribution, and improvements in the system, process, and communication surrounding conciliation availability and work profiles (in other words, 'cultural changes' throughout). This is likely to have started in May 2022. As such, 'business as usual' in its current form is not a true representation of the pre-programme state. The impact of these cultural changes, and if or how they have value post-implementation (when, theoretically they should be replaced by automation), will be explored in the qualitative interviews.

It would also be challenging to deliver a randomised control trial within the timeframes available for the evaluation. The new method of allocating cases did not come online until 19 January 2023, while the evaluation and the service design work are each funded only up to the end of March 2023. Any pausing of work around Workstream 3 (as you would need for a 'control arm' of a randomised control trial) would delay full roll-out and decommissioning of old systems or processes required to finalise this workstream. It would also prevent this from being completed by the end of March deadline.

Evaluation design

The impact of Workstream 3 will therefore be assessed through a quasi-experimental design comparing outcomes pre- and post-implementation of the workstream. This is not as robust a design as a randomised control trial. Because we are drawing cases from different time periods, it is possible that the overall profile of cases or claimants will be different in a way which could impact on outcomes (and we know there is a degree of seasonality to Acas's business). It is also possible that different external factors will be at play that will impact both on this profile and on the way in which people behave in relation to their claims (for example, the cost of living crisis). We will use matching techniques based on known characteristics to try to correct for these differences as far as possible (our approach to this is outlined below), but we will need to be slightly more cautious in our conclusions than would be the case for a randomised control trial methodology.

Outcomes will be predominantly recorded by Dynamics 365 (Acas's case management system). Statistical matching between pre- and post-implementation cases will be used to understand the effect of the intervention, as all cases after a point in time will be allocated based on the new algorithm. 

The quasi-experimental design will be complemented by a service user survey 6 weeks after notification (in other words, after the end of the early conciliation window), and by a parallel running, process evaluation using qualitative methods.

Evaluation period

The pre-implementation period that we will use for the evaluation is January 2022 to May 2022. This period represents the best window for a 'return to the new normal' following covid-19 (coronavirus) disruption, and before any influence of cultural changes from the workstream. 

The post-implementation period will run for 3 weeks, starting when the algorithm went live on 19 January 2023. This is the longest the post-implementation period could run within the parameters of the Smarter Resolutions programme implementation plan (as outlined in challenges previously mentioned). This does mean that the post-implementation period will cover the very first week of rollout which is not ideal but necessary given the time constraints. To exclude it would lead to a large reduction in the sample size for analysis.

Participants

The participants in this evaluation are:

• claimants and representatives who notify Acas they wish to use early conciliation within the given post-implementation window (service user survey, qualitative interviews) – these cases will be statistically matched to pre-implementation cases for comparison
• conciliation allocation team (CAT) members and conciliators (qualitative discussions)
• employer representatives (qualitative discussions) where claims related to them have been raised within the given window

Outcomes

Acas expects the new availability and allocation systems, processes and algorithm will lead to the short-term outcomes shown in the logic model, which in turn will result in longer-term impacts. These outcomes and impacts for Workstream 3, and the associated indicators for measuring them, are summarised in the evaluation framework below. 

We will be able to compare pre- and post-implementation cases in both the CMS and online survey data. (For online survey data, this will be achieved by looking at the control group from randomised control trial A who had experienced very few of the changes relating to the Smarter Resolutions programme.)

Table 4.1 Indicators attributed to each outcome

To confirm that it will be possible to draw data on the suggested outcomes from the Acas CMS system, we have conducted some preliminary analysis on a small number of cases. This is outlined in Appendix A. 

Statistical matching to a control

The pre- and post- design approach to assessing impact will be improved by using statistical matching techniques.

Using a pre- and post- design means that the allocation of cases between the treatment group and the comparison group is not random (unlike for the randomised control trials we are using to evaluate Workstream 1 and Workstream 2). The lack of a randomised control trial design, however, will be replaced by 3 stages of analysis to provide a counterfactual:

• Dynamics data will be compiled for the period January 2022 to May 2022, when the automated algorithm was not in operation and no 'cultural' shifts had occurred to the manual processes, as a result of Workstream 3 discovery insights
• propensity score matching (PSM) will be used to select cases from this pool of 'untreated' cases, that are statistically similar to the cases that passed through Acas early conciliation after Workstream 3 is implemented
• analysis will also be undertaken looking at (the differences between) cases that are allocated through Workstream 3 and those that the Acas team allocate both to understand the extent to which the algorithm has been used and compare outcomes
• post-matching analysis will be used to look at any remaining imbalance in the samples after PSM

To some extent, just using the first stage can give a robust comparator. Before the workstream, key outcomes will be reached – such as the time taken to allocate a case – without the use of the automated algorithm. This can be compared with the evidence from more recent cases where the algorithm is implemented. The pre- or post- analysis can provide a reasonably robust estimate of the change in outcome attributable to the workstream.

However, there are 2 dimensions to selection issues that might affect the comparability of the pre and post periods. Firstly, the case mix may differ over the 2 periods, for example in the type of cases, claimant characteristics or respondent characteristics. A second dimension is that the period of early conciliation differs for the 2 groups with the control being in early-to-mid-2022 and the treated being at the very start of 2023.

We will use statistical matching to help with the first issue. PSM matches each 'treated' case to one that most closely resembles it from the untreated group using a score (and in the event of 2 identical scores picking a case at random). The similarity between the 2 sets of cases (the score) is measured in terms of characteristics before the treatment, and – in circumstances variously referred to as exogeneity or selection on observables – this allows difference in the outcome variables between the treated and the matched controls to be attributed to the treatment (see Abadie and Imbens, 2006; Rosenbaum and Rubin,1983).

For each case, in Dynamics, there will be fields about the nature of the case, the claimant and the respondent that can be used for this matching exercise. Table 4.2 gives some key variables that will be available. The characterising of each case will not just use Dynamics data, but also use data linking to widen beyond the data available in the Dynamics data, especially in characterising the entity that responds. For example, companies can be linked to the FAME database of company accounts, providing indicators of size, industry and location for limited liability entities. The unlinked organisations – such as the bodies that are not registered with Companies House – will also be characterised identifying public bodies through keywords in organisation name.

Table 4.2: Characterising cases using CMS

Note: T stands for Dynamics table name; V is the variable name within the table.

Table 4.2 provides a first look at the variables that are available. For the claimant, the use of a representative can be derived and the period of employment is in the Dynamics contact information. For the respondent, as well as the use of a representative, the organisation name can be linked to organisational data. There are also case-level variables. Jurisdiction will need some further categorisation perhaps identifying a few of the most common claim types made; there are some fields such as value of claims that are then specific to some jurisdictions. The extent which these can be integrated will be explored and integrated recognising that the field may mean a different attribute dependent on the claim (for example, value could relate to pay, expenses, and so on).

Sample sizes

The sample size calculation requires assumptions that typically cannot be rigorously tested until the data has been collected. However, using assumptions based on past experience is a crucial part of design, establishing the level of confidence that could be associated with empirical work.

The sampling frame for Workstream 3 are the cases related to claimants and representatives who submit a notification form between 19 January and 5 February 2023, and cases in MI data between January and May 2022 which have been statistically matched to these. 

Through Workstream 1, we have seen an average of 1,000 notifications about employment disputes that are appropriate to go to conciliators each week. Each of these 1,000 disputes may relate to multiple individuals. The post-implementation period will run for 3 weeks. Therefore, the estimated sample size for the treated cases is around 3,000 cases, with another 3,000 in the matched pre-implementation group.

Claimant and representative follow-up online survey 

The claimant and representative survey relies on people agreeing to take part in the research. After the six-week period (after submitting the notification form) is reached, we will send an email invite and up to 3 reminders to maximise response to the online survey. This will be followed by a period of telephone calling to further boost response.

The survey will only be administered to claimants and representatives in the post-implementation period. This can be compared with individuals who have completed the survey in the control arm of randomised control trial A (who have been exposed to no or minimal elements of the Smarter Resolution programme). These individuals will be statistically matched across the 2 groups.

Line of questioning:

1. Have you completed the early conciliation process? – yes or no 
2. Before the initial contact with the conciliator, to what extent did you feel worried about the employment dispute? After the initial contact with the conciliator, to what extent did you feel worried about the employment dispute? 
3. To what extent do you agree or disagree with the statement 'the conciliator had the right skills and knowledge to resolve the dispute'?
4. To what extent do you agree or disagree with the statement 'I feel the case was handled efficiently during the early conciliation process'?
5. How satisfied or dissatisfied are you with the outcome from early conciliation? 1 very dissatisfied – 5 very satisfied
6. How satisfied or dissatisfied are you with Acas's early conciliation service? 1 very dissatisfied – 5 very satisfied
7. On a scale from 'very difficult' to 'very easy', how would you describe your experience of using the early conciliation service?
8. To what extent do you agree or disagree:

i. The early conciliation process felt fair
ii. The early conciliation process helped me understand the strengths and weaknesses of the case
iii. The early conciliation process helped me, or me and the claimant, make a decision about whether to take the claim to employment tribunal
iv. Overall, the early conciliation process helped me feel better informed
v. Overall, the process of early conciliation met my expectations
vi. The early conciliation outcome or settlement met my, or me and my claimant's, expectations

We are anticipating an overall response rate of 30% (15% achieved from an initial email invites or reminders, and a further 15% achieved through telephone chasing). Our assumptions are based on a previous evaluation of early conciliation users (33% response rate – see the Acas individual conciliation evaluation report 2019) and current fieldwork on randomised control trial A (30%). The survey response rate assumptions are set out in the table below.

Table 4.3: Survey response rate assumptions

Projections for the randomised control trial A survey estimate a total of 750 cases to compare to for pre-implementation. It is important to note that the timing of randomised control trial A means some 'cultural' changes (for example, better practice relating to manual distribution, and improvements in the system, process, and communication surrounding conciliation availability and work profiles) may have taken place to improve processes related to Workstream 3 for the comparison group in the survey population. This is therefore not as pure as the comparisons that will be made between the pre- and post-implementation period using the MI data. 

If we achieve a sample of 900 for the intervention group, with the comparator randomised control trial A survey having achieved a sample of 750, we will need to see a difference of between 2.9 and 4.7 percentage points between the groups (at the 95% confidence level) for the difference to be statistically significant. 

Qualitative interviews

To capture conciliator, conciliation allocation team member, service user (claimants or claimant representatives) and employer experiences of how the workstream is being delivered, as well as the experiences of these stakeholders and the perceived outcomes, we will also conduct qualitative discussions with post-implementation group conciliators, employees and employers. These will be conducted virtually and are summarised below.

Table 4.4: Participants in qualitative research

All discussions will be conducted using video-conferencing and will be conducted by the IFF Research team. All interviews will be recorded and transcribed securely (with express permission).

Key areas of topic coverage for Acas staff

• how does the early conciliation process compare between the manual approach and new optimised allocation?
• fidelity to intended workstream design (conciliation allocation team and conciliators)
• variations by claimant or track types; views on quality of information and implications on early conciliation (conciliation allocation team)
• views on time burden (conciliation allocation team)
• impact on early conciliation process (conciliators)
• opportunities or challenges (conciliation allocation team and conciliators)
• extent to which the conciliation allocation team are focusing on more complex or exceptional cases (conciliation allocation team)

What are the lessons learned for maximising benefits, for each stage in the early conciliation process (conciliation allocation team and conciliators)?

For service users and employer representatives, we will explore satisfaction with the service, perceived suitability and expertise of the allocated conciliator to case needs, and views on how much time they had for resolution in early conciliation. 

Contact details for relevant staff will be provided by Acas, for recruitment and fieldwork in February and March 2023. 

Service users (claimants or their representatives) will be recruited from the survey, with explicit permission to recontact on this basis collected at the end of the questionnaire. Contact details for relevant employers will be provided by Acas. Recruitment and fieldwork with employees or employers will take place in March and April 2023. For participating, interviewees will be provided with an incentive of £40. 

Our analytical approach to qualitative work is outlined in Appendix B.

Risks 

Workstream 4 logic and intended outcomes

Workstream 4 intends to optimise the service for handling large group claims. 

In October 2021, Acas created a dedicated group claims team (GCT) to deal with any claim based on the same facts with 10 or more claimants against the same respondent. In July 2022, the small group claims team was established to deal with group claims that have 2 to 10 claimants. The 2 teams are collectively known as the Group Claims Cluster. Before the establishment of the Group Claims Cluster, group claims were handled by individual conciliators and then 'blended teams'.

This evaluation work focuses on the work of large group claims (10 or more claimants) handled by the group claims team. The process for dealing with them is complex, time consuming and frustrating for claimants, their representatives, and Acas staff.

Insight work identified some key weaknesses in the current approach to large group claims, including (but not limited to): 

• poor levels of information submitted through group notification forms, resulting in more call time spent understanding the case
• intended outcomes for the case not collected at notification, which, if collected, could increase efficiencies in call time
• the process for identifying large group cases is manual; the team try and spot 'same respondent same facts' across individual and group notifications
• inefficiencies in case flags or categorisation in the system (for example, incident type, jurisdiction) add to administrative time

Workstream 4 intends to address these weaknesses and optimise the service for large group claims through the interventions listed below.

Intervention

The main features of the Workstream 4 intervention that will be evaluated are as follows:

• Auto-allocation to the group claims team (any group cases where there are more than 10 claimants should be automatically assigned to the group claims team and an email is sent to the claimant representative). This went live in May 2022. Previously, all allocation was done manually by conciliation allocation team members. 
• Changes to the group notification form, to enable claimant representatives to add additional information and give permission for conciliators to speak to respondent(s) first. This started in December 2021, with the form going live in July 2022. Previously, conciliators had to seek this permission in first contact with the claimant, reducing the timeframe in which conciliators could engage with respondents (employers or employer representatives). 
• A respondent allocation report which helps the group claims team spot cases with 'same facts same respondent'. This was delivered in August 2022. Previously, the individual conciliators and the blended teams spent time spotting individual and group cases that they should be handling. To spot cases that should be related, conciliation allocation team members were often trying to identify cases that were in the early conciliation support officer (ECSO) queue that have the same respondent and the same facts, but had to manually open each case to look at the jurisdictions.

One of the planned outputs under Workstream 4 was a move to better webinar software for mass signings. After testing new software, the decision was taken not to proceed with this. Instead, improvements were made to the use of the existing software for mass signings. 

Workstream 4 logic model

This model captures our understanding of the workstream. This includes our assumptions of how the programme will operate and how claimants and representatives will experience it. It illustrates the mechanisms for change and how activities are translated into impacts. 

The rationale for Workstream 4 is that if greater automation and quality of data coming through notification is occurring, this will reduce costs of early conciliation and increase time available for resolution.

The assumptions underpinning the Workstream 4 logic model are:

• the new notification form and information coming through it will provide data accurate and complete enough for automatic allocation to the group claims team 
• the new system will sufficiently link individual claims together
• rapport with employer representatives can build when communication from conciliators can happen sooner in the process
• most claims close by 6 weeks after notification is submitted

The key mechanisms developed through the service design work are: 

• a new notification form which allows additional information (open text boxes where claimants can explain the details of their case) and permission to speak to respondent(s) first (the development of this form is taking place in 2 phases, the second version of the new form was not rolled out until January 2023 and hence needs to sit outside of this evaluation)
• new functions in the software for mass signings
• a respondent allocations report to help the group cases team spot 'same facts same respondent' cases
• an effective system for auto-allocating large cases to the group cases team

The above outputs are expected to lead to short-term outcomes:

• claimant representatives have an improved experience of completing a group notification form
• there is a reduced time gap from notification to group cases team contacting employer representatives, resulting in improved relationships between conciliators and this group
• less work done by other conciliators before large group cases are transferred to the group cases team
• greater proportion of large group claims are allocated directly to the group cases team
• a reduced time gap from notification to allocation to the group cases team, and resulting reduced time gap from notification to first contact with a conciliator
• reduced administration time for group cases team during mass signings

Acas hypothesises that if these short-term outcomes are achieved, the service, claimants and representatives will subsequently benefit in other ways. These mid-term outcomes are:

• stronger relationships between conciliators and representatives which will lead to early warning about potential upcoming large group claims – which might in turn lead to earlier warning of potential collective disputes (where a group of employees bring a case against an employer, as opposed to several employees bringing individual linked cases)
• improved user experience for claimants, respondents and representatives

(Collective disputes can include legal action, but also many other workplace disputes. For example, annual pay reviews, other pay issues, contract terms and conditions, changes in working practices, discipline and dismissal, if an employee representative or a group of people are involved, redundancy consultation and redundancy selection, trade union recognition. These are very important due to their potential impact on workplaces and wider society.)

Ultimately, the efficiencies introduced through Workstream 4 should result in more effective conciliation for large cases and more large cases settled in conciliation. Thereby, costs to Acas and the tribunal system should be reduced.

Barriers to a randomised control trial

Many of the challenges in timelines presented for Workstream 3 apply to Workstream 4. Additionally:

• The nature of large group claims means the challenge of comparatively few large cases. This means that a randomised control trial (RCT) could not have achieved the required level of sensitivity without running for a very long time.
• Some of the efficiencies or improvements in customer experience for large group cases rely on improvements that would be achieved for repeat claims. (This is because the specialist team would have particular knowledge or relationships about these organisations or representatives, that they could leverage in subsequent contacts.) This would be diluted if claims were allocated at random.

Evaluation design

The impact of Workstream 4 will be assessed by looking at the resource and process changes due to the better identification and handling of large cases. The evaluation will involve the comparison of pre- and post-intervention case data to assess improvements in outcomes.

The small number of large group cases means that the options for evaluation are more limited than for the other workstreams. There are around 40 to 50 large group cases per month. This means that we are less likely to be able to detect changes through comparison of pre- and post-intervention case data (although we will still explore this). Also, we would not be able to generate a useful survey sample without running the evaluation for a longer period than is available for the evaluation. There is also quite a lot of variation in:

• the cases handled by the large group teams, in terms of the number of individual claims that are subsumed within them
• the time over which cases are appended to the claim which adds further complexity 

Large group cases will be identified in Dynamics 365 (Acas's case management system). An analytical dataset will then be created which shows the progress of these cases (communication at each stage, at what point additional cases were added, and so on). 

The evaluation will be carried out in stages. Analysis will look at the pre-implementation period to explore changes in comparison with the post-implementation period, with these periods defined below. This will be complemented by qualitative fieldwork with:

• the group claims team managers and conciliator
• employee representatives who have had their claim handled by the large group claim team

Evaluation period

The pre- and post-implementation periods (timings and length) for Workstream 4 are likely to be:

• Pre-period: January to April 2022 prior to the interventions being in place. 
• Post-period: October to December 2022 after implementation of the initial new group notification form, respondent allocations report and auto-allocation of group cases. This period will also be prior to the implementation of the revised group notification form which will need to sit outside of the scope of the evaluation. 

Participants

The participants in this evaluation are:

• group claim team conciliators and their supervisors – conciliators will be a mix of those who understood and experienced the previous processes for identifying and handling large group claims (in other words, exposed to pre-implementation), and those who are new to the processes. Managers should have exposure to pre- and post-implementation (qualitative interviews)
• employee representatives (qualitative discussions) responding to large group claims, who have pre- and post-implementation exposure to Acas large group claim processes

Outcomes

These outcomes and impacts for Workstream 4, and the associated indicators for measuring them, are summarised in the evaluation framework below.

We will be able to compare pre- and post-implementation cases in the CMS, supported by insight from qualitative interviews.

Table 6.1 Indicators attributed to each outcome

Qualitative interviews

We suggest qualitative fieldwork involves:

• 2 in-depth interviews with the group claims team manager and deputy
• a focus group with group claims team conciliators 
• 20 in-depth interviews with claimant representatives who have had claims handled by the group claims team

The types of research topics we could explore in these interviews would be as shown in the following table.

All discussions will be conducted using video-conferencing and will be conducted by the IFF Research team. All interviews will be recorded and transcribed securely (with express permission).

Contact details for relevant Acas staff and employee representatives will be provided by Acas. Recruitment and fieldwork will take place in January to February 2023. 

For participating, employee representatives will be provided with an incentive of £40. 

Our analytical approach to qualitative work is outlined in Appendix B.

Bibliography 

Abadie, A., and Imbens, G. (2006) Large sample properties of matching estimators foraverage treatment. Econometrica, vol. 74, January, pp. 235–267. 

Rosenbaum, P., and Rubin, D. (1983) The central role of the propensity score in observational studies for causal effects. Biometrika, vol. 70, pp. 41–55.

As a check to make sure that it will be possible to extract the data on outcomes detailed in the evaluation frameworks, we have spoken to data leads, reviewed data dictionaries and carried out exploratory analysis of test data. This section details the analysis of test data that we have conducted. 

Acas manages its cases through a Microsoft Dynamics 365 relationship management software. This tracks cases from their submission to Acas over time to various outcomes letting Acas administrators, conciliators and other staff record progress, events and communications of each case. In terms of what data is recorded, the pathway of a case involves:

• completing various forms
• some formal outputs of the conciliation process
• many notes as the individuals involved in a case interact

Acas provided the research team with an extract of a dozen cases. This has been used to code a series of procedures to translate the administrative records into variables useable in the evaluation. This section describes the structure of the data and the analytical data that has been compiled from it. 

Dynamics and case management data

An extract of the data with about 12 cases was provided to the research team in summer 2022. The dataset contains 14 separate tables, with each of the cases being recorded in some or all of these. The tables reflect:

• various case specific information (such as the notification form)
• information about transactors (such as contact details of claimants, respondents, and so on)
• information about transactions (such as events that occur as a case progresses)
• characteristics of the case (such as its jurisdictions both during conciliation and the as heard in the employment tribunal)

There are several identifiers in the database. The key one is the case identifier, which is included in almost all tables and present in any data that is about a case. There are also a few tables that are look-ups (such as for the jurisdiction coding) which are not case specific and so do not have a case ID. Across the tables, the key ones are the table relating to the notification and the associated early conciliation support officer (ECSO) form. Then, there are tables listing the steps in allocating the case (called allocationhistory) and the steps thereafter (called Incident). 

Processing the Dynamics data

Stata code has been written that first extracts the data from each of the sheets in the workbook. Four of the sheets are complex, structured to have multiple lines that can only be linked to a record by its position and a first procedure cleans this up.

The bulk of the code links the data together by case ID and then collates events. For an event, as well as the date, textual or categorical data is included. So, an event such as the notification will be associated with the type of case (for example, standard versus fast). The main code then constructs a dataset with a row for each case and a series of variables drawing from the event data. Typical events include the submission date, the allocation of the case, through to the outcome.

Conciliation outcomes

The compiled data tracks cases from their submission to Acas using the Dynamics management information systems. This section describes the outcome variables that can be derived as the cases are tracked in the management data.

Estimating outcomes and the underlying Dynamics data

Table A.1 maps out the outcomes and the case level Dynamics indicators that would be used to measure the outcome.

The outcome of the conciliation process is described in a single indicator variable, which has categories of "Informal resolution", "COT3", "Respondent declines early conciliation", "Cannot contact respondent", and so on.

(Note that the outcome of the case is not the same as an evaluation outcome. The latter is the range of outcomes expected through Smarter Resolution investments.)

In Table A.1, this is the table (indicated with a T) "Incident" and the variable (V) in that table is acasoutcome. 

There are dates associated with this: the drawing up of the certificate (creation variable) and then the date of the outcome (usually on the day of the certificate creation, but not always). These variables are available for all cases that have completed the early conciliation. It could be used to define which cases can and cannot be analysed in the evaluation when a data extract is drawn.

Note: T stands for Dynamics table name; V is the variable name within the table.

The table highlights one of the evaluation outcomes that can be derived from these variables, "percentage of cases settled at early conciliation stage". The evaluation outcomes are the variables that are to be compared for the cases that benefit from Smarter Evaluation workstreams against the control to understand additional effects. Settling more cases at early conciliation stage would be a benefit, reducing the costs of going to tribunal.

A second outcome that can be measured using the Dynamics data is the time between case start and the outcome. This is a measure of efficiency in conciliation.

The outcomes after the early conciliation process relate to the case progressing to tribunal. Dates are recorded in Dynamics for the submission and receipt of the claimant's ET1, the respondent's ET3, and the start date of the tribunal. These indicators can be used to identify the claims that go to tribunal.

Many of the Smarter Resolutions interventions assist in the correct allocation of a case, based on:

• an improved understanding in claimants (Workstream 1)
• clearer claims being made initially (Workstream 2)
• automated allocation (Workstream 3) 

A Dynamics table called AllocationHistory records the timings of the start of allocation and then each subsequent allocation to a conciliator. This sometimes takes only one step but, where a case is initially allocated to a conciliator that cannot progress the case, the number of steps to a final allocation can be both counted and timed. 

Further quality measures of the case allocation and conciliation processes

Table A.2 considers how the management information can be analysed to look at the quality of the processes. 

The first 2 outcomes will use the Dynamics tables about the email and phone interactions with case participants. The Email and Phone tables list each incident with the case ID, but also:

• when the event occurred
• who was involved in the call or email (for example, the respondent or claimant)
• qualitative descriptions of the incidents

The approach taken here will be to estimate the frequency of events between the key dates of a case, such as the number calls or emails between notification and allocation. This can be calculated using the dates in Table 1 and then the 2 Dynamics tables, called Phone and Email, which lists the events. The counts of events will approximate how much further clarification was needed to achieve a stage in the process. The outcome desired is that, because Smarter Conciliation results in more informed participants, they will not need to interact with Acas as frequently and these counts will drop for the treated.

Different cuts could be analysed also, such as the emails and calls between Acas and the claimant side or respondent side. A further analysis is the level of interaction:

• between notification and allocation (clarifying the allocation of the claim)
• between notification and the outcome (the early conciliation process)

Note: T stands for Dynamics table name; V is the variable name within the table.

The acasnotification table also contains the assessments made about the validity of a case at notification. This can be used to look at the level of completeness with which a claimant is able to fill out the notification form and whether the claimants have been able to identify for themselves whether a claim is valid. For the former, the table includes a flag about the sufficiency of information. For the latter, 2 flags indicate whether the claim is eligible for early conciliation or employment tribunal. 

Our analytical approach for the qualitative research will be both deductive and inductive. 

Deductive because we will be testing the theory of change and logic model using the qualitative evidence (as well as other evaluation evidence). Inductive because we will also build upwards from the views of participants – incorporating elements of 'grounded theory' analysis. For example, the thematic review and continual analysis of hypothesis from participants dialogue and researchers' impressions of the discussion (such as pauses or tone). The latter ensures we do not miss new or different explanations or experiences that we had not anticipated at the start of the evaluation. 

We will also take an iterative and summative approach to analysis. Analysis will begin informally during fieldwork itself as our research team:

• work closely together throughout the fieldwork period
• feedback headline findings to each other as discussions are conducted
• continually update our approach and thinking as we amass data 

All interviews will be written up in detail, including verbatim quotes, in an analytical framework in Excel. The framework will be structured around the logic model and research questions, and include key sample data, to allow for comparison of findings by different characteristics. 

The data will then be analysed to search for themes and trends, both present and absent.

Once qualitative analysis is complete, we will then compare those findings with other evidence to challenge and address gaps. Director-led analysis sessions will bring this thinking together, encourage challenge of assumptions and identify areas for further, targeted analysis.