Evaluating the Heterogeneous Impacts of the Improving Access to Psychological Therapies (IAPT) Programme

Mental health conditions account for 30% of non-fatal diseases and 10% of the overall disease burden, including death and disability worldwide (World Bank & World Health Organization, 2016). In the UK, mental health problems are the largest single cause of disability, with one in four adults experiencing at least one diagnosable mental health problem in any given year. Poor mental health has significant adverse effects on individuals and their immediate community (physical health, longevity, employment, social relations, and overall life satisfaction) and on the economy as a whole. The Organisation for Economic Co-operation and Development (OECD) estimates the total costs of mental ill health in the UK at approximately 3.5% to 4% of gross domestic product (GDP) (OECD, 2017). Apart from putting additional pressure on the healthcare system, mental ill health reduces employment, productivity, and accounts for almost half of disability benefits. The COVID-19 pandemic and associated lockdowns have worsened the mental health of countless individuals in the UK. It is expected that mental health issues to be high on the policy agenda in the recovery from COVID-19 and in a post-COVID-19 world.

In 2008, the National Health Service (NHS) began with the nationwide implementation of the Improving Access to Psychological Therapies (IAPT). One of the ambitions of IAPT is to achieve at least a 50% recovery rate. While this goal was achieved in 2017, treatment outcomes differ substantially between patients of different socio-economic and ethnic groups as well as different geographical locations. For instance, in 2017, there was a 17% difference in the recovery of patients between the most and the least deprived areas (58.1% for the least and 41% for the most deprived area, see Moller et al. (2019)). Recovery rates were also found to differ significantly across patients’ religion, disability, ethnicity, and sexual orientation.

The goal of this project is to help policy-makers understand the sources of heterogeneity (dissimilarities and diversity) in the effectiveness of IAPT treatments for different patients and in the performance of different service providers. The London School of Economics and Political Science (LSE) seek to answer the following research questions:

1. What is the treatment effect of the IAPT programme on patient outcomes?

2. Which types of patients benefit most and least from which type of treatment, and why? What are the characteristics of the service provider (e.g. waiting times, average number of sessions, share of referrals who completed the treatment) and local area (e.g. urban or rural location, local infrastructure, or local deprivation and unemployment) that are associated with a higher or lower probability of individual recovery?

3. Building on identified sources of heterogeneity on patient level, what makes a successful IAPT service provider? How important are the characteristics that service provides can control (e.g. waiting times) and cannot control (e.g. local area characteristics). Should the expected average recovery rates of services vary in line with the socio-economic characteristics of an area?

In order to address these research questions, LSE require access to the IAPT dataset from NHS England. To understand the sources of variation in the recovery rates of IAPT patients, LSE must first distinguish the causal effect of a treatment programme from confounding factors such as natural recovery (or deterioration). The plan is to begin with the estimation of individual treatment effects using pseudonymised patient-level data. The treatment effect is a causal quantity that measures the difference between the patient outcome after treatment and the hypothetical outcome if the same patient had not obtained treatment. This is a challenging task that has not been done previously for IAPT treatments. Prior work provides only descriptive evidence that is useful for analysing treatment outcomes, i.e. the change in the intensity of symptoms from before and after treatment. The challenge in building a causal treatment model to control for confounders with the IAPT data is that the IAPT programme is not a randomised controlled experiment. LSE propose to construct a quasi-control group using variation in waitlists, i.e. waiting times between the initial assessment and the treatment sessions.

Another intention is to combine different data sources and use them as efficiently as possible. In particular, LSE will combine the IAPT data with publicly available geographical information that will enable LSE to construct measures of wealth, education, ethnicity, and other characteristics of the area, where each service is located. Statistically, such data will help control for otherwise unobserved socio-economic effects. LSE's combined dataset will include patient characteristics (i.e. demographic and socio-economic characteristics, diagnoses, treatments, treatment outcomes, service characteristics (i.e. number of new referrals, the number of patients who started/finished treatment, waiting times, and other indicators), and session details during the course of treatment), and local area characteristics (i.e. incomes, employment, education, crime, quality of housing and living environment) of the geographical areas where services are located (which LSE will collect separately from publicly available data sources). Importantly, LSE plan to use intermediate patient-level data, i.e. data on the sessions attended during the course of the treatment, not only because LSE believe the dynamics of the treatment process are informative for explaining treatment effects, but also to enable LSE's model to inform proposals for future treatments that would be conditional on treatment paths.

Record level data is crucial for the purpose of the analysis. The chosen timeframe, covering IAPT data between 2012 and 2020, would allow enough observations to perform the heterogeneity analysis, i.e. to have a sufficient amount of patients with different combinations of characteristics. It would also allow LSE to analyse the evolution in the performance of the programme over time. The project requires use of all IAPT packages, including Core package, Waiting Time package, Disability and Patient Experience packages. Since LSE intend to identify the inequality in the effects of the programmes for the patients with different characteristics, LSE need to be able to observe all the possible characteristics that are available in the dataset. Also, LSE's identification strategy for treatment effect estimation relies on the ability to control for potential confounders. Omitting some patients’ characteristics might result in constructing a less reliable quasi-control group.

The NHS’s Mental Health Implementation Plan 2019/20 – 2023/24 states that one of IAPT’s current priorities is “[…] reducing geographic variation between services and reducing inequalities in […] outcomes for particular population groups”. The need to understand the source of these variations will be more pressing than ever with the ongoing COVID-19 pandemic. Indeed, the pandemic has been having a direct effect on increasing the number of patients in the short to mid-term and indirect effects as the NHS, strained by COVID-19, must allocate resources to identify and treat patients as efficiently as possible.

LSE's findings about the treatment effect of the programme and its heterogeneities will allow LSE to better understand the sources of inequality in treatment outcomes of the IAPT patients. They will reveal which patient groups are more responsive to particular treatments in particular services, allowing LSE to identify the best diagnosis-treatment match. LSE will also learn if different patient groups are systematically influenced by different service and local area characteristics and in which ways. LSE will use the sources of heterogeneity identified in this analysis and their relevant importance to understand what drives heterogeneity in the share of recovered patients among different service providers. The sources of the variation that can be controlled by the service, e.g. waiting times, will form the basis for relevant policy recommendations. The sources of variation that cannot be controlled by the service, e.g. local area characteristics in terms of health or deprivation levels, if found important for the recovery rates, can be used to construct different expected outcomes for services located in different areas.

LSE's team will be hosted by the Community Wellbeing Programme in the Centre for Health Economics (CEP) at the LSE. The CEP is one of the leading interdisciplinary economics and policy think tanks worldwide and has been awarded Economic and Social Research Council Research Institute status (being one of only two institutes that have received this recognition so far). The LSE is the sole data controller who also processes the data for this study. The LSE also provides funding for the project. The study team are all substantive employees of the LSE, and they will receive advice from experienced academic mentors and scientific advisors in the field based at the University of Surrey and the University of Oxford. These mentors and advisors have significant experience in the fields of mental health, the economics of wellbeing, cost-effectiveness as well as cognitive behavioural therapy. These organisations do not determine the purpose or the means of the data processing, and will not be accessing NHS England data under this agreement. The study team is also advised from a general policy perspective by the former Cabinet Secretary at the time the scheme was rolled out.

The LSE is a ‘public authority’, as defined in the Data Protection Act 2018, with a principal object of the organisation being research and its dissemination. The processing of pseudonymised personal data, including special category data, is necessary to carry out medical research that serves the public interest. The legal basis for processing personal data is: Article 6(1)e of the GDPR, ‘processing is necessary for the performance of a task carried out in the public interest’; and Article 9(2) j of the GDPR ‘processing is necessary for archiving purposes in the public interest, scientific or historical research purposes’. The processing of sensitive personal data is in the public interest as the results of this work will help to identify the ways in which services can be improved and patients and treatments better matched, informing evidence-based health policy on how to improve mental ill health in the UK and beyond.

Reference List

Moller, N. P., Ryan, G., Rollings, J., & Barkham, M. (2019). The 2018 UK NHS England annual report on the Improving Access to Psychological Therapies programme: A brief commentary. BMC Psychiatry, 19(1), 252. https://doi.org/10.1186/s12888-019-2235-z

OECD. (2017). OECD Mental Health Performance Framework.

World Bank Group, & World Health Organization. (2016). Out of the Shadows: Making Mental Health a Global Development Priority. https://www.who.int/mental_health/advocacy/wb_background_paper.pdf?ua=1

The London School of Economics and Political Science (LSE) require access to pseudonymised patient-level information from the Improving Access to Psychological Therapies (IAPT) Data Set. The LSE requires a one-off transfer of the patient-level IAPT data covering the period between 2012 and 2020.

Data will be processed by the LSE, which is the sole data controller and processor. Data processing will only be carried out by substantive employees of the LSE. LSE's Centre for Economic Performance (CEP) has rich experience in using sensitive data, including NHS data, for economic research and it provides all the necessary infrastructure to guarantee that all the security requirements of storing data are met. The data set will be uploaded in an extra secured area of LSE’s network where the storage architecture is compliant with NHS England’s Data Security and Protection Toolkit. There will be no subsequent flows of IAPT data from the LSE.

Data will only be stored offsite at Amazon Web Services London data centre. Amazon Web Services supply Cloud Services for the London School of Economics and are therefore listed as a data processor. They supply support to the system, but do not access data. Therefore, any access to the data held under this agreement would be considered a breach of the agreement. This includes granting of access to the database[s] containing the data.

Researchers who will work with NHS England data need to request access to the environment where it is stored. When the team that manages the environment have approved the request, the researcher can connect to the environment through a Virtual Private Network (VPN) via Multi-Factor Authentication into a remote desktop environment. No data can be downloaded onto individual devices. All data processing and analysis occurs using networked LSE devices on campus, or remotely via the VPN.

High standards of data handling are further guaranteed by the fact that all researchers who access data have the relevant skills in working with sensitive data. All members of the team have been appropriately trained in data protection and confidentiality, they have also passed the Office for National Statistics (ONS) course on safe data handling and received Approved Researcher status from the ONS.

In order to protect patient confidentiality, when presenting results calculated from IAPT record level data, the following suppression rules are to be applied:

· any figures based on a count of between 0 and 4 referrals are to be suppressed by replacing the number with an asterisk (*);

· all sub-national counts are to be rounded to the nearest 5;

· sub-national rates (which are presented as percentages and are based on unrounded numbers) are to be rounded to the nearest whole percent

· national rates are to be rounded to one decimal place.

IAPT data will be grouped according to region. Groups of individuals within a region can then be linked to publicly available data containing local area characteristics in which patients and services are located, such as those found in the Office for National Statistics (ONS) online database. Since data will not be linked to other individual-level data, only data at the level of the Lower Layer Super Output Area (LSOA), LSE estimate the risk of reidentification to be negligible. There will be no requirement/attempt to re-identify individuals.

From the IAPT data, the LSE will match patients receiving treatment for mental health problems with patients diagnosed with the same problems who should have been treated but were not, yet the progress of their symptoms was recorded over time as though they were receiving treatment. These patients will be referred to as the ‘waitlist control group’.

The LSE will ensure a match between treated and waitlist control patients by controlling for observable regional and individual characteristics as well as centre fixed effects. Importantly, patients in treatment and waitlist control groups are matched based on type of initial diagnosis and self-reported duration of symptoms when they obtain their initial diagnosis and have their baseline measures taken. That is, only patients within the same centre who had the same initial diagnosis and the same self-reported duration of symptoms are compared; the only difference between them being that the centre can only treat a limited number of patients at the same time. Sample selection bias can be minimised by controlling for service types (i.e. look at within-service variation) and initial health status (by matching severity of symptoms and other observed characteristics).

LSE anticipate that the state of the patients in the waitlist group might change due to natural recovery and deterioration. LSE will ensure that these natural changes are similar between the treatment and control groups by controlling for the symptom’s duration of the patients in LSE's model. This will guarantee that LSE compare the patients at a similar stage of the illness.

LSE will focus on average individual treatment effects and inequalities therein. For each patient, LSE observe individual characteristics such as age, gender, ethnicity, religion, sexual orientation, long-standing health conditions, employment status, and reported social scale scores, as well as the characteristics of treatment such as referral, diagnosis, treatment mode, and waiting times. Some of those are collected during each of the appointments (on average, patients have seven treatment sessions).

LSE will begin by estimating the treatment effect on the treated of the IAPT programme. LSE use three outcomes for this model: depression scale, anxiety scale, and an indicator for recovery, i.e. the scores on both scales are below the clinical cut-off at the end of treatment.

LSE then will explore heterogeneity of treatment effects. LSE will estimate five models, sequentially adding different sets of characteristics to each subsequent model (Diagnosis and prescribed treatment characteristics - Model 1, adding patient characteristics in Model 2, service characteristics in Model 3, local area characteristics in Model 4, treatment dynamics (data on the attended sessions) in Model 5). Sequentially adding potential sources of heterogeneity will allow us to better understand the characteristics which are associated with the inequality in treatment effects. Models 1 to 4 will use the data at the start of the treatment and will allow us to draw conclusions about expected treatment effects for different patients when they join the programme. Model 5 will also incorporate intermediate data, i.e. data on the number of attended sessions and their characteristics. This will be informative about how final treatment effects can be improved during treatment (e.g. whether some characteristics of early sessions are associated with different treatment effects), which can be of interest for clinicians and therapists.

LSE intend to release a LSE-CEP Discussion Paper: “Using Machine Learning to Evaluate Average and Heterogeneous Impacts of a Nationwide Public Health Service: The Improving Access to Psychological Therapies (IAPT) Programme”. The LSE-CEP Discussion Paper series is amongst the most widely quoted and read discussion and working paper series in the world. LSE anticipate submitting the discussion paper to a leading economics journal (Review of Economics and Statistics, Journal of Public Economics, or Journal of Health Economics). Of particular academic interest will be LSE's ability to estimate the causal effect of a nationwide public mental health service on patient outcomes, including heterogeneous treatment effects, and LSE's insights on using waiting times to construct a control group.

These academic deliverables are expected to be accompanied by non-technical, general-audience summaries in form of LSE-CEP CentrePiece articles (the quarterly magazine of the CEP, with 500k downloads a year), policy briefs targeted at public and health policy-makers as well as mental health practitioners, clinicians, therapists; and blog entries on the web sites of organisations LSE have published with over the years (What Works Centre for Wellbeing and VoxEU). These activities are flagged by press releases, to be led by the media and communications team at LSE-CEP and other departments (LSE Department of Psychological and Behavioural Science, LSE Department of Health Policy, University of Oxford Department of Experimental Psychology), all of which have slightly different target groups.

Finally, LSE intend to present LSE's findings at general economics (American Economic Association Annual Meeting, Royal Economic Society Conference, and European Economic Association Congress), health economics (International Health Economics Association Congress and European Health Economics Association Conference), and econometrics conferences (Econometric Society Meetings and International Association of Applied Econometrics Conference) as well as interdisciplinary conferences in public and health policy, including related seminars (LSE-CEP Wellbeing Seminar and LSE-PBS Research Seminar) and workshops. Through LSE's scientific advisors, LSE are in the unique position to disseminate LSE's research and findings to the highest policy level (e.g. by being able to access the All Party Parliamentary Group on Wellbeing Economics, an officially recognised cross-party group of MPs and Lords in the UK Parliament).

In terms of key stakeholders, LSE's research and findings are expected to be of interest to public and health policy-makers; mental health practitioners, clinicians, therapists; and researchers in the field of health economics and microeconomics broadly. LSE recognise the importance of involving these stakeholders from early on to inform specific aspects of LSE's research. To this end, LSE are looking to organise a kick-off workshop at the beginning of LSE's project (month 1), leveraging LSE's scientific advisors’ and mentor’s networks to invite selected representatives from each stakeholder group. Midway through the project, LSE aim to share preliminary results in the form of draft discussion papers to obtain further comments and suggestions. A final conference is being planned for the end of the project (month 24), involving a larger group of representatives from each stakeholder group to present LSE's findings formally as well as media and press effectively. Speakers will include LSE'S scientific advisors, mentor, and other leading specialists in the field.

The LSE-CEP Community Wellbeing Programme has strong networks across policy and practice, which have led to substantial prior policy impact. Importantly, the IAPT programme itself was conceptualised, designed, and preliminarily evaluated by LSE's scientific advisors. The CEP work was also pivotal for the inclusion of wellbeing cost-effectiveness in the updated HM Treasury Green Book and the establishment of mental health support teams in schools.

This project caters to several groups of beneficiaries, including researchers in public health, mental health, and wellbeing; health economics, economics, and econometrics; public administration; and applied researchers in public and health policy; clinical research and therapy; and in the third sector such as mental health charities, both in the UK and worldwide. LSE expect to provide the benefits after 24 months.

LSE hopes to investigate the sources of inequalities in the effectiveness of the IAPT programme with the goal to inform the policies on improving services and delivering the most relevant treatment to each patient. Of particular interest is expected to be the estimation of the treatment effects of the IAPT programme, a nationwide public mental health programme which is widely recognised as the largest and the most ambitious programme of applied cognitive behavioural therapy in the world. Apart from estimating the average treatment effect, LSE intend to estimate heterogeneous treatment effects by service and patient characteristics as well as characteristics of the area in which services and patients are located. This is anticipated to be a significant value added to the existing literature which has important implications not only to LSE's beneficiaries involved in the further development of IAPT and its current priority of “[…] reducing inequalities in […] outcomes for particular population groups”; but also to LSE's beneficiaries involved in the design, testing, and implementation of similar emerging mental health programmes around the world.

LSE's project caters to several groups of beneficiaries, including researchers in public health, mental health, and wellbeing; health economics, economics, and econometrics; public administration; and applied researchers in public and health policy; clinical research and therapy; and in the third sector such as mental health charities.

At a more practical level, LSE's findings are likely to have important implications for individuals in applied research positions in public and health policy, clinical research and therapy, and mental health practitioners. Understanding the sources of the variation in the effectiveness of treatment that can be controlled by a single therapist, a service provider, or the overall programme are hoped to inform evidence-based policy recommendations on how to improve treatment practices, service operations, or overall programme guidelines. LSE's analysis of the dynamics of the treatment process (i.e. the number and the characteristics of attended sessions) is expected to shed light on whether and how treatment effects can be improved during the course of the treatment. LSE expects to identify the kind of patients who respond better to treatment and to better understand current inequality in the treatment outcomes of the programme. The results of this work are expected to help to identify the ways in which services can be improved and patients and treatments better matched.

Finally, LSE's analysis is anticipated to be of interest to administrators of the IAPT programme and similar initiatives due to LSE's ability to identify the sources of the variation in the effectiveness of treatment that cannot be controlled by a single therapist, the service provider, or the overall programme, for example, local area characteristics such as local deprivation or infrastructure. This is likely to have important implications for local and central Government beyond public health, as they can inform the decisions on how to improve public mental health services and can guide more efficient resource allocation, which is particularly important given scarce resources, especially in the recovery from COVID-19.

Although it is difficult to conclusively say how many patients in England are likely to benefit from LSE's work, LSE can make an estimate. England has a population of 56 million. At any point in time, approximately 5 percent of the population is living with depression or anxiety, yielding about 2.8 million individuals. IAPT currently treats about 15 percent of these and aims at raising this share to about 25 percent (Five Year Mental Health Forward Plan). LSE take the latter target figure. Hence, LSE expect that about 700,000 patients are likely to directly benefit from LSE's work in the future, and in particular, the most vulnerable amongst these who are currently lagging behind the average in terms of treatment effectiveness.

The CEP are working closely with the ‘What Works Centre for Wellbeing’, and have a direct connection to a member of the House of Lords. Mentors to the project include the initiators and designers of the IAPT programme, and leading researchers on mental health and social care in the UK. These organisations and individuals are well placed to facilitate the CEP in translating the results of the analysis into policy.

LSE measure the policy impact of LSE's project by citations in policy publications (for example, by the NHS itself or related UK Government Departments or Agencies, such as the Department of Health and Social Care, UK Health Security Agency or Office for Health Improvement and Disparities; or by charities working in the sector, like the Mental Health Foundation). Its academic impact will be measured by citations in academic publications (that is, how often LSE's published paper is being cited in other papers).

LSE used variation in waiting time to estimate IAPT programme’s causal effect that is not contaminated by natural recovery of natural deterioration of mental health. LSE demonstrated that treated patients are more likely to reliably improve and reliable recover and less likely to reliably deteriorate. LSE find significant heterogeneities in how patients with different characteristics respond to the treatment. Even those who benefit the least are more likely to reliably improve and reliable recover as the result of the treatment. This does not hold for reliable deterioration. More details can be found in the CEP discussion paper CEPDP1982.