Mixed methods evaluation of the Getting it Right First Time programme - improvements to NHS orthopaedic care in England

Background:

Researchers from University College London (UCL) are conducting an evaluation of the high profile ‘Getting it Right First Time’ (GIRFT) improvement programme currently being rolled out nationally in orthopaedic surgery. The first GIRFT report was published in 2012, by the then president of the British Orthopaedic Association (BOA). It recommended changes to NHS orthopaedic practice to improve patient outcomes, and achieve cost savings. NHS England subsequently funded a national professional pilot which involved senior clinicians visiting providers across the country to offer bespoke recommendations about improving care. By July 2015, the GIRFT project team, which is hosted on behalf of the BOA, at the Royal National Orthopaedic Hospital in Stanmore, had carried out reviews with 120 NHS Trusts in England. The project highlighted significant variations in practice and outcomes; device and procedure selection; costs; and infection rates. Consequently, the Department of Health and Social Care commissioned a three-year programme to address these challenges initially in orthopaedics, and then across ten other clinical specialities.

The wider GIRFT programme has now been expanded to over 37 specialties. Work is underway to enhance patient safety by addressing the complex issue of unwarranted variation in elective orthopaedic surgery. The programme aims to address this, by prioritising the following areas: increasing minimum volumes of complex procedures undertaken by individual surgeons, enforcing ring-fenced beds to reduce infection rates, reducing length of stay and waiting times to improve efficiency for patients, improving theatre efficiency, aiming for more competitive procurement processes, reducing unnecessary procedures (i.e. therapeutic arthroscopies), and gaining a better understanding of the causes of litigation. The programme also has strong links with Lord Carter’s Procurement and Efficiency programme.

Evaluation:

The GIRFT programme seeks to change practice, (e.g. prosthesis choice) in order to improve patient outcomes (e.g. infection rates). It can be defined as a complex intervention, containing several interacting components, targeting different stakeholders and addressing a number of different behaviours. There is, however, limited evidence to support the approach being used. Whilst there have been reorganisations of a number of acute NHS services in recent years, there have been few examples on this scale. The UCL evaluation aims to contribute to the development of that evidence base by studying in depth the implementation of changes to orthopaedic services.

UCL are evaluating the intervention as a whole, including the original pilot and the work now being carried out. The researchers are focusing on outcomes directly relating to patient care, for the commonest elective orthopaedic procedures: total hip replacement (THR) and total knee replacement (TKR). Through this, they will identify lessons to inform future service improvement efforts. A cohort of patients for linkage will be provided from the National Joint Registry (NJR), and returned to UCL.

Relationship between UCL and GIRFT:

Although some members of the GIRFT team are listed as co-authors on the published study protocol, UCL are undertaking an independent evaluation of the programme. Indeed, the GIRFT Team are keen to ensure that the UCL evaluation provides an independent analysis of their work. The GIRFT team are neither funders nor co-applicants on the revised NJR application (or any other data requests they are making as part of the evaluation). To facilitate understanding of the programme, the GIRFT team provide UCL with key documents, dates and details of their visits and re-visits, as well as providing UCL with access to key events undertake non-participant observations. UCL maintain contact with the GIRFT team to ensure both remain up-to-date with any developments of the national programme, which may influence UCL's ability to collect data, as well as influence the development of the findings. However, they have no role in the conduct of the evaluation.

The researchers’ overall aim is to evaluate the implementation of a complex intervention, the GIRFT programme which seeks to improve the quality and cost-effectiveness of NHS orthopaedic care in England. In doing so, they will identify lessons to guide future improvement work in other services. Their objectives are to use formative and summative evaluation methods to:

1. Examine the key processes of – and factors influencing - the ongoing development and implementation of planned improvements to orthopaedic services, nationally and locally, including any unintended consequences;

2. Assess whether the GIRFT programme has reduced variations in

orthopaedic practice and expenditure, and improved patient outcome measures;

3. Estimate the economic impacts (e.g. cost savings) of these changes; and

4. Explore patient and public perceptions of the planned improvements to care.

Reflecting these objectives, the evaluation comprises four work streams: (WS1) an organisational study; (WS2) a quantitative study; (WS3) an economic study; and (WS4) patient and public focus groups.

Data request:

This agreement refers to the quantitative study (WS2) and the economic study (WS3). In order to assess whether the GIRFT programme has reduced variations in orthopaedic practice, and improved patient outcomes, longitudinal, provider-level data are required. With details of all admissions to NHS hospitals in England, the Hospital Episode Statistics database represents the only means by which the UCL researchers can evaluate the impact of the intervention nationally. Similarly, there are no alternative sources to the Patient Reported Outcome Measures database, which would provide data from all providers.

The researchers plan to link the data being requested from NHS Digital (i.e. Hospital Episode Statistics and Patient Reported Outcome Measures) to data from the National Joint Registry. The dataset will then be used to examine: the proportion of procedures by low volume surgeons and in low volume hospitals; the use of prostheses not recommended by GIRFT; rates of readmission and revisions; outcomes reported by patients in their post-operative PROMs questionnaires; and resource use such as prosthesis type and length of hospital stay). Researchers will also inform an estimate of the economic impacts (e.g. cost savings) of the changes.

The researchers would like to request the following for the amendment and renewal in March 2019:

1. Re-supply of HES and PROMs extracts in replacement of original HES and PROMs extracts

2. Linkage of HES and PROMs (hip and knee procedures only) to an extended cohort (to 31/3/18) from the NJR

3. Historic HES episodes for 2008/09 for cohort patients from 2009/10 (this will be an addition as it was not requested in the previous version of this agreement). This is to gain a fuller understanding of the co-morbidity data of those in the cohort.

4. Historic and subsequent HES episodes for all cohort patients

5. Up to date PROMs data for the additional patients in the cohort

Clarification of Roles:

UCL have provided the following information to clarify the roles of parties mentioned in their fair processing notice and protocol documents.

1. NIHR CLARHC (Collaboration for Leadership in Applied Health Research and Care) North Thames

National Institute of Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) conduct world class applied health research which will have a direct impact on the health of patients with long term conditions and on the health of the public. CLAHRCs are collaborations between universities, the NHS (local service providers and commissioners), local authorities, industry, patients and the public, the local Academic Health Science Network (AHSN) and other relevant organisations in the region. NIHR CLAHRC North Thames is one of 13 CLAHRCs across England. It covers the geographical area of north central and north east London, south and west Hertfordshire, south Bedfordshire and south west and mid-Essex. The research team at UCL work within the NIHR CLAHRC North Thames structure.

2. The GIRFT programme team

This is an independent evaluation. The GIRFT programme team have no role in the conduct of the research, nor are they data controllers. They do not determine the purposes for which and the manner in which any personal data are to be processed by the data processors.

3. Royal National Orthopaedic Hospital (RNOH)

The GIRFT programme began from the RNOH, as the majority of the original GIRFT programme team were employed by this organisation. It is now co-delivered by the RNOH and NHS Improvement. As this is an independent evaluation, the RNOH have no role in the conduct of the research, nor are they data controllers. They do not determine the purposes for which and the manner in which any personal data are to be processed by the data processors.

4. British Orthopaedic Association (BOA)

The BOA is the Surgical Specialty Association for Trauma and Orthopaedics in the UK. The 2012 GIRFT ‘national professional pilot’ was hosted by the BOA (and RNOH), as the clinical lead and founder of the GIRFT programme – was the president of the BOA at that time. As previously described, the GIRFT programme is now co-delivered by the RNOH and NHS Improvement. As this is an independent evaluation, the BOA have no role in the conduct of the research, nor are they data controllers. They do not determine the purposes for which and the manner in which any personal data are to be processed by the data processors.

The proposed dissemination plan is multi-faceted, acknowledging the wide range of stakeholder groups with an interest in the GIRFT programme. The researchers will draw on expertise available within NIHR CLAHRC North Thames to support dissemination (e.g. communication and implementation expertise), to ensure that outputs are produced in the most appropriate format and targeted to the most appropriate audience, to speed up the adoption of the evaluation findings into practice.

Likely outputs that will utilise the linked NJR-HES-PROMs dataset are listed below:

• Two manuscripts are proposed for peer-reviewed journals aimed at the orthopaedic community (target journals include British Medical Journal and JAMA Surgery). Acknowledging the likely turnaround timelines for journal publication, it is anticipated that all manuscripts will be in press by mid-2020. The researchers will also explore publishing a summary of the findings in the Journal of Trauma and Orthopaedics, the BOA’s journal for trauma and orthopaedics professionals (mid 2019).

• As National Joint Registry (NJR) data will be used in the evaluation, the researchers are required to provide a 6-month interim progress report, and final report to the NJR (dates conditional on data release and final HQIP approval). These reports will be targeted at members of the NJR Research Sub-committee (clinicians, researchers, statisticians and economists); and will not be published on the NJR website or be made publicly available.

The number of each output type may be subject to change, depending on the level of interest in the work. Key findings are likely to include descriptive changes, impact of the intervention on processes, and impact of the intervention on outcomes and cost-effectiveness.

Likely outputs derived from the linked NJR-HES-PROMs dataset, as well as the wider evaluation are listed below:

• For those delivering and funding the programme (GIRFT programme team, NHS Improvement and the Royal National Orthopaedic Hospital (RNOH)) – a workshop will be held towards the end of the evaluation to summarise and discuss the findings. The audience will include project managers, clinicians, healthcare professionals, data analysts, policy makers and civil servants (Summer 2019).

• There has been considerable interest in the evaluation amongst the orthopaedic community. Therefore, a summary presentation will be offered to specialist groups, such as the British Hip Society (Summer 2019), aimed at health care professionals, patient representatives and researchers in the field of orthopaedic surgery. In addition, an abstract of the findings will be submitted to the British Orthopaedic Association’s (BOA) National Congress (September 2019).

• An abstract of the findings will also be submitted to the 2019 UK Health Services Research Network symposium (July 2019). Attendees include researchers, clinicians, policymakers and patient representatives with an interest in the organisation and delivery of health care services.

• Drawing upon findings from the wider evaluation, three further journal articles are also proposed for the wider academic community, with an interest in organisation and delivery of future improvement programmes (target journals include Implementation Science, Social Science and Medicine, and Health Policy). Again, it is anticipated that all manuscripts will be in press by mid-2020.

To reach as wide an audience as possible, researchers at UCL will make use of the NIHR CLAHRC North Thames social media channels, including tweeting links to outputs online, as well as posting links on the NIHR CLAHRC North Thames and UCL websites. CLAHRC BITEs (Brokering Innovation Through Evidence) will also be produced, which are headline ‘need to know’ summaries of high quality evidence based research findings, made available through the national NIHR CLAHRC network.

Patient and public involvement is a core part of the evaluation. A patient representative is a member of the bi-monthly steering committee, which reviews and advises on study progress. This individual has also been involved in the design and delivery of qualitative research components, for example co-facilitating focus groups. Researchers at UCL have also drawn on the support of the NIHR CLAHRC North Thames Communications and Patient and Public Involvement / Engagement (PPI/E) Officer and lay Research Advisory Panel (RAP). Members of the RAP have provided feedback on the design of the evaluation and study documentation. They also contributed to a consultation exercises, seeking public feedback about the acceptability of using pseudonymised patient data in the study. This was approved by the Health Research Authority Confidentiality Advisory Group.

All outputs will be aggregated with small number suppression in line with the HES analysis guide.

This aspect of the evaluation has not yet yielded benefits, as researchers at UCL are still in the process of obtaining a fully linked NJR-HES-PROMs dataset to the applicable cohorts This is required to answer the research questions.

Following successful receipt of the linked dataset in early 2019, and dissemination of the findings, researchers at UCL anticipate the benefits outlined in Section 5d.ii. will be achieved by mid-2020.

The NJR have agreed to provide researchers with an additional year of data to extend the cohort (1st April 2017 – 31st March 2018) required for the linkage. The researchers require NHS Digital to re-do linkage for the HES and PROMs datasets for the cohort in order to satisfactorily answer the research questions and deliver the outputs and benefits as stated above.

Background:

Researchers from University College London (UCL) are conducting an evaluation of the high profile ‘Getting it Right First Time’ (GIRFT) improvement programme currently being rolled out nationally in orthopaedic surgery. The first GIRFT report was published in 2012, by the then president of the British Orthopaedic Association (BOA). It recommended changes to NHS orthopaedic practice to improve patient outcomes, and achieve cost savings. NHS England subsequently funded a national professional pilot which involved senior clinicians visiting providers across the country to offer bespoke recommendations about improving care. By July 2015, the GIRFT project team, which is hosted on behalf of the BOA, at the Royal National Orthopaedic Hospital in Stanmore, had carried out reviews with 120 NHS Trusts in England. The project highlighted significant variations in practice and outcomes; device and procedure selection; costs; and infection rates. Consequently, the Department of Health and Social Care commissioned a three-year programme to address these challenges initially in orthopaedics, and then across ten other clinical specialities.

The wider GIRFT programme has now been expanded to over 37 specialties. Work is underway to enhance patient safety by addressing the complex issue of unwarranted variation in elective orthopaedic surgery. The programme aims to address this, by prioritising the following areas: increasing minimum volumes of complex procedures undertaken by individual surgeons, enforcing ring-fenced beds to reduce infection rates, reducing length of stay and waiting times to improve efficiency for patients, improving theatre efficiency, aiming for more competitive procurement processes, reducing unnecessary procedures (i.e. therapeutic arthroscopies), and gaining a better understanding of the causes of litigation. The programme also has strong links with Lord Carter’s Procurement and Efficiency programme.

Evaluation:

The GIRFT programme seeks to change practice, (e.g. prosthesis choice) in order to improve patient outcomes (e.g. infection rates). It can be defined as a complex intervention, containing several interacting components, targeting different stakeholders and addressing a number of different behaviours. There is, however, limited evidence to support the approach being used. Whilst there have been reorganisations of a number of acute NHS services in recent years, there have been few examples on this scale. The UCL evaluation aims to contribute to the development of that evidence base by studying in depth the implementation of changes to orthopaedic services.

UCL are evaluating the intervention as a whole, including the original pilot and the work now being carried out. The researchers are focusing on outcomes directly relating to patient care, for the commonest elective orthopaedic procedures: total hip replacement (THR) and total knee replacement (TKR). Through this, they will identify lessons to inform future service improvement efforts. A cohort of patients for linkage will be provided from the National Joint Registry (NJR), and returned to UCL.

Relationship between UCL and GIRFT:

Although some members of the GIRFT team are listed as co-authors on the published study protocol, UCL are undertaking an independent evaluation of the programme. Indeed, the GIRFT Team are keen to ensure that the UCL evaluation provides an independent analysis of their work. The GIRFT team are neither funders nor co-applicants on the revised NJR application (or any other data requests they are making as part of the evaluation). To facilitate understanding of the programme, the GIRFT team provide UCL with key documents, dates and details of their visits and re-visits, as well as providing UCL with access to key events undertake non-participant observations. UCL maintain contact with the GIRFT team to ensure both remain up-to-date with any developments of the national programme, which may influence UCL's ability to collect data, as well as influence the development of the findings. However, they have no role in the conduct of the evaluation.

The researchers’ overall aim is to evaluate the implementation of a complex intervention, the GIRFT programme which seeks to improve the quality and cost-effectiveness of NHS orthopaedic care in England. In doing so, they will identify lessons to guide future improvement work in other services. Their objectives are to use formative and summative evaluation methods to:

1. Examine the key processes of – and factors influencing - the ongoing development and implementation of planned improvements to orthopaedic services, nationally and locally, including any unintended consequences;

2. Assess whether the GIRFT programme has reduced variations in

orthopaedic practice and expenditure, and improved patient outcome measures;

3. Estimate the economic impacts (e.g. cost savings) of these changes; and

4. Explore patient and public perceptions of the planned improvements to care.

Reflecting these objectives, the evaluation comprises four work streams: (WS1) an organisational study; (WS2) a quantitative study; (WS3) an economic study; and (WS4) patient and public focus groups.

Data request:

This agreement refers to the quantitative study (WS2) and the economic study (WS3). In order to assess whether the GIRFT programme has reduced variations in orthopaedic practice, and improved patient outcomes, longitudinal, provider-level data are required. With details of all admissions to NHS hospitals in England, the Hospital Episode Statistics database represents the only means by which the UCL researchers can evaluate the impact of the intervention nationally. Similarly, there are no alternative sources to the Patient Reported Outcome Measures database, which would provide data from all providers.

The researchers plan to link the data being requested from NHS Digital (i.e. Hospital Episode Statistics and Patient Reported Outcome Measures) to data from the National Joint Registry. The dataset will then be used to examine: the proportion of procedures by low volume surgeons and in low volume hospitals; the use of prostheses not recommended by GIRFT; rates of readmission and revisions; outcomes reported by patients in their post-operative PROMs questionnaires; and resource use such as prosthesis type and length of hospital stay). Researchers will also inform an estimate of the economic impacts (e.g. cost savings) of the changes.

Clarification of Roles:

UCL have provided the following information to clarify the roles of parties mentioned in their fair processing notice and protocol documents.

1. NIHR CLARHC (Collaboration for Leadership in Applied Health Research and Care) North Thames

National Institute of Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) conduct world class applied health research which will have a direct impact on the health of patients with long term conditions and on the health of the public. CLAHRCs are collaborations between universities, the NHS (local service providers and commissioners), local authorities, industry, patients and the public, the local Academic Health Science Network (AHSN) and other relevant organisations in the region. NIHR CLAHRC North Thames is one of 13 CLAHRCs across England. It covers the geographical area of north central and north east London, south and west Hertfordshire, south Bedfordshire and south west and mid-Essex. The research team at UCL work within the NIHR CLAHRC North Thames structure.

2. The GIRFT programme team

This is an independent evaluation. The GIRFT programme team have no role in the conduct of the research, nor are they data controllers. They do not determine the purposes for which and the manner in which any personal data are to be processed by the data processors.

3. Royal National Orthopaedic Hospital (RNOH)

The GIRFT programme began from the RNOH, as the majority of the original GIRFT programme team were employed by this organisation. It is now co-delivered by the RNOH and NHS Improvement. As this is an independent evaluation, the RNOH have no role in the conduct of the research, nor are they data controllers. They do not determine the purposes for which and the manner in which any personal data are to be processed by the data processors.

4. British Orthopaedic Association (BOA)

The BOA is the Surgical Specialty Association for Trauma and Orthopaedics in the UK. The 2012 GIRFT ‘national professional pilot’ was hosted by the BOA (and RNOH), as the clinical lead and founder of the GIRFT programme – was the president of the BOA at that time. As previously described, the GIRFT programme is now co-delivered by the RNOH and NHS Improvement. As this is an independent evaluation, the BOA have no role in the conduct of the research, nor are they data controllers. They do not determine the purposes for which and the manner in which any personal data are to be processed by the data processors.

The proposed dissemination plan is multi-faceted, acknowledging the wide range of stakeholder groups with an interest in the GIRFT programme. The researchers will draw on expertise available within NIHR CLAHRC North Thames to support dissemination (e.g. communication and implementation expertise), to ensure that outputs are produced in the most appropriate format and targeted to the most appropriate audience, to speed up the adoption of the evaluation findings into practice.

Likely outputs that will utilise the linked NJR-HES-PROMs dataset are listed below:

• Two manuscripts are proposed for peer-reviewed journals aimed at the orthopaedic community (target journals include British Medical Journal and JAMA Surgery). Acknowledging the likely turnaround timelines for journal publication, it is anticipated that all manuscripts will be in press by mid-2020. The researchers will also explore publishing a summary of the findings in the Journal of Trauma and Orthopaedics, the BOA’s journal for trauma and orthopaedics professionals (mid 2019).

• As National Joint Registry (NJR) data will be used in the evaluation, the researchers are required to provide a 6-month interim progress report, and final report to the NJR (dates conditional on data release and final HQIP approval). These reports will be targeted at members of the NJR Research Sub-committee (clinicians, researchers, statisticians and economists); and will not be published on the NJR website or be made publicly available.

The number of each output type may be subject to change, depending on the level of interest in the work. Key findings are likely to include descriptive changes, impact of the intervention on processes, and impact of the intervention on outcomes and cost-effectiveness.

Likely outputs derived from the linked NJR-HES-PROMs dataset, as well as the wider evaluation are listed below:

• For those delivering and funding the programme (GIRFT programme team, NHS Improvement and the Royal National Orthopaedic Hospital (RNOH)) – a workshop will be held towards the end of the evaluation to summarise and discuss the findings. The audience will include project managers, clinicians, healthcare professionals, data analysts, policy makers and civil servants (Summer 2019).

• There has been considerable interest in the evaluation amongst the orthopaedic community. Therefore, a summary presentation will be offered to specialist groups, such as the British Hip Society (Summer 2019), aimed at health care professionals, patient representatives and researchers in the field of orthopaedic surgery. In addition, an abstract of the findings will be submitted to the British Orthopaedic Association’s (BOA) National Congress (September 2019).

• An abstract of the findings will also be submitted to the 2019 UK Health Services Research Network symposium (July 2019). Attendees include researchers, clinicians, policymakers and patient representatives with an interest in the organisation and delivery of health care services.

• Drawing upon findings from the wider evaluation, three further journal articles are also proposed for the wider academic community, with an interest in organisation and delivery of future improvement programmes (target journals include Implementation Science, Social Science and Medicine, and Health Policy). Again, it is anticipated that all manuscripts will be in press by mid-2020.

To reach as wide an audience as possible, researchers at UCL will make use of the NIHR CLAHRC North Thames social media channels, including tweeting links to outputs online, as well as posting links on the NIHR CLAHRC North Thames and UCL websites. CLAHRC BITEs (Brokering Innovation Through Evidence) will also be produced, which are headline ‘need to know’ summaries of high quality evidence based research findings, made available through the national NIHR CLAHRC network.

Patient and public involvement is a core part of the evaluation. A patient representative is a member of the bi-monthly steering committee, which reviews and advises on study progress. This individual has also been involved in the design and delivery of qualitative research components, for example co-facilitating focus groups. Researchers at UCL have also drawn on the support of the NIHR CLAHRC North Thames Communications and Patient and Public Involvement / Engagement (PPI/E) Officer and lay Research Advisory Panel (RAP). Members of the RAP have provided feedback on the design of the evaluation and study documentation. They also contributed to a consultation exercises, seeking public feedback about the acceptability of using psedonymised patient data in the study. This was approved by the Health Research Authority Confidentiality Advisory Group.

All outputs will be aggregated with small number suppression in line with the HES analysis guide.

This aspect of the evaluation has not yet yielded benefits, as researchers at UCL are still in the process of obtaining a linked NJR-HES-PROMs dataset. This is required to answer the research questions.

Following successful receipt of the linked dataset in early 2019, and dissemination of the findings, researchers at UCL anticipate the benefits outlined in Section 5d.ii. will be achieved by mid-2020.