Safety and quality indicators for hospital performance: an observational study in English hospitals

This Data Sharing Agreement permits the retention of the data provided under previous iterations of this Agreement for an interim period. This is a pragmatic approach to provide an active Agreement whilst enabling The University of Sheffield to complete the necessary actions to enable a subsequent application to extend the Agreement meeting all applicable data sharing standards as published in NHS Digital’s website (see: https://digital.nhs.uk/services/data-access-request-service-dars/dars-guidance).

There is considerable public interest in the early identification of poorly performing hospitals which are providing unsafe or poor quality care to the patient populations they serve. The analysis of large sets of routine, observational data will be used to assess and compare the safety and quality of care provided by different hospitals.

“Unsafe” hospitals may be characterised as those having higher numbers of serious adverse events in low risk admissions than could be expected just by chance. Although similar ideas have been examined previously, these studies have been based on low risk diagnostic groups rather than low risk patients. Based on this concept, and using different measures to assess expected and observed outcomes, the performance of hospitals will be compared. Evidence of validity will be gathered by triangulating with other sources of evidence about safety concerns; by examining face validity based on determining what sort of 'low risk' patients having adverse events are being identified; and examining temporal stability as an indication that a 'characteristic' of the hospital is being measured. While some of the differences between hospitals are a result of intrinsic patient related factors (such as differing levels of deprivation, geography, comorbidity, and age), other differences seem likely to be intrinsic to the service factors offered within each hospital.

This research complements the work done in the development and validation of the Summary Hospital-level Mortality Index (SHMI), and in the development of methods for risk standardization and performance measures. The index was commissioned by the department of health and developed by ScHARR using five years of HES patient data. The SHMI provides a simple numerical tool for use in the identification of hospitals in need of further investigation.

The project will provide evidence for policy makers and hospital administrators to target appropriate interventions relating to the quality and safety of treatment offered to patients.

The applicant is requesting a de-identified data set and will not request sensitive items of data. The data set requested will be analysed and the analysis presented at research meetings. It is envisaged that research publications in peer reviewed papers will also be generated from the data. These conference presentations and papers will report aggregated results across patient episodes and the results will be based on statistical analysis generated from the data (typically in the format of tables, graphical representations and text).

The dataset itself will not be released to any third party, including other staff within the University of Sheffield. The individual patient episodes within the data will not be disclosed at any stage in the reporting of the results.

The GDPR lawful basis for University of Sheffield to process this data is Article 6(1)(e) 'task in the public interest' and Article 9(2)(j) 'scientific research'.

Five years of routine population level data will be examined. The data requested expands over a period of five years to ensure that the standardised risk model detects trends and is consistent with that of previous studies. The stability of the model through time is essential to ensure its validity; in particular, to ensure that meaningful changes can be detected and better distinguished from noise.

Data will be drawn from HES Admitted Patient Care for non-specialist hospitals in England. The processing will be undertaken within a number of stages as outlined below:

Stage 1: Secure management and analysis of hospital episode statistics (HES) for all in-patient admissions to non-specialist English hospitals.

Stage 2: Development and validation of hospital performance measures focused on patient safety, and statistical modelling to estimate the risk of adverse events (e.g. death in hospital) for every admission. Logistic regression models, using the covariates explored in the development of SHMI will be employed to estimate the risk of death (or other serious adverse event such as unexpected transfer to critical care) for each admission. Analysis of possible association between the day of the week an admission takes place and an increased risk of adverse outcome.

Stage 3: Comparison of risk adjusted event rates indicating unsafe performance between hospitals, and development of appropriate graphical methods such as funnel plots to compare casemix or risk adjusted event rates between hospitals.

Stage 4: Development of a critique of the methods to provide a clear interpretation of the results and their limitations.

Stage 5: Elaboration of a report on the findings, and potential submission to publication.

The data will be processed and analysed within a single department within the University of Sheffield, the School of Health and Related Research (ScHARR). The data will not, in any circumstances, be released to a third party, nor will it be used for commercial purposes.

The data will be accessed by a restricted number of authorised individuals within the study team. Authorisation to the specific project system folder will be given to a strictly limited number of individuals, namely the principal investigators and the statistician. All these individuals work within the same department (ScHARR) at the University of Sheffield and are subject to strict confidentiality and secure data management policies.

The data will be cleaned and analysed by the statisticians, the analysis of the dataset will be performed in line with the protocol approved by ScHARR Ethics committee, and will abide by the University’ of Sheffield’s Ethics Policy. The data will be reported to the other members the School in the form of a report detailing the aggregated results.

The project folder, containing the data, will be located in a networked PC that is username and password protected. Only specific users and specific on-site machines will be granted permission to the project folder. All access will be logged.

After the three year period the data will be destroyed under supervision of ScHARR IT staff.

The Department (School of Health and Related Research) has achieved IGTK approval level ‘satisfactory’ in order to process large routine data.

The outcomes expected from this analysis include:

• a report of the analysis that can identifying hospitals that need further investigation, indicators of hospital performance will be produced for each non-specialist hospital (November 2019),

• a Master’s thesis (though this should not be considered a major output within this project) was submitted on October 2018

• abstracts, posters presented at academic conferences (with a target date of March 2020),

• a first journal article will be submitted to peer reviewed health-related journals (with a target date of November 2019).

• a possible second journal article will be submitted to peer reviewed health-related journals (with a target date February 2019).

The project will provide evidence for policy makers and hospital administrators to target appropriate interventions relating to the quality and safety of treatment offered to patients.

The data set will continue to be analysed, and the findings of the analysis will be presented at research meetings. It is envisaged that at least two research publications in peer reviewed papers will be generated from the data. These conference presentations and papers will report aggregated results ACROSS patient episodes and the results will be based on statistical analysis generated from the data (typically in the format of tables, graphical representations and text).

Target journals for publication include the BMJ, the Journal of Health Services Research and Policy, the BMC Medical Research Methodology, and the BMC Health Services Research, the BMJ Quality and Safety.

The data set itself will not be released to any third party, including other staff within the University of Sheffield. The individual patient episodes within the data will not be disclosed at any stage in the reporting of the results. Furthermore, the report will not serve to identify any hospitals. It will be focused on the potential of the method to identify any hospitals where there are concerns about the relative safety of care. It will therefore be focused on potential users of the methods (i.e. Department of Health and HSCIC).

The report will be sent directly to contacts in the Department of Health and also the SHMI Technical Working Group at HSCIC, with whom ScHARR developed the SHMI model.

The number of patient deaths that has been associated with non-optimal medical practices under hospital care has been of increasing concern to health and social care policy makers and practitioners. An increased demand for accountability at the level of these institutions has spurred the need for better measures of performance, and of comparison, between health care services.

A significant variation in mortality rates according to week day of admission has been shown by several studies made in English hospitals. A greater understanding of this phenomenon would be of great benefit to administrators, regulators, and practitioners, as a potential problem locator identifying areas where further investigations and funding should focus into.

This study has provided updated measures of hospital performance, identifying outlying (underperforming) hospitals. The study has also provided a hospital specific analysis of possible association between admission’s day of the week and increased mortality rates. Further specific analyses will focus on subgroups of the patient population (such as elderly and frail patients and in particular those admitted during the weekend). These analyses will be examined for stability over time, and variability.

The potential benefits to accrue to the Health and Social care systems will be an evidence based, current report on the performance of non-specialist hospital in England, distinguishing issues of safety and quality of care, and including an analysis of stability through time and trending. The report will be targeted at health and social care regulators and practitioners as well as scholars.

This project continues the work of SHMI and is being processed by the same SHMI team who have been addressing the issue of hospital performance since the early 2000s, and although not new, the issue is still very much relevant to the current healthcare system. Many crucial questions regarding the understanding of the complexity of hospital safety and performance remain unanswered. The particular concern of the team is that an analysis of hospital performance should distinguish between high and low risk patient groups, and should investigate its relation to other factors such as day of week of admission. Although University of Sheffield have made significant progress in the understanding the dynamics between hospital safety and performance, many other crucial questions remain unanswered, and some of the work and analysis may require revisiting as results go through the refereeing and publishing process. The main output from this research will be a report of the analysis that can identify hospitals that might need further investigation because of safety issues (identified by relatively poor performance in low risk patients) or quality issues (identified by relatively poor performance in high risk patients). If hospitals are identified that manifest poor performance that cannot be explained by the data this report will be supplied directly to the Department of Health.

It is difficult to quantify how policy would be impacted by this work, but this research does continue the work done for SHMI, and so SHMI makes a good example. Around 60% of all deaths occur in hospital and preventing avoidable deaths is an essential objective for health services. The Francis Report on the Mid-Staffordshire Hospital Trust in Feb 2013 showed that excess mortality for the Trust was associated with poor care. The Summary Hospital Mortality Index (SHMI) was developed as a direct result of research carried out at the School of Health and Related Research (ScHARR). 14 hospitals (many of which were also identified ScHARRs report of 2011) were identified by the Department of Health as having unacceptably high mortality, over two years using the Sheffield SHMI, amongst other measures. The consequence was that the Care Quality Commission sent teams in to investigate the care of patients at these hospitals and this was reported in the Keogh report (2013) which ultimately impacted staff, patients and hospital systems with the aim of improving patient outcomes. University of Sheffield expect this work will have similar impact, as it aims to improve significantly on existing methodologies.

University of Sheffield have identified two novel mortality indices, the SHMI-Q, and the SHMI-S, based on the SHMI methodology. The two indices are tailored to assess the performance of hospitals that is specifically related to the quality (SHMI-Q) and the safety (SHMI-S) of care provided to patients, based on the analysis of routinely collected data.

University of Sheffield consider unsafe, care provided by hospitals who experience a higher than expected mortality in-patient admissions deemed of relatively low mortality risk. In contrast, quality of care is seen as care provided by hospitals who experience lower than expected levels of mortality in admissions deemed, of relatively high mortality risk.

The five year HES data set requested from the NHS Digital was used to develop these indicators. Annual SHMI-Q and SHMI-S indices were computed to compare the performance of 136 acute general hospitals in England between April 2010 and April 2015, and identify trusts where Quality and Safety of hospital care was observed.

Within this period, University of Sheffield identified 28 trusts where care deemed unsafe was present, and 32 trusts where care provided was deemed of quality.

University of Sheffield hope this work will feed into the current mechanism for assessing and monitoring hospital performance, as a way to discern between trusts providing care that is deemed unsafe and trusts providing care that is not necessarily of quality, going beyond the current performance assessment that is purely based on the detection of higher than expected mortality.