Making CCG operational planning more robust by using a large activity sample size to derive analytics

Statistical and mathematical modelling is the process of taking a real world problem or system and creating a simplified description of it (a model) – using mathematical concepts and language. The model is designed to capture features of interest in the real world system and provide means to study the interrelation of those features, make predictions, or study the possible effects of changing some of those features.

In a healthcare planning context, a model will typically take a mixture of quantitative/numerical variables (corresponding to concepts such as arrivals at a hospital, length of stay, age, cost, lab results, clinical scores, or counts of activity) and categorical variables (corresponding to things like type of illness, sex, or smoking status), and codify the relationships between them using mathematical formulae.

In situations where outcomes are uncertain (such as when making predictions, when there is potential measurement error, or when there is a random element to when or how individual events occur), the concept of a probability distribution is particularly useful in constructing models. A probability distribution takes the measure of interest (hospital length of stay, for example) and quantifies the chance that each value of that measure occurs. Real life data can often be well approximated by theoretical probability distributions which are described by established mathematical formulae and whose properties are well understood. This is a more fine-grained (and realistic) approach than simply describing the measure by its average (mean or median) value alone.

An extremely simple and unrealistic mathematical model might describe, for example, length of stay in an inpatient ward as a simple formula which calculates the predicted length of stay for a given patient would be some constant multiplied by their age on admission. A more sophisticated model might predict the length of stay based on a combination of age, sex and frailty, multiplying each by a constant and adding them together to get the expected length of stay.

An alternative model might account for some of the natural random variability in the process – obviously, not all patients admitted to a ward where the average length of stay is 3 days will stay for exactly 3 days. To account for this, the model might instead predict lengths of stay for individual patients by choosing values at random from a probability distribution. The particular distribution could be chosen so that it agrees with some easily obtainable summary measures (such as the mean and standard deviation) of the actual data. The choice of length of stay value to use from the distribution could be made by a process similar to flipping a coin or rolling a die, then reading of the value which corresponds to the number which comes up. More sophisticated approaches, and/or ones which made different assumptions, could also be taken.

The terms “events” and “shocks” in the sections below have been used interchangeably to refer to factors which affect the generalisability of the data or its reflection of system behaviour expected/normal circumstances, including both data quality problems and deliberate operational interventions.

NHS Bristol, North Somerset and South Gloucestershire Clinical Commissioning Group (BNSSG CCG) commissions, plans and monitors healthcare services for a population of approximately 1.1 million people in a sub-region of South West England based around the Bristol urban centre. It was created in 2018 by the merger of three predecessor organisations and as part of that process took on greatly enhanced responsibility for system-wide data analytics, including the creation of a specific modelling and analytics (M&A) team. This team’s remit is to provide advanced analytical support on both business-as-usual (BAU) and transformation projects to the CCG and the local Sustainability and Transformation Partnership/emergent Integrated Care System. This involves work with local acute, mental health, community and primary care providers, and to enhance existing business intelligence and analytics capacity in the system by providing access to more sophisticated and accurate methods and outputs than were historically available. Note that this reorganisation included the in-housing of many analytic functions traditionally out-sourced to Commissioning Support Units, hence BNSSG CCG data requirements may be more extensive than those for CCGs in which analytics expertise is bought from an external supplier.

Standard reporting against nationally reportable standards, and traditional, basic, activity and cost summaries are routinely produced by non-specialist analysts using SUS (Secondary Use Services) SEM (a specific file format) data restricted to the CCG’s registered population.

However, data which is both more granular and raw in definition, and more extensive in geographical coverage (for the whole of England), is required by highly specialised mathematical analysts within the CCG’s M&A team to construct metrics needed for more sophisticated modelling techniques. The BNSSG CCG will use Hospital Episode Statistics (HES) data to derive robust estimates of key operational metrics including probability distributions for inpatient hospital care length of stay (LOS), inpatient, outpatient, and emergency arrival rates, waiting times and associated measures (as opposed to single number summary statistics). All of these will be determined for defined clinical, demographic, time-period and potentially geographic (e.g. urban vs rural) and provider-type profiles in order to make them meaningfully applicable to operational and strategic planning, service redesign, and intervention analysis projects.

The analysis will use pseudonymised non-sensitive patient-level data to derive empirical probability distributions for the processes of interest, and from these fit theoretical distributions which can be used in simulation modelling and statistical analysis of existing and proposed patient pathways. Using local data alone is not a reliable option since; when considering activity within specified clinical, demographic and time profiles the effects of individual extreme values in the data, periods of poor data quality/missing data, service transfers, interventions and system reconfiguration, as well as the inevitable small numbers in some groupings (even in a population of greater than 1 million), mean derived distributions would be unreliable and the level of variability which would need to be included in estimate would severely constrain its usefulness. By both greatly increasing the effective “sample size” and smoothing out the effect of local time-limited service changes and data quality issues, using all-England patient level data will mitigate the effect of these events/shocks in any given local system (such as temporary consultant vacancies, ward closures, service suspensions, electronic system changes, etc.) and allow for the construction of robust, practically usable distribution estimates. These will in turn facilitate improved accuracy in projection, planning and monitoring of activity which relies on models built using those metrics. Using data at an all-England level as opposed to single CCG level is consistent with the approach historically taken by the NHS nationally, for example in the construction of NHS England’s Indicative Hospital Activity Model (IHAM).

Examples of the type of modelling which the outputs of this project will be used to support include (but are not restricted to): discrete event simulation (DES) modelling of multi-stage patient pathways which include both hospital admission and outpatient appointments; system dynamics modelling of urgent care patient flows; and comparison of local admitted patient care pathway performance for specific treatments following local policy interventions to expected “do nothing” scenarios. Modelling is currently being done in all of these contexts within BNSSG, but the precision, accuracy and interpretability of the results is constrained by local data limitations and could be greatly improved by the use of more robust distributions and metrics, derived from data with national coverage.

Because much hospital activity exhibits seasonality, and to account for the effect of year-on-year changing policy and financial incentives, a minimum of three years of data is required to derive reliable estimates. To make it relevant to future and current planning and analysis, the data needs to be from the most recent available periods. To ensure the maximum possible feasible data minimisation, this request has thus been restricted to the most recent three years of data. HES data has been chosen because it contains a sufficient degree of granularity to allow sub-setting into groups which match how specific services are (or may be) constructed, in terms of which patient groups they serve.

All organisations party to this agreement must comply with the Data Sharing Framework Contract requirements, including those regarding the use (and purposes of that use) by “Personnel” (as defined within the Data Sharing Framework Contract i.e.: employees, agents and contractors of the Data Recipient who may have access to that data).

There will be no data linkage undertaken with NHS Digital data provided under this agreement that is not already noted in the agreement.

NHS Digital will provide HES APC (Admitted Patient Care), A&E (Accident and Emergency) and OP (Outpatient) data to BNSSG CCG, via NHS South Central and West Commissioning Support Unit (SCWCSU), which provides IT and database administration support services to the CCG. The data controller will be the BNSSG CCG, and they will also be the main data processor. SCWCSU will also be a data processor, but its activities will be restricted to download and storage of the data on behalf of BNSSG CCG. The CSU/CCG infrastructure is within the NHS’s secure N3 network and the HES data will be stored in a secure MS SQL Server data warehouse within that infrastructure. Access to that data will be restricted, through Microsoft Active Directory access control, solely to permanent staff members of BNSSG CCG - specifically the M&A team. SCWCSU will act only upon specific instruction from BNSSG CCG M&A on this project, and the data will be stored only at those addresses listed in the storage addresses section of this application. There will be no flow of data into NHS Digital. All substantive analysis and use of the record level data will be restricted to the BNSSG CCG M&A team - all of who are substantive employees of the organisation.

Data minimisation has been applied through restricting the timeframe of the data requested to the most recent three years only, by requesting only pseudonymised data, and by excluding those fields classified as “sensitive”. Further minimisation, such as filtering by age or geography or excluding specific data fields from the request, were considered but deemed incompatible with the purposes specified under “objective for processing” above.

The HES data will not be available to third parties and will not be taken out of BNSSG CCG offices. Record level data will not be shared outside the BNSSG CCG. Results of analysis will be at an aggregate level and will be shared only at that level (in the form of graphs, summary statistics, and parameters to derived probability distributions), with small number suppression applied as per HES guidelines. It will not be possible to identify specific patients, clinicians or other individuals from these outputs.

The data will not be onwardly linked with any other record-level patient data. The narrative descriptions of coded data values in the HES extract (e.g. treatment function codes, administrative category codes etc.) may be looked up in the NHS Data Dictionary.

Data will relate to the whole of England, not just the registered population of BNSSG CCG – this is an explicit requirement of the project, as explained above in the “objective for processing” section.

Results of the analysis will be aggregated to service/pathway level, incorporating small number suppression as per the HES guidelines where necessary, and will not identify (or allow the identification of) specific patients, clinicians or other individuals.

The following outputs are anticipated to be created upon accessing the HES data from NHS Digital.

Specialty-level growth projections with updated estimates taking into account national trends - this is a major project due for completion 2023.

Specialty-level length of stay estimates, with updated estimates taking into account national information - this is a major project due for completion 2023.

A range of similar analyses to determine appropriate metrics to support system-wide operational planning and monitoring (including but not restricted to demand and capacity planning). Full detail and timelines are to be reactively determined from having first completed initial examination and assessment of the data, but this is not anticipated to be extending beyond 2023.

The majority of these outputs (all of which will be aggregated with small number suppression in line with the HES analysis guide) will be contained within written Word document reports, PowerPoint Presentations and potentially a visualisation tool for interactive data for use within BNSSG CCG and the wider BNSSG NHS-funded system (including acute hospitals, community and mental health providers, and GPs).

Results may also be used in research articles for peer reviewed academic journals and be presented at healthcare or academic conferences. For example, the M&A team has previously published results of local analytics work in academic journals (the Journal of the Operational Research Society, and Operations Research for Healthcare), and presented findings to groups including Bristol Health Partners (a collaboration between local NHS organisations and universities), the NHS Wales Modelling Collaborative, and the annual conference of the Operational Research Society, and the annual meeting of The Health Foundation, in order to share important practical results with the wider NHS and healthcare research community. Similarly, results may be shared with NHS colleagues through engagement with forums such as the NHS-R Community, the Association of Healthcare Analysts, and through sharing of reports on forums such as the NHS Future Collaboration message board and by informal correspondence with colleagues at NHS England/Improvement and other CCGs/ICSs.

Sharing with other CCGs

A number of distribution channels for such outputs are currently in various stages of maturation:

The NHS-R Community (a national project backed by the Health Foundation and NHS England/Improvement which brings analysts working with the statistical software together through conferences, workshops, regional networks and a website, with long term intentions to create a shared computer code repository accessible to NHS analysts and partner organisations). NHS Bristol, North Somerset and South Gloucestershire CCG is an active participant in this network and is running workshops at its forthcoming annual conference, and would anticipate doing the same at future events, featuring the outputs of the work completed using the HES data.

Local and regional NHS/healthcare analytics networks.

NHS Bristol, North Somerset and South Gloucestershire CCG is a leader in a local west of England analytics network being developed by Bristol Health Partners and a range of providers and planners from their own area, including some with cross-boundary responsibilities. Part of the purpose of this network is to establish communications between analysts in separate organisations so they can directly share expertise, analytical outputs and computer code. NHS Bristol, North Somerset, and South Gloucestershire CCG is also leading work setting up links which will form the basis of an extended network involving neighbouring CCG areas (for example with Gloucestershire CCG). In addition, NHS Bristol, North Somerset and South Gloucestershire CCG is working with regional networks in other areas, for example it is running workshops for the West Midlands analytical network on novel population segmentation methods it has developed, including sharing computer code to perform the analysis, and would envisage doing the same with the results of the work done with the HES data.

There are already close working relationships with the University of Bath Management School Centre for Healthcare Innovation and Improvement (where one of the team is a visiting research fellow) and the University of Bristol Population Health Sciences Institute and Elizabeth Blackwell Institute, as well as the NIHR South-West Applied Research Collaboration (ARC). NHS Bristol, North Somerset and South Gloucestershire CCG would be able to share relevant high-level outputs of the HES work with academics working in partnership with NHS organisations on defined projects. Availability of usable real-world data is well established a major barrier to realising practical benefits from research projects in academic statistical modelling and operational research (acknowledged in the work of the national Plethora project for example, which was set up in part to bridge the gap between academic knowledge/technique generation and practical application of those outputs by healthcare analysts at the NHS coalface).

Dissemination of findings would also include posting of relevant results and outputs on the future NHS Collaboration platform, in addition to posting of relevant code, reusable output summary data, documentation, and (high level aggregate) data (with small number suppression) to an NHS BNSSG Analytics GitHub repository (subject to IG approval).

Publication of practical studies performed using the HES data in academic journals. This could include case studies of simulation projects carried out using the data, or specific statistical analyses. For example, the applicant is currently working on an investigation into the effect of day of week of hospital discharge on total length of stay and is subsequent association with other factors such as readmission rates. These are live questions in local policy formulation and are equally relevant to other organisations.

All outputs will only contain aggregated data with small number suppression applied in line with the HES analysis guide.

Update for extension, July 2020:

During the first six months of the project substantial progress has been made on the initial stages of the work outlined above (despite the inevitable disruption cased by the Covid-19 crisis, which the team working on this project have had to divert substantial amounts of their resources towards).

On initial receipt of the data some time was devoted to data exploration to facilitate understanding of data formats, data completeness, and key features, prior to proceeding with the specific analytical (sub)-projects. A number of those projects were then initiated, with more pending.

The data has been used to produce length of stay estimates for hospital admissions, at treatment specialty level. This has been done by fitting a range of probability distributions to data filtered by treatment speciality and choosing between different candidate distributions by minimising Aikaike Information Criterion. The results of this work (in terms of a named probability distribution, and numerical parameters for that distribution, for each treatment specialty) have been used to update a distribution fitting tool included in open source discrete event simulation software developed by the BNSSG M&A team, which has been made freely available online and widely promoted within NHS analytical communities. Further work to fit length of stay distributions to different sub-sets of data, such as by geography type or patient group, is ongoing.

A major piece of work to test the commonly used assumption that hospital/ward occupancy should be targeted at 85% of capacity in order to achieve the optimal balance between utilisation and the frequency of "crisis days" (when patients have to be turned away or outlined) is currently well advanced. An extensive series of simulations are being performed using the data to estimate the effects of different combinations of occupancy target, ward/hospital size, and what different effects are seen between treatment specialities and other specified groupings of activity. This analysis (which requires a substantial amount of computer time) is currently underway and the results will initially be investigated and validated within the local BNSSG system, prior to write-up for publication in a peer reviewed academic journal and presentation to the NHS analytical community through established peer networking groups, training sessions, and forums.

Another project to investigate the effects of day of week of admission and discharge on overall hospital length of stay is also underway. As with the 85% project, this work involves the use of computer simulation and distribution fitting to compare observed lengths of stay with what the lengths of stay would have been under the assumption that the probability of discharge was independent of the day of week (through censored maximum likelihood estimation). This work in in progress, with computer code written and dataset formatting performed. Once results have been finalised and analysed, the intention is to publish them in a peer reviewed journal and share them with the wider analytical community.

Other projects on readmission rates, validation of locally used ad hoc benchmarking measures, and factors affecting hospital length of stay are planned but awaiting the completion of the projects already begun, and the release of internal analytical capacity as the Covid-19 crisis eases.

Work on assessing the validity of the 85% occupancy target for inpatient wards through simulation, at treatment function level, has been completed and submitted to the Journal of Health Services Research and Policy for publication (currently awaiting peer review). Further work promoting and explaining the consequences of this work will be completed through engagement with NHS professional groups including the NHS-R Community and AphA, and through ongoing peer group meetings which the BNSSG M&A team regularly hold with analytical colleagues in other NHS organisations from across the country. Work on admitted patient length of stay is ongoing and will be promoted through similar routes once completed.

The potential benefits of accurate and reliable operational planning for healthcare service delivery in general are well established. It is also well established that unreliability of projections and metrics can make such planning difficult.

This data is being requested to support a project which directly addresses that problem by seeking to improve the accuracy and reliability of analytics used for practical, local system-wide operational planning of CCG-level healthcare services.

As already referred to in previous sections, reliance on analysis solely of local data to derive, for example, expected arrival, admission and length of stay metrics for particular types of service, can be subject to local events and shocks which mask the "natural" metrics of the service and distort analysis based solely on that local data. Additionally, when analysing data at a fine level of granularity (e.g. deriving metrics for a highly specialised service), the "sample size" in the local data is in some cases too small to make valid statistical inferences from, or else the small sample size leads to unhelpful levels of uncertainty in the estimates. This poses a problem when working to construct accurate and reliable operational and contract plans for the local system.

The national data will improve BNSSG CCG’s picture of (for example) underlying growth and enable the CCG to "fill in the gaps" which are inevitable in low sample sizes, and that can cause volatility in any resulting projections. For example, where an otherwise irregular surge in activity - perhaps only due to the opening of additional clinic slots - can have a material impact on any associated growth figures, which would not necessarily be borne-out in future years. Or where the odd very long LOS can have a drastic effect on the averages deduced.

The outputs will have a direct effect of decision making in a variety of strategic and operational settings. For example, inpatient length of stay (LOS) measures will be used in modelling which directly informs choices of capacity (in terms of bed numbers, staffing etc.) in service reconfigurations and design of new services. This includes multi-organisational patient care pathways - activity metrics will be used directly in the production and verification of activity projections used for operational planning and funding of future service levels; activity, LOS, and other measures will be used to derive more realistic performance measures for local services, allowing the generation of models to detect variations from expected activity.

The availability of robust metrics derived from data with a national coverage will greatly enhance the reliability of these modelling outputs with expected operational and financial benefits in the local health system, and potential for similar benefits to be realised by other areas building on the BNSSG work. The financial scale of the impact is hoped to optimise the use of the budget and the operational effect on correct service capacity estimates has a non-quantitative but significant effect on patient care and experience, in terms of waiting times, capacity to deliver appropriate treatment in urgent care settings, and the ability of the local system to absorb shocks in terms of changes to and variation in activity levels.

Initial use of the outputs in a local context would be within a matter of months and some wider dissemination of results and measures could follow soon after, while detailed assessment of the utility of the derived metrics would be assessed and adjusted over a period of subsequent financial years up to the project end in 2023.

National benefits

Probability distribution library

A key high value initial output of NHS Bristol, North Somerset and South Gloucestershire CCG's work with the national HES data will be the creation of a directory of length of stay/arrival rate probability distributions for a wide range of service point types, which can be scaled to match local requirements based on locally estimated averages.

For example, distributions associated with component service points of a knee surgery pathway, identified by filtering the data by a combination of treatment specialty, diagnosis, and procedure codes (necessary to construct real-life services from the data – from experience, NHS Bristol, North Somerset and South Gloucestershire CCG knows that while simple use of treatment specialty codes or high level diagnostic measures in isolation may be useful for some accounting purposes, it maps poorly onto the operational representation of actual services).

Distributions will be derived at these generic service levels, but more specific results will also be produced by further filtering by demographics, geography and/or provider type. The purpose of this further filtering will be to distinguish expected unique characteristics of the service/component wards and clinics in a range of meaningfully defined practical settings. One such distinction of settings could be a knee surgery pathway at a large specialist urban hospital with a population skewed to a younger age range, versus the same service at a smaller rural hospital serving a substantially older population.

Depending on the service type, and whether filtering yields a sufficient quantity of records in the given setting, time-specific distributions may also be derived for some service types (to account for time of day, day of week, or time of year seasonal changes in arrival rates or lengths of stay).

Distributions will be fitted to the data using standard techniques (such as maximum likelihood, moment matching etc.), implemented through the statistical programming environment R, which is rapidly gaining traction as the tool of choice for advanced analytics in the NHS.

The resulting library of arrival and length of stay distributions will (by design) be equally useful to any CCG, STP, ICS or other system-planning NHS organisation (and indeed to providers engaged in planning of pathways substantively contained within their own organisations) with a sufficiently advanced analytic workforce. The distributions would form a set of generic building blocks which a wide range of hypothetical services, in a wide range of practical settings, could be constructed.

They will effectively form a reference set of expected measures which could be used to construct models of activity when designing a new service which does not currently exist, when modifying the configuration of an existing service, or when assessing the performance of an existing service against the assumed generic case. By design, they will be generically applicable to any area of the country – local circumstances will be accounted for by choosing a set of measures which capture characteristics which analysts/modellers intend to be present in their local system, rather than using the geographical source of the data as a proxy for these characteristics (except when a geographical variable is the explicit characteristic of interest, e.g. in distinguishing between urban and rural areas).

One direct use of these measures will be in demand and capacity models, for which length of stay is a key input. In constructing simulation models (such as discrete event simulation) – one of the barriers to uptake of simulation within the NHS is the lack of availability of appropriate data to inform such models. NHS Bristol, North Somerset and South Gloucestershire CCG has separately developed its own simulation software which will be distributed nationally to NHS organisations in the coming months, and the library of distributions constructed using HES data could be subsequently disseminated to organisations using that (or other) software.

Relationship between day of discharge and length of stay (and other measures)

NHS Bristol, North Somerset and South Gloucestershire CCG has already instigated a local study into the statistical relationship between the day of week on which patients are discharged and the total hospital length of stay of those groups of patients, and also into potentially associated measures such as 30-day readmission rates. The stability of the results is potentially affected by non-recurrent factors in local data but by using all-England HES data, subsetted explicitly for the relevant characteristics, it should be possible to derive stable results which are generally applicable nationally and across specific areas. These results could then be used to assess the desirability of interventions to modify day-of-week discharge patterns and resource allocation in local systems.

From the local investigations already performed, it appears that there could be substantial scope for improving hospital efficiency, since many discharges appear to be delayed over the weekend until early the following week. Reducing these delays through a 7-day service would reduce hospital length of stay by 10%, which would correspond to material bed number savings, while receiving none of the drawbacks. National data is required to test this finding further, in both accounting for potential sample size insufficiency and checking whether local results hold nationally.

Other projects

There are a number of other system-level projects, both in scoping and already started, for which national data would be beneficial. For example, in re-visiting the 85% bed occupancy target where work in underway to assess whether this commonly used target for average bed occupancy should be tailored to ward and service types. The general principle is that more variability in arrivals and length of stay mean the average occupancy target should be pitched lower (in order to accommodate the greater impact of peaks and troughs), while less variability would mean being able to make more routine use out of the bed base with a lesser threat of damaging pinch points.

Update for Extension, July 2020:

Work on the projects set out above has been initiated and is currently underway. As planned, the majority of this work is still in the stage of modelling and analysis being performed and results being assessed internally, with wider dissemination of results expected later in the year through journal publication and engagement with NHS analytical networks. However, initial work on admitted hospital length of stay at treatment specialty level has already been completed and used to update the length of stay fitting tool built into open source, freely-available discrete event simulation software developed by the BNSSG M&A team (for specific use in modelling NHS care pathways).