Safety INdEx of Prehospital On Scene Triage (SINEPOST): The derivation and validation of a risk prediction model to support ambulance clinical transport decisions on scene.

The University of Sheffield (UoS) are proposing to link an existing regional cohort of patients from the Yorkshire Ambulance Service (YAS) electronic Patient Care Records with Emergency Care data (using both the Emergency Care Data Set (ECDS) and the Hospital Episode Statistics Accident & Emergency (HES AE) datasets). NHS Digital hold the Emergency Care datasets and the data linkage will be completed by NHS Digital.

This request is for the purpose of medical research which aims to determine whether ambulance service clinical data can predict an avoidable attendance at the Emergency Department (ED) in adults using newly developed risk prediction models. These models could subsequently be used to develop a tool for paramedics on scene which can help them to determine the likelihood of treatment at an ED being of benefit to the patient.

In 2014 in Yorkshire, up to 16.9% of patients could have avoided being taken by ambulance to the ED. When the ED is busy, ambulances have to wait a long time to hand over the care of their patients. This delay stops ambulances being free to respond to the next emergency. The research conducted in the past using ambulance service episodes only used epidemiological descriptions as their output. Furthermore, they only collected data that is created in the ambulance call centre, whereas this project uses the clinical information on scene created by clinicians. Linking up individual patient care pathways from their YAS attendance data through to their respective ED admission, (the details of which are given in the ECDS and HES AE datasets) will allow the UoS to formulate the new risk prediction models and subsequent decision-making tools for paramedics on scene.

Within the ECDS and HES AE datasets there are variables which outline the experience of each patient. This includes what investigations and treatments they had, how they left the department and whether they reattended. These variables can be transformed into a single outcome variable which would show whether each individual patient had a clinically necessary attendance or whether their experience would have been better in a different clinical setting. By creating this single variable from the ECDS and HES AE datasets, the risk prediction model can then be developed. All the prehospital (ambulance service) data variants would be used to inform the model as to which of these variants are more predictive of a clinically necessary attendance or an avoidable one.

All subjects are adults defined as >18 years of age at the time of attendance. They all have contacted the ambulance service and received a face-to-face assessment with a paramedic. They all were transported to an ED in Yorkshire. The cohort for this study from the YAS will contain 328,763 attendances to the ED between July 1st 2019 and February 29th 2020. The dates proposed are the minimal amount needed to overcome seasonal biases that can be inherent in emergency care.

The cohort provided by YAS to NHS Digital will contain identifiable patient information (NHS number, date of birth, sex and postcode (unit level)) to allow for linkage to the Emergency Care data. A more limited number of fields are not possible to specify or collect as the nature of YAS data is such that for different cases, different personal identifiers will be missing.

The data provided by NHS Digital to the UoS will be record-level pseudonymised ECDS and HES AE data. Only variables that will provide the potential to predict the ED attendance outcome will be requested. There are no alternatives or less intrusive ways that would allow the purposes of the research to be fulfilled due to the required sample size making it impossible to consent every participant within the study.

This project will use data from the whole of Yorkshire. This region covers a wide range of different geographical settings which will overcome certain biases (such as urban vs rural and social deprivation). This is the smallest geographical footprint which will allow our model to account for these differences when applied on the national scale. Accessing emergency care data for Yorkshire from NHS Digital will cover ED attendances which were conveyed to: Airedale General Hospital, Barnsley Hospital, Bradford Royal Infirmary, Calderdale Royal Hospital, Dewsbury and district Hospital, Doncaster Royal Infirmary, Harrogate district Hospital, Huddersfield Royal Infirmary, Hull Royal Infirmary, Leeds General Infirmary, Northern General Hospital, Pinderfields Hospital, Rotherham Hospital, Scarborough Hospital, St. James’s University Hospital and York Hospital.

Consultations with the YAS, the UoS and the Research Design Service have all helped to minimise the amount of data requested to fulfil the required study purposes whilst satisfying the required sample size for the study.

The justification for the processing of this request is Article 6(1)(e) of the GDPR as this research project has been designed and funded with the premise of being in the public interest. The application also falls under Article 9 (2) (j), as scientific research. It is anticipated that the research output of the risk prediction model and subsequent tools for paramedics on scene will be of benefit to all patients that call the ambulance service; leading to more patients in the UK getting the right care; first time.

The UoS are the sole data controller who also process data. They will make all final decisions for the study design. The lead applicant is a student with the UoS to complete an National Institute of Health Research (NIHR)/Health Education England (HEE) Integrated Clinical Academic Doctoral Research Fellowship.

The YAS are the sponsor of the research study and will contribute the data to identify the cohort to NHS Digital. The lead applicant is a substantive employee of the YAS but is completing this project in the capacity of student with UoS. The YAS will not have access to NHS Digital data.

Access to the pseudonymised data will be limited to named individuals within the UoS who have successfully completed Information Governance training specified by the School of Health and Related Research (ScHARR) and agreed to abide by the School’s Information Governance Policy. All such individuals are substantive employees or students of the UoS.

Any breaches by substantive employees or students will result in disciplinary action with the UoS. This includes student expulsion if required. Any students undertaking a fellowship with the NIHR would need to report this immediately which would breach the fellowship contract and further disciplinary actions would be taken by the sponsor (YAS) and the NIHR.

This application is part of an NIHR/ HEE funded research project which commenced on the 1st April 2019 and is due to finish in March 2022. It is a stand-alone project which does not form part of a wider project or collaboration. The NIHR will not have access to NHS Digital data.

The data flows are summarised as follows:

1) The Business Intelligence team of the Yorkshire Ambulance Service (YAS-BI) will run a single data query of the electronic Patient Care Records (ePCRs) stored in their data warehouse. This will retrieve all information for every patient aged 18 years or older that was treated by a paramedic, then transported and booked in as a patient to an ED between July 2019 and February 2020. Unique IDs will be generated, assigning a random number to each individual ePCR. The retrieved ePCRs will be run through the national ‘opt out’ database with any qualifying records being removed. Records in this form will not be seen by the University of Sheffield (UoS).

2) The YAS-BI will proceed to split the whole dataset into two extracts. The first is the identifiable information and the second is the clinical information. Both extracts will contain the unique ID variable, but no identifiers will be contained within the clinical information extract. The YAS-BI will send the clinical information securely to the ScHARR at the UoS, and send the identifiable information (NHS number, date of birth, sex and postcode (unit level)) to NHS Digital. Once YAS-BI have sent the data extracts to their appropriate destinations, their data processing ends.

3) NHS Digital will use the submitted identifiable information to link the cohort to their respective Emergency Care Data Set (ECDS) and Hospital Episode Statistics (HES) Accident & Emergency (AE) records (an anticipated 328,763 records). The methodology used is transparent and well documented. Following data linkage, NHS Digital will remove all identifiable information and securely deliver the linked pseudonymised records containing special category (health) data to the School of Health and Related Research (ScHARR) where they will be stored in accordance with the UoS Information Governance policies. NHS Digital will also return the ‘unlinked’ unique ID only for any individuals where no ECDS or HES AE data could be retrieved.

4) The ScHARR will merge the linked and unlinked records from NHS Digital with the YAS clinical information already held using the previously generated unique ID. Any ambulance service data with no associated ED record would undergo bias screening before being destroyed. This means ensuring there are no significant differences between the patients who are going to be deleted and those who remain in the dataset for analysis. The remaining data will form one coherent pseudonymised record level dataset which maps the journey of each patient from ambulance service to ED. Data will be analysed in the ScHARR but stored on a remote virtual machine (VM) hosted on UoS infrastructure and maintained by the UoS Corporate Information and Computing Services (CiCS). CiCS administrators have policies in place which address network security (especially threats from outside the campus network) and software maintenance. Access to the VM (only possible from specified IP addresses) is granted only to a limited number of user accounts, all of which require authentication by username and password. Remote working is completed on UoS computers in users’ homes. These computers have encrypted hard drives, with encryption keys stored solely by UoS Information Technology staff. Other UoS storage locations listed provide increased security.

5) There will be no onward flows of record level data under this data sharing agreement. The UoS will be the only data processor from the point of receiving the linked ECDS and HES AE data from NHS Digital. The data processing will only be carried out by substantive employees and students of the UoS who have been appropriately trained in data protection and confidentiality. There will be no requirement or attempts to re-identify individuals. To mitigate the risk of any individual becoming identified once the data has been transferred to the UoS, each variable will be checked for any small parameters of data. These will have complete case deletion and therefore will not be re-identified during the data processing. Any publications that arise from this project will only include aggregated data. This further mitigates re-identification at the output stage.

In order to generate the risk prediction models which can be used by paramedics to help prevent avoidable ED attendance in future, the ScHARR will process the data as follows:

1) There will be an initial exploration to examine the quantity of candidate variables within the YAS clinical dataset and the parameters within each one. The candidate variables can be broadly divided into three categories. The first are the physical factors such as blood pressure, temperature etc. The second are the social factors for example patient location, mobility, or whether they have dementia or not. The purpose of including social factors is so the tool accounts for each patient’s social support requirements. The third are the interventional factors, which describe what the paramedics did with the patient on scene. Examples include what drugs they gave, or any interventions they needed. An exploration will also identify missing data and highlight any variables that may need cleaning or preparing in anticipation of model development.

2) The single outcome variable will be created using the variables derived from the ECDS and HES AE (collectively – ED) data, including investigations and treatments performed, how patients left the department, and whether they reattended. Once created, the ED variables will be removed from the dataset. This completes the data preparation stage.

3) Any missing data from the full linked dataset will be assessed for randomness using statistical methods. Any missing data within the ambulance service clinical variables will be assessed for its ability to bias the models and removed or retained as appropriate. Following on from this, the data will be separated into linked and unlinked cases. These will then be compared to assess for screening bias. Once this is complete, the unlinked data will be destroyed.

4) Predictive models will be developed and applied to the linked data independently. The reason for building different models is they can be more or less accurate in making predictions depending on the dataset. As this has not been done before, the plan is to build three types of models and then select the highest performing one. The three methods being used are: logistic regression, random forest modelling and a neural network. The methods vary in how they make predictions. For example, logistic regression looks at how each candidate variable is associated with the outcome. A random forest model builds multiple decision trees and then aggregates the outcome of each one. Neural networks are a combination of both of these techniques. Due to their differences in design, by building all three on the data it will give the best chance of creating an accurate prediction model that can be successful on future live data. Also, building three models with different methodology allows for alternative model selection at the implementation stage. For example, neural networks struggle to handle any missing data. Even if it was the most accurate model built, if there is naturally missing data in the dataset then it will not work in practice. Alternatively, logistic regression can handle missing data when making a prediction.

5) There will be no predetermined candidate variables selected. Instead, all clinical variables collected during the paramedic patient assessment and/or treatment will be included in the initial development. It is anticipated there will be thirty-nine variables across three categories described above (physical, social and interventional). Within each method being used (logistic regression, random forest modelling and neural networks), there will be a statistical way of eliminating variables that do not have a strong association with the outcome. This is to ensure the final model is parsimonious. A parsimonious model is one which is as simple (fewer variables) as possible without losing accuracy. Very complicated models with lots of input variables can often fail and are harder to implement in practice.

6) In order to produce an accurate model, it is important to be able to evaluate its performance. For each model, a contingency table will be devised so summary accuracy statistics can be calculated which explain how accurate each model is at making predictions. In addition, a calibration plot will show the agreement between observed and expected results. Calibration is assessing what the model predicted vs what actually happened. In addition to this, it is important that discrimination is accounted for and analysed. This is important as the models being built in this study are classifying patients. Discrimination is assessing how many times the model correctly identifies a person with the outcome and one without and is a measure of accuracy. By using all the statistics mentioned, the models can be directly compared. This helps with model selection, but also allows for other people to assess how accurate the model is.

NHS Digital reminds all organisations party to this agreement of the need to comply with the Data Sharing Framework Contract requirements, including those regarding the use (and purposes of that use) by “Personnel” (as defined within the Data Sharing Framework Contract ie: employees, agents and contractors of the Data Recipient who may have access to that data).

The ultimate goal of this research is to produce a toolkit that can support paramedics in deciding the likelihood of an avoidable Emergency Department (ED) attendance before transporting the patient. This could be built into an electronic Patient Care Record and used to support clinical assessment and decision making on scene.

In the process of developing this toolkit, it is anticipated that new knowledge will be created regarding:

- what clinical assessment variables contribute most to risk stratifying patients on scene

- how best to use risk prediction modelling in the prehospital setting

- how to influence/ change a paramedic’s decision on scene by presenting them with patient risk

- how to best manage patient expectations on scene

- what the potential feasibility is of more complicated models - such as machine learning - in assisting clinical decision making.

All such new knowledge would be disseminated throughout the relevant sectors of the NHS via several mechanisms. Firstly, the research would be presented to the Association of Ambulance Chief Executives. This would highlight the findings and scope feasibility to implement nationwide. Contacts made at this level would then be followed up at regional and local level. Contact with the National Leads for Urgent and Emergency Care would be maintained throughout the project, and feedback will be invited from them at various stages. Workshops will be put on for NHS staff and the work will be presented to the Urgent and Emergency Care Review Team (UECRT) and the National Ambulance Commissioners Network (NEWS). A Lay executive summary will be produced in digital format to disseminate widely on stakeholders’ web pages.

This research will also produce a thesis submitted as part of a PhD award at the UoS. There will be submissions to peer review journals and presentations at conferences. It is anticipated the main results of the study will be published in the British Medical Journal (BMJ) or PLOSone in an open access format. The results will be presented at an international medical conference such as Emergency Medical Services Expo or the International Conference of Emergency Medicine (ICEM). The publications and conferences are likely to take place in the summer / autumn of 2023. As well as publications and conference presentations on the final outcomes of the work, a systematic review of risk prediction modelling in the urgent and emergency care system will be completed and published in BMC Diagnostic Prognostic Research as part of this research project. Any publications, presentations and summaries that arise from this project will only include aggregated data in line with the HES Analysis guide.

This project has previously received a grant of £500 from the Research Design Service Yorkshire and Humber to support the development of patient public involvement (PPI). A small group has been set up combining existing PPI groups such as the Sheffield Emergency Care Forum (SECF) and other members of the public identified using the aforementioned grant. Members of the public from this group will be invited to co-produce a video that will be created for the wider public. This will introduce the concepts of not transporting all patients, and of using a tool to help clinicians with making the decision to transport or not. This video will be disseminated through social media such as Youtube, Twitter, LinkedIn and Facebook at the end of the project (March 2022). Across these platforms, the YAS has the potential to reach >40k followers, and the UoS a further 500k.

Members of the PPI group will also be given the opportunity to co-author conference abstracts and present findings with the lead researcher. The groups will be presented with updates on the research at regular intervals and invited to steer the project. As this is a technological study, the PPI group are on a project specific ‘Whatsapp’ group. This allows them to have a longitudinal relationship with the research and to ask questions freely outside of the regular meetings. This platform was chosen as it is securely encrypted end-to-end. By involving the public in these ways, it helps ensure that the end model will be accepted by the public and patients. There will be no record level data shared via this messaging group.

The research outputs are all planned to be actualised before project completion, or within 12 months of project completion (March 2022).

Beyond the scope of the PhD project and the development of the risk prediction model, the ScHARR would look to secure funding grants for the continuation of this research. Further research would involve collaboration with NHS Digital to create a novel data product combining an emerging data set (the Ambulance Data Set (ADS)) and the existing Emergency Care Data Set. This would allow for routine model updating and national external validation. Specific follow on work is likely to include refining the toolkit, then implementing and testing it under real world conditions such as a randomised controlled trial. In order to do this, the developed algorithm would be registered as a medical device. The UoS would work with Medipex to secure the intellectual property for the algorithm. The YAS would be the responsible party for any intellectual property unless they are deemed to be insufficiently monitoring dissemination and then the National Institute for Health Research (NIHR) has the right to take control. The NIHR are clear on the data and knowledge ownership, management, access rights, open access and rights usage.

In terms of potential commercial exploitation now or in the future, this would be a decision after the project and would be informed by Medipex. This process ensures that the Intellectual Property (IP) stays within the NHS. It also means the YAS would produce the licensing with the support of Medipex.

Previous studies have shown that up to 16% of all ambulance transports could be avoided. Linking up ambulance service data with Emergency Care Data Set and Hospital Episode Statistics Accident & Emergency data will allow the University of Sheffield (UoS) to map the journeys of individual patients from their on-scene intervention to the Emergency Department (ED). The processing of this data will subsequently allow an algorithm to be developed which could be built into electronic Patient Care Records and support paramedics in making discriminate decisions on scene. Based on 2019/20 data of ambulance conveyances to EDs in England, the new algorithm could have helped to prevent as many as 865,202 avoidable ED attendances over this year. This would have avoided a lot of unnecessary stress for patients, and could have provided much better experiences in a more appropriate clinical setting.

Using the NHS reference tariffs, it is estimated that preventing this number of avoidable ED attendances could allow a more effective resource utilisation of almost £86m year on year – money that could be redistributed to other sources of care. Reducing the number of unavoidable ED attendances could also save time for ED staff, allowing true emergency cases to be focused upon. This in turn would have a positive impact for the ambulance service. When the ED is busy, ambulances wait a long time to hand over the care of their patients. This delay stops ambulances being free to respond to the next emergency. Freeing up more ambulances will reduce ambulance response times, with the algorithm then allowing more patients to receive the most appropriate care first time.

Even if the algorithm is unsuccessful, the information gained should inform ambulance services what the important clinical variables are that can be used in a deconstructed manner outside of an algorithm to help prevent avoidable ED attendances. Dissemination of the data through academic peer review journals, presentations and conferences will help to ensure the integrity of the research output and improve the impact of the project. The intention is that the number of transportations to an ED that do not receive a clinical benefit will be significantly reduced within the next 5 years, because of this knowledge dissemination and algorithm development. As an estimate, if the tool was enrolled nationally and resulted in a 1% decrease in overall conveyance, that would be 4328 fewer transportations to ED every month (based on January 2020 NHS England data and not accounting for inflation in demand which is currently 3-5% per annum).

Once the algorithm is developed, the UoS intend to work with key collaborators at a national level to implement it in an observational format at first. If successful, the algorithm would then be trialled in ambulance sites around the UK. The Yorkshire Ambulance Service (YAS), sponsoring the project, should achieve benefits in the form of reduced response times, and valuable new knowledge for their paramedics on the best course of action for their patients. The YAS would also seek to protect the intellectual property of this project as there is a potential for the organisation to develop a product or service from the model. This could be licensed to other organisations. The potential monetary gains for the organisation could be used to deliver patient care and improve the service provision of YAS patients. The UoS as the data controller and the National Institute for Health Research as project funder will have supported the completion of a PhD. The main benefit of this work however will be to patients in England who should receive the most appropriate care for their situation, first time, in less time.

Yielded Benefits is not a requirement for new applications.