An evaluation of the relationship between simulation-based training assessment tools and performance in real world settings

Text removed is struck through; text added is underlined. Unchanged paragraphs are summarised rather than repeated.

Objective for processing

Imperial College London’s Big Data and Analytical Unit (BDAU) requires an extract of HES data linked to consented surgeon simulation-based skill assessment data for use in a research study: ‘An evaluation of the relationship between simulation-based training assessment tools and performance in real world settings’. This study aims to establish what, if any, association there is between simulation-based skills assessment and clinical and patient outcomes. The purpose of this agreement is to allow honorary staff access to the data provided under previous iterations of this agreement, in order to allow this study, “An evaluation of the relationship between simulation-based training assessment tools and performance in real world settings”, to continue. Imperial College London is the sole data controller and data processor. All processing will take place in the BDAU secure environment, abiding by BDAU ISO 27001 certified standard operating procedures, and all data which is exported will be suppressed in line with NHS Digital guidance on data minimisation. For the purpose of this study, Imperial College London requires an extract of Hospital Episode Statistics (HES) Admitted Patient Care (APC) data from 2006/07 to 2015/16 linked to consented surgeon simulation-based skill assessment data for use in a research study: ‘An evaluation of the relationship between simulation-based training assessment tools and performance in real world settings’. This study aims to establish what, if any, association there is between simulation-based skills assessment and clinical and patient outcomes. The pseudonymised linked data required for the purpose of this study was provided under previous iterations of this agreement. This study is being undertaken by a PhD student within Imperial College London and has contributed to a PhD thesis in addition to other outputs intended to maximise the benefit of the work (publication and presentation of the work). Imperial College London is solely responsible for decisions made to determine how and why the data will be used, hence is the sole data controller and processor for this study.. No other organisations are involved in this project. All processing will take place in the Big Data and Analytical Unit Secure Environment (BDAU SE), an ISO 27001 certified research environment with strictly controlled access policy. All data which is exported from the BDAU SE will be aggregated with small numbers suppressed in line with NHS England guidance on data minimisation. The Big Data and Analytical Unit (BDAU) is a multidiscipline team within Imperial College London which collaborates with a large network of researchers across the college with the aim of ensuring the maximum use, impact and dissemination of research using healthcare data. This study was being undertaken by a PhD student within Imperial College London and has contributed to a PhD thesis in addition to other outputs intended to maximise the benefit of the work (publication and presentation of the work). The PhD student now has an honorary affiliation with Imperial College and is requesting under this agreement to continue to have access to the data in order to write up a peer-reviewed publication and be able to update the analysis if requested following the peer review process. Data will be accessed by: • Individuals holding an honorary contract under the supervision of a substantive employee of Imperial College London for the purposes described in this DSA only. Imperial College London must maintain records in a single location that cover the following details of each individual given access under an honorary contract: o Their substantive employer; o Their role in respect of the purpose for the processing specified in the DSA; o The start date and end date of the duration in which the Data will be accessed by the individual under an honorary contract; o The necessity for the Data to be accessed by the person(s) holding an honorary contract, instead of a substantive employee of an organisation named as controller or a processor in this DSA; o Confirmation that an appropriate contract is in place which follows the relevant guidance and is countersigned by the substantive employer of the honorary contract holder. [1 paragraph unchanged] The study will use uses a de-identified pseudonymised linked dataset prepared by NHS Digital England to compare the performance during simulation as collected by assessment score cards [131 words unchanged] in that they can then be used to encourage redesign of training. [2 paragraphs unchanged] The legal basis for processing the NHS Digital data under GDPR is Article 6 (1)(e) - "processing is necessary for the performance of a task in the public interest or in the exercise of official authority vested in the controller" and Article 9(2)(j) - "processing is necessary for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes". Under GDPR, the lawful basis on which processing of data from NHS England concerning this study is laid out by Articles 6(1)(e) and 9(2)(j) namely that processing is necessary for the performance of a task carried out in the exercise of official authority, such authority to make provision for research being vested in the University by virtue of the Royal Charter establishing the University dated 1907, and, that in respect of special category personal data, processing is necessary for research purposes carried out in accordance with Article 89(1) of the GDPR. Imperial College London are a public authority and the outcomes of this research is for the benefit of public interest because the primary aim of this research is to evaluate the relationship between simulation-based training assessment tools and performance in real world settings, result of which can guide the direction of surgical education and improve surgical care.

Processing activities

The simulation training scores for consented surgeons along with their consultant GMC ID in the appropriate format were transferred from the BDAU Big Data and Analytical Unit (BDAU) at Imperial College London to NHS Digital England for the purposes of linking. NHS Digital England linked this to the individual episodes in hospital episode statistics (HES) admitted patient care (APC) data for the years between 2006 and 2016 2006/07 to 2015/16 and returned a de-identified pseudonymised linked dataset with minimisation applied as per section 3a. to Big Data and Analytical Unit Secure Environment (BDAU SE). This ensures ensured that the dataset returned to the BDAU is completely pseudonymised as the BDAU will not receive any other identifiable or further linkable data. data due to their ISO 27001 certified policies. Data has been minimised to 24 fields of HES APC. The data minimisation applied will ensure ensures that patients are not re-identifiable even with a known consultant. Data access is stored in strictly controlled by the Imperial College London’s BDAU secure environment located in Imperial College data centre at Virtus SDC Limited through a robust user and can be accessed remotely by users once the user registration process is completed. Data is only provided once the appropriate dataset registration process is completed. process. No one other than BDAU staff can authorise access to the data. The BDAU SE is a secure environment research environment, providing a standard operating/access model, secure data storage and processing environment, and analysis software. It is ISO 27001 certified and is compliant with NHS Digital England Data Security and Protection Toolkit. All processing will also take place in the BDAU secure environment, abiding by BDAU ISO certified standard operating procedures and all data which is exported will be supressed in line with NHS Digital guidance on data minimisation. Virtus staff will not have access to the NHS Digital data in the BDAU Secure Environment. The BDAU SE can be accessed remotely (via a screen view) over a VPN connection by users using Multi-Factor Authentication (MFA) once the user registration process is completed. Data is only provided once the appropriate dataset registration process is completed. All data files and directories within the BDAU SE are encrypted using (ZFS) AES-256 encryption. Only the sysadmin (IT staff) has access to the filesystem encryption keys. To access the data in the BDAU SE, researchers will require approval from the study chief investigator (in their role as Asset Owner). Once approval is given, researchers are required to: 1) pass the Imperial Data Protection Awareness, including GDPR, and Information Security Awareness training 2) agree, with possible disciplinary action for noncompliance, to BDAU ISO 27001 certified standard operating procedures by signing the BDAU SE User and Dataset Registration forms. All training and forms are required to be renewed on an annual basis. Users who do not pass the annual renewal of training and forms will be automatically removed from access. Only Imperial College London personnel who have completed the BDAU user registration process are able to access the BDAU SE. The registration process includes completion of GDPR/data protection and information security awareness training and some BDAU SE forms. Access to data is restricted to one researcher, who was a PhD student throughout the previous iterations of this agreement and now has honorary affiliation with Imperial College to be able to publish the study findings in a peer-reviewed publication, and that researcher’s supervisors if necessary only for the purposes outlined in this Data Sharing Agreement. All individuals having access to this data are bound to the policies, procedures and equivalent controls of the BDAU SE and Imperial College London, either as substantive employees of Imperial College or having honorary contracts. They are all trained in Data Protection including GDPR and Information Security Awareness before they can access and analyse the data. NHS Digital securely transferred the resulting pseudonymised extract of HES data to Imperial College London. Imperial College London store the data on a server in the BDAU Secure Environment (SE). Data access is strictly controlled by the BDAU through a robust dataset registration process. No one other than BDAU staff can authorise access to the data. The raw data provided by NHS England will be analysed solely in the BDAU SE. Any further analysis done outside the BDAU SE (usually for visualisation purposes for output) will be done using data that has been aggregated with small numbers suppressed in line with the HES Analysis Guide. The data will be analysed to investigate the impact of simulation-based training on surgical performance of consented participant surgeons. This will involve statistical analysis using standard and innovative statistical programmes inside the BDAU SE. Results will graphed and compared at an aggregate level. Access to data is restricted to one researcher, a PhD student, and that researcher’s supervisors if necessary (usually not required), only for the purposes outlined in this Data Sharing Agreement. The student and supervisors are bound to the policies, procedures and equivalent controls of the BDAU SE and Imperial College London as substantive employees of the College. At no point will the data being provided by NHS England be used to identify any individual whether patient or consultant. The raw data provided by NHS Digital will be analysed solely in the BDAU SE. Any further analysis done outside the BDAU SE (usually for visualisation purposes for output) will be done using data that has been aggregated with small numbers suppressed in line with the HES Analysis Guide. The data will be analysed to investigate the impact of simulation-based training on surgical performance of consented participant surgeons. This will involve statistical analysis using standard and innovative statistical programmes inside the BDAU SE. Results will graphed and compared at an aggregate level. At no point will the data being provided by NHS Digital be used to identify any individual whether patient or consultant.

Expected output

[3 paragraphs unchanged] 2022 2024/25 – Relationship Between Performance Assessments in Surgery and Clinical Outcomes [2 paragraphs unchanged] 2022– 2024/25 – American College of Surgeons Accredited Education Institutes - annual meeting 2022– 2024/25 – Association of General Surgeons of Great Britain and Ireland Conference It is anticipated that these will be available online for any stakeholder to access. Where there are extreme outliers, this data will be suppressed and aggregate figures will be used i.e. deciles rather than individual performance scores. The same will apply to any outlier mortality, complication or length of stay derived from the HES data. The results will be used to evolve surgical performance assessment tools and contribute towards the development of an evidence-based surgical performance assessment strategy for both trainee and expert surgeons. This will also be translatable to performance assessment in the adoption of new techniques i.e. robotic surgery. With a robust evidence base, policy can be formed in order to ensure optimal surgical practice and appropriate governance of associated morbidity where this may be attributable to individual surgeon performance. This would not be implemented punitively but as a guide to improve practice and thereby improving patient outcomes. The intention would be that there will be a modification to already established methods of public reporting such as the National Bowel Cancer Audit Project where surgeon level performance data is available and guides patient choice. [3 paragraphs unchanged] The outputs of this study will be directly communicated to surgeons at workshops. Only aggregated results with small numbers supressed in line with the HES Analysis Guide will be used and surgeon identity will be protected. The outputs will [91 words unchanged] International which helped advise for local and global standards for clinical practice. All data which is used for outputs will be anonymous summary aggregate [16 words unchanged] the HES analysis guide. No raw data will be transferred outside the BDAU SE Big Data and neither Analytical Unit (BDAU) Secure Environment (SE) and the data nor the outputs will not be used for commercial purposes.

Expected measurable benefits

[1 paragraph unchanged] This is the largest study to date to attempt to quantify the [70 words unchanged] surgeons to ensure that their standard remains at an accepted national level. This represents both a national and international impact upon surgical performance and governance. This also extends across all surgical specialties and may affect the quality of care of millions of patients accessing surgical care. Optimisation of surgical efficiency will also confer a cost saving in the reduction of complications and shortening of length of stay. This will consequently reduce the costs associated with litigation. The results will be used to evolve surgical performance assessment tools and contribute towards the development of an evidence-based surgical performance assessment strategy for both trainee and expert surgeons. This will also be translatable to performance assessment in the adoption of new techniques i.e. robotic surgery. With a robust evidence base, policy could be formed in order to ensure optimal surgical practice and appropriate governance of associated morbidity where this may be attributable to individual surgeon performance. This would not be implemented punitively but as a guide to improve practice and thereby improving patient outcomes. The benefit will be to the NHS as an organisation, surgical training programmes and the patient. The intention would be that there will be a modification to already established methods of public reporting such as the National Bowel Cancer Audit Project where surgeon level performance data is available and guides patient choice. Further work will be required to more accurately derive specific performance markers and recording of complications across all grades. This will require further longitudinal studies of surgeons across a 5-10 year period. [1 paragraph unchanged]

Benefits reported

Imperial College London have yet to realise any yielded benefits. The data [63 words unchanged] and the focus of research resources directed in the field of COVID-19. The work is currently under a drafting stage and the data will be required in order to address any further analysis required by the reviewers. As the primary investigator working full time in the NHS, the COVID pandemic has significantly impacted availability due to more pressing clinical commitments in the interest of patient care. There have also been significant staff shortages which has also prevented access to additional time to generate high-impact publications. It is anticipated that this will no longer exist as a barrier.

Text removed is struck through; text added is underlined. Unchanged paragraphs are summarised rather than repeated.

Objective for processing

[2 paragraphs unchanged] This study is being undertaken by a PhD student within Imperial College London and will contribute has contributed to a PhD thesis in addition to other outputs intended to maximise the benefit of the work. work (publication and presentation of the work). [3 paragraphs unchanged] The number of participating surgeons will be is 20 which has been shown to be robust enough for these types [65 words unchanged] existing training tools or if redesign is needed for existing training tools. The data was released in May 2019 and has resulted in a richer pool of data and therefore the analysis requires required advanced statistics. statistics and further refinement may be required to publish the results of the study. The advent of COVID-19 has led to delays in the complete analysis [11 words unchanged] and the focus of research resources directed in the field of COVID-19. [1 paragraph unchanged]

Processing activities

[1 paragraph unchanged] Data is stored in the Imperial College London’s BDAU secure environment located in Imperial College data centre at Virtus SDC Limited. Limited and can be accessed remotely by users once the user registration process is completed. Data is only provided once the appropriate dataset registration process is completed. The BDAU secure environment is ISO 27001 certified and compliant with NHS Digital Data Security and Protection Toolkit. All processing will also take place in the BDAU secure environment, abiding [13 words unchanged] will be supressed in line with NHS Digital guidance on data minimisation. Virtus staff will not have access to the NHS Digital data in the BDAU Secure Environment. Only Imperial College London personnel who have completed the BDAU user registration process are able to access the BDAU SE. The registration process includes completion of GDPR/data protection and information security awareness training and some BDAU SE forms. [4 paragraphs unchanged]

Expected output

[3 paragraphs unchanged] 2021 – Impact of simulation training on performance of surgeons 2022 – Relationship Between Performance Assessments in Surgery and Clinical Outcomes [2 paragraphs unchanged] 2021– 2022– American College of Surgeons Accredited Education Institutes - annual meeting 2021– 2022– Association of General Surgeons of Great Britain and Ireland Conference [5 paragraphs unchanged]

Expected measurable benefits

Further dissemination of this research will allow healthcare providers to understand the relationship between simulation training and clinical outcomes. This increases the ability to ensure that healthcare providers can accurately deem what training is necessary to provide better care for their patients and provide the appropriate training to keep clinical skills at the highest standard. Through similar methods, surgical skills assessment will identify training requirements, leading to targeted training and improved surgical skills. This is a further benefit in that simulation skills assessment can, if deemed to be linked to real-world performance, assess surgical skills for trainees who have not yet build up enough routinely collected administrative data for analysis. The intention of this research is to identify whether the current tools of assessment in surgery have a relationship with actual observed clinical outcomes. The current method of evaluating this is through the use of formative assessments and HES data. There has been no robust evidence to demonstrate that these assessment tools translate into improved clinical outcomes nor is there evidence to demonstrate that the clinical outcomes recorded are consistent with better care. This is the largest study to date to attempt to quantify the relationship between the two. The results of this are expected to be valuable for guiding the direction of surgical education and improved surgical care by the way of tools and performance markers that have an evidence-based association with better patient care. This may result in the development of new tools that would be incorporated into surgical training programmes. This also could guide the development of ongoing assessment of fully qualified surgeons to ensure that their standard remains at an accepted national level. In summary, this research is expected to contribute to better surgical training, better patient outcomes and potential ongoing assessment of expert surgeons to ensure that a minimum standard is met.

Benefits reported

Imperial College London have yet to realise any yielded benefits. The data was made available by the Data Applications team in May 2019 The increased granularity of the data requires advanced statistics and direction from statistics experts. 2019. The sensitivity of data means that analysis can only occur at Imperial College and therefore there are also geographical constraints. Furthermore, the advent of a second COVID-19 wave has led to delays in the complete analysis of this data. data and submission of academic publications. This is in part due to clinical commitments and the focus of research resources directed in the field of COVID-19.

Text removed is struck through; text added is underlined. Unchanged paragraphs are summarised rather than repeated.

Objective for processing

[1 paragraph unchanged] Imperial College London is the sole data controller and data processor. All [14 words unchanged] certified standard operating procedures, and all data which is exported will be supressed suppressed in line with NHS Digital guidance on data minimisation. [5 paragraphs unchanged] May 2020 – Since the data request was extended to allow time to send in the ID's (which is due to a delay in the flow of data to receive these which is not related to the NHS Digital data flow), to do the data analysis for the aforementioned project and create the outputs and benefits mentioned below. The data was released in May 2019 and has resulted in a [34 words unchanged] and the focus of research resources directed in the field of COVID-19. [1 paragraph unchanged]

Processing activities

On approval of this data sharing agreement, the The simulation training scores for consented surgeons along with their consultant GMC ID [37 words unchanged] years between 2006 and 2016 and returned a de-identified dataset with minimisation applied. applied as per section 3a. This ensures that the dataset returned to the BDAU is completely pseudonymised as the BDAU will not receive any other identifiable or further linkable data. Data will be has been minimised to 24 fields of HES APC. The data minimisation applied will ensure that patients are not re-identifiable even with a known consultant. [1 paragraph unchanged] NHS Digital will securely transfer transferred the resulting pseudonymised extract of HES data to Imperial College London. Imperial College London will store the data on a server in the BDAU Secure Environment (SE). [14 words unchanged] No one other than BDAU staff can authorise access to the data. Access to data will be is restricted to one researcher, a PhD student, and that researcher’s supervisors if [29 words unchanged] BDAU SE and Imperial College London as substantive employees of the College. The raw data provided by NHS Digital will be analysed solely in [17 words unchanged] will be done using data that has been aggregated with small numbers supressed suppressed in line with the HES Analysis Guide. The data will be analysed [26 words unchanged] the BDAU SE. Results will graphed and compared at an aggregate level. [1 paragraph unchanged]

Expected output

[3 paragraphs unchanged] 2019 2021 – Impact of simulation training on performance of surgeons [2 paragraphs unchanged] 2019 – 2021– American College of Surgeons Accredited Education Institutes - annual meeting 2019 – 2021– Association of General Surgeons of Great Britain and Ireland Conference [5 paragraphs unchanged]

Benefits reported

Imperial College London have yet to realise any yielded benefits as no data has been disseminated under previous approved versions of this agreement - as no GMC ID's of the colorectal surgeons were submitted to permit the HES data extraction. Imperial College London have yet to realise any yielded benefits. The data was made available by the Data Applications team in May 2019 The increased granularity of the data requires advanced statistics and direction from statistics experts. The sensitivity of data means that analysis can only occur at Imperial College and therefore there are also geographical constraints. Furthermore, the advent of COVID-19 has led to delays in the complete analysis of this data. This is in part due to clinical commitments and the focus of research resources directed in the field of COVID-19.

Unchanged: Expected measurable benefits.

Text removed is struck through; text added is underlined. Unchanged paragraphs are summarised rather than repeated.

Objective for processing

[1 paragraph unchanged] Imperial College London is the sole data controller and data processor. All processing will take place in the BDAU secure environment, abiding by BDAU ISO 27001 certified standard operating procedures, and all data which is exported will be supressed in line with NHS Digital guidance on data minimisation. [5 paragraphs unchanged] August 2018 - No May 2020 – Since the data has been disseminated under previous approved versions of this agreement - as no GMC ID's of the colorectal surgeons were submitted to permit the HES data extraction. Imperial College London have therefore submitted an extension request was extended to allow for extension of this agreement to allow them time to send in the ID's (which is due to a delay [23 words unchanged] for the aforementioned project and create the outputs and benefits mentioned below. The data was released in May 2019 and has resulted in a richer pool of data and therefore the analysis requires advanced statistics. The advent of COVID-19 has led to delays in the complete analysis of this data. This is in part due to clinical commitments and the focus of research resources directed in the field of COVID-19. The legal basis for processing the NHS Digital data under GDPR is Article 6 (1)(e) - "processing is necessary for the performance of a task in the public interest or in the exercise of official authority vested in the controller" and Article 9(2)(j) - "processing is necessary for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes".

Processing activities

On approval of this data sharing agreement, the simulation training scores for consented surgeons along with their consultant GMC ID in the appropriate format will be were transferred from the BDAU to NHS Digital for the purposes of linking. NHS Digital will link linked this to the individual episodes in hospital episode statistics (HES) admitted patient care (APC) data for the years between 2006 and 2016 and return returned a de-identified dataset with minimisation applied as per section 3a. applied. This ensures that the dataset returned to the BDAU is completely pseudonymised as the BDAU will not receive any other identifiable or further linkable data. Data will be minimised to 24 fields of HES APC. The data minimisation method applied will ensure that patients are not re-identifiable even with a known consultant. Data is stored in the BDAU secure environment located in Imperial College data centre at Virtus SDC Limited. All processing will also take place in the BDAU secure environment, abiding by BDAU ISO certified standard operating procedures and all data which is exported will be supressed in line with NHS Digital guidance on data minimisation. [4 paragraphs unchanged]

Unchanged: Expected output, Expected measurable benefits, Benefits reported.