Response to results error for the May 2017 diet of the MRCS Part B
18 Feb 2019
Statement in response to the issue affecting the results of the May 2017 diet of the MRCS Part B in which a number of candidates were given incorrect scores.
Whilst this did not affect the pass/fail outcome for the majority of this cohort, a small number were given the wrong overall result.
Though conducted by individual Colleges, the examinations are overseen by the Intercollegiate Basic Surgical Examinations (ICBSE), a function administered and managed by the Royal College of Surgeons (RCS) England on behalf of all four surgical colleges in the British Isles. The matter has now been investigated, and we set out below the actions we have taken to reduce the risk of recurrence.
We identified as the source of the issue a manual intervention, undertaken by the member of staff in the ICBSE office responsible for the processing of marks awarded to candidates. In this episode, this involved removal of incomplete data from the dataset that related to a candidate who withdrew midway through one of the examination circuits.
This action was taken after the quality assurance processes had been completed and was not conducted correctly. It led to a misalignment of candidate marks within the overall dataset that was not identified prior to the finalisation of results and dissemination of these to candidates.
As a result of the incident and subsequent investigation we drew the following conclusions:
- If a candidate withdraws part way through an examination circuit, all marks relating to that candidate must be clearly identified at source and removed from the standard setting process.
- There must be clear communication from the examination site to the results administrator at ICBSE, ICBSE Manager and those involved in the standard setting, if a candidate withdraws before or during the exam.
- The system should not rely on a single member of ICBSE staff to process the results. The process must be able to be performed by more than one individual and needs to be simplified.
- An electronic making system would be beneficial in removing sources of error in the process and would allow more straightforward data collation and analysis.
- Data analysis may be aided by the use of more appropriate software.
ICBSE has now undertaken the following actions:
- A procedure has been established – and has already been implemented – to ensure that, in the event of a candidate withdrawing midway through an examination circuit, their scores are removed from the spreadsheet prior to submission to ICBSE.
- This procedure re-affirms that the administering College will communicate to ICBSE the circumstances in which the candidate’s scores have been removed and that any amendments to the results is escalated accordingly within ICBSE, to ensure that this can be subjected to necessary checking and confirmation prior to the finalisation of results.
- The processing of results is now subject to a more stringent series of internal checks within ICBSE, based on key metrics to ensure that any issues with the collation of data are identified and addressed prior to the finalisation of results.
- Further development of the marking process will be initiated at the earliest opportunity. A tender specification for an electronic marking system has been agreed and tenders will be invited for arbitration by ICBSE.
Both the Joint Surgical Colleges Meeting (through which all four College Presidents oversee the working of intercollegiate functions) and ICBSE believe that the implementation of these actions – both immediate and longer-term – will ensure that the quality assurance of the MRCS examination and its associated processes is strengthened accordingly.
Once again, we would like to take the opportunity to apologise for the inconvenience the results error caused you and hope the outcome of the review goes some way to reassure you that appropriate measures are being taken to ensure this does not happen again.