SOJ Neurology

Training surgeons is a dynamic process which is evolving towards more objective measures of assessing psychomotor technical skills. The current method for assessing neurosurgical trainees' technical skills is the in-training evaluation report (ITER). Despite the wide use of ITER in neurosurgical training programs, multiple authors have criticized its assessment validity. A number of other tools, designed and validated in the last two decades to assess the technical and cognitive skills of surgical procedures, will be discussed. Advances in computer-based technology have created significant opportunities for implementing new neurosurgical training paradigms focusing on improving neurosurgical bimanual psychomotor skill acquisition, better documenting surgical skills assessment and potentially enhancing procedural outcomes. A number of simulators have been developed for training residents involving a variety of neurosurgical purposes, ranging from simple procedures to full preoperative planning simulation. This communication will focus on our recent results utilizing the NeuroTouch virtual reality simulator platform which provides haptic feedback to investigate a series of questions that relate to the concept of surgical expertise.

Providing excellent care to patients is the ultimate goal of all health care specialties; for surgeons, proficiency involving psychomotor skills is an essential component of this care. The training of surgeons is a dynamic process which is evolving towards more objective measures of assessing technical skills [1]. The assessment of the psychomotor skills of a resident by a series of consultant surgeons is subjective and by its nature incomplete. Virtual reality simulator technology together with novel metrics have the potential to modify and improve resident assessment and training in safe simulated environments where one can repeat the simulated procedure(s) with an appropriate demonstrator and a metric feedback. In 2004, Baker et al reported that 185,000 patients of 2.5 million admissions per year in the Canadian hospitals were exposed to a surgical adverse event and 36% of those events considered preventable [2]. In 2000 The Institute of Medicine (IOM) published a report called "To Err is Human" concluding that many surgical errors can be attributed to poor surgical training [3]. More recently duty-hour restriction protocols have been implemented, reducing working hours and "on-calls" for surgical residents and thereby decreasing resident operating room training opportunities to improve their surgical skills [4]. Reducing surgical errors and improving the quality of surgical training relates directly to psychomotor skill competence which is the ability to perform certain surgical skills successfully[5]. "Surgical competency" combines anatomical and procedural knowledge with technical and social skills to solve familiar and novel situations so as to provide adequate patient care [6]. This definition has its focus on "adequate" rather than "excellent" patient care. Most authors agree that new methods of training are required [7], yet a fundamental question remains as to whether future pedagogical programs should train surgeons to an "expert" or only a "competent" level of performance?

To help address this question we describe the current surgical training model, assess the formal method for surgical trainee assessment and outline some of our recent studies utilizing the virtual reality simulator NeuroTouch platform to evaluate resident and neurosurgeon psychomotor performance.

Present day surgical technical skills training is very similar to a traditional apprenticeship model outlined by Lave and Collins [8,9]. The trainees are observed, taught and assessed by their instructors who are usually practicing physicians. Dr. Halsted at Johns Hopkins Hospital implemented a defined apprenticeship model in surgery in the late nineteenth century [10,11]. This North American model has become the dominant method of surgical training worldwide. However this "apprenticeship model" that has served well for more than a century, for a number of reasons, is in need of re-evaluation.

The standard surgical residency programs combine textbooks, cadaver dissection, and intraoperative instruction. These formats are useful but may not optimize training for the bimanual psychomotor skills acquisition necessary for surgical expertise. First, textbook-based anatomy is two-dimensional, limited to fixed views, and often difficult to extrapolate to real intra-operative anatomy [12]. Second, technical skills acquired on cadaveric tissue lack realism due to the absence of bleeding, presence of formalin fixed normal and/ or diseased tissues which do not provide the appropriate surgical "feel" and limitless surgical exposure possibilities which may not be appropriate for human procedures. Furthermore, not all students have access to these "industry sponsored surgical master classes" [13]. As a result, cadaver dissection typically accounts for only a small fraction of a surgical resident's education. Finally, learning and teaching in the operating room can be stressful, time constrained and due to patient safety concern, only limited surgical anatomy can be exposed and studied[12]. Hence an opportunity to learn three-dimensional anatomy, relevant surgical exposure, the "feel of real tissues" and a variety of techniques outside of high-stress operating room environments in calm and risk-free virtual reality simulated settings, must be balanced against the "apprenticeship models' advantages and disadvantages"[14]. Surgical teaching and training also needs to be further developed so that students can have the opportunity to learn at their own pace and virtual reality surgical simulation may be one avenue to facilitate this experience [14].

The current method for assessing neurosurgical trainees' technical skills is the in-training evaluation report (ITER). This measures global competency based upon three broad categories: knowledge, skills and attitude. Surgical educators provide ITERs for their trainees at the end of each rotation (i.e., two to four months) and at the end of the year for promotion purposes. Program directors summarize these ITERs to provide feedback for trainees. ITERs are considered an appropriate and valuable assessment tool to monitor certain aspects that may not be evident in formal examinations. These include the trainee's communication skills, history taking ability, clinical examinations skills, and professional attitudes. Other benefits of ITER show that it is cost-effective, less intrusive, and does not interfere with day-to-day practice [15].

Despite the wide use of ITER in neurosurgical training programs, multiple authors have criticized its assessment validity [16,17]. They argue that the surgical educators who perform the assessments receive little or no training on how to properly use ITERs and that the retrospective nature of ITER for evaluation is therefore based on the rater's memory rather than on real time documentation. Another concern is that ITER completion may be influenced by what is referred to as the "halo effect" in which the rater's judgment of trainee's performance in one area is influenced by his or her performance in another [16,17]. In Canadian neurosurgical programs, ITERs assess seven major competencies 1. Medical expertise, 2. Communication skills, 3. Collaboration skills, 4. Health advocacy, 5. Management skills, 6. Scholar, and 7. Professionalism. "Surgical skills competency assessment" is but one of many subcategories under the heading "medical expertise competency". The final assessment does not outline specific surgical deficiencies, which reduces the value of the feedback [17]. Due to these deficiencies, perceived or otherwise, in the ITERS as an assessment tool, multiple other surgical technical skills assessment tools have been developed and introduced in the last two decades. In the next section we will discuss their advantages and limitations.

As defined by Ericsson and Charness expert performance is a laboratory technique that is used to assess expertise in a simulated environment which should mimic the real task as closely as possible [18]. This concept outlines that surgical skills should be measured in a simulated environment under controlled conditions but this simulated environment should mimic reality. Performing technical skills in the operating room under specific conditions is a stressful and complex experience, resulting in variable responses based on the operative situation that is being addressed. A neurosurgeon assessing a resident operating has no ability to assess the amount of force that a resident is applying while using instruments except by an assessment of tissue injury which occurs during and/or following the procedure. The continuous dynamic process of introduction of new surgical tools into the operating room makes it difficult to assess resident technical skills since both residents and their teachers are dealing with these novel improvements simultaneously [7]. These issues suggests that surgical educators need to develop and use more objective assessment tools that are reliable, reproducible and valid to assess trainee surgical skill acquisition and progress [7].

A number of tools have been designed and validated in the last two decades to assess the technical and cognitive skills of surgical procedures [19-24]. In 1995 the Objective Structured Clinical Examination (OSCE) was introduced, followed by the Objective Structured Assessment of Technical Skills (OSATS), a global rating scale, McGill Inanimate System for Laparoscopic Skills, and the Imperial College Surgical Assessment Device (ICSAD) [21,25-27].

All these assessment tools are prone to rater subjectivity as a consequence of broad criteria within each scale, which are not clearly defined and have poor test-retest reliability. Gélinas- Phaneuf and Del Maestro (personal communication) failed to provide validity of a new scale called global assessment of intraoperative neurosurgical skills (GOALS). This may have been due to variation in the type of neurosurgical procedures and the nature of the skills performed.

Reliability and validity are two important facets of a training and assessment tool. Reliability relates to the extent of reproducibility and consistency of an assessment tool when evaluating the same individual on different occasions in the same task and with no intervening learning. Validity is whether or not the tool measures what it intends to measure [28]. Validity testing is generally more complex compared to reliability as it includes face, content, construct, concurrent and predictive validity. Face validity is defined as the extent to which assessment conditions resemble a real life situation. Content validity is defined as the extent to which certain attributes being measured are measured by an assessment tool. Both face and content validity are usually measured by expert opinions using a questionnaire [28]. Using face and content validity in the context of developing a new technology for training neurosurgical skills implies that the technology is realistic and measures skills required for a specific activity or skill being assessed [28]. Construct validity should establish correlation with operative experience and be able to discriminate between novice and expert performance. Concurrent validity is established when the scores from a new measurement procedure are directly related to the scores from a well-established measurement procedure for the same construct; that is there is consistent relationship between the scores from the two measurement procedures [28].

The current model of neurosurgical psychomotor training is based primarily on an apprenticeship. Opportunities for deliberate practice outside of the operating room environment are uncommon. A series of studies have demonstrated that simulation has an important role to play in the acquisition of surgical skills [29-32]. Simulation provides an opportunity for deliberate practice. The student achieves the desired learning outcome in a safe simulated environment and can repeat the simulated procedure(s) and receive appropriate demonstrator and metric feedback.

Assessment of resident psychomotor skills by consultant surgeons is subjective and commonly retrospective [16,17]. Validated objective criteria of surgical competence are not available to the neurosurgical curriculum since validated criteria have not yet been developed.

Virtual reality simulators with haptic feedback can be used to explore and validate a series of issues relating directly to the question of neurosurgical expertise. This haptic or kinesthetic communication (feedback) recreates the sense of touch by applying forces, vibrations, or motions to the surgical user. This mechanical stimulation can be used to create a virtual brain containing tumor tissue with realistic appropriate pulsation in a computer simulation. Tactile sensors can be incorporated into these simulators with haptic systems which measure forces exerted by the user on the interface. These advances in computerbased technology have created a significant opportunity for implementing new neurosurgical training paradigms focusing on improving neurosurgical skill acquisition, enhancing procedural outcomes, and better documenting surgical skills assessment. In the past decade many virtual reality simulators have been developed for a variety of distinct neurosurgical purposes, ranging from training residents in simple procedures to full preoperative planning simulation [33]. The topic of simulation in neurosurgery has recently been extensively reviewed by Rehder et al [34].

Two virtual reality systems containing haptic feedback technology have been designed to specifically address neurosurgical technical skills improvement [24,35-42]. The ImmersiveTouch system, developed at the University of Illinois (Chicago) integrates haptic instrument feedback with a head and hand tracking system to provides a three dimensional visualization of a virtual patient's anatomy [35]. This system has been validated for lateral ventriculostomy [40]. Junior neurosurgery resident success rate improved significantly after performing one ImmersiveTouch virtual procedure[41]. Success rates in these studies were based on the end result rather than on the technical aspects (metrics) of the procedure.

The Neurosurgical Simulation Research and Training Centre at the Montreal Neurological Institute and Hospital along with other centers working collaboratively with the National Research Council (Canada) developed a computer-based virtual reality simulation platform called NeuroTouch (Figure 1A and B)[24,36]. This platform was designed to provide neurosurgery trainees the opportunity for deliberate practice and to assess their level of psychomotor competency [24]. Multiple metrics including those involved in bimanual psychomotor performance that reflect the operator performance for a surgical task can be measured by the NeuroTouch platform [24,36- 39].

We will now focus on our studies related to the development and validation of tier 1, tier 2 and advanced tier 2 metrics utilizing the NeuroTouch platform. Tier I, tier 2, and advanced tier 2 metrics were designed to assess bimanual psychomotor performance during the resection of a variety of simulated tumors (Figure 1C). Tier 1 metrics were developed which included blood loss, tumor percentage resected, and simulated 'normal' brain volume removed while tier 2 metrics included total tip path length, maximum and sum of forces used by instruments[38]. Advanced tier 2 metrics include efficiency index, coordination index, ultrasonic aspirator path length index and ultrasonic aspirator bimanual forces ratio and focused on the ability of the operator to coordinate the use of two hands during the resection of different simulated tumors [36,39]. After developing the tier 1 and tier 2 metrics our studies focused on using a simulated aspirator in the dominant hand (Figure 1D). The resection of 9 different simulated brain tumours (18 tumors in all) with different visual and stiffness characteristics were utilized to assess psychomotor performance [37]. These studies found no statistical difference between resident and neurosurgeon psychomotor performance [37]. However for the first time we were able to develop proficiency performance benchmarks for these simulated tumor resection scenarios [37]. In these studies the tier 1 and tier 2 metrics employed focused only on the motor skills of the dominant hand with simple scenarios and appeared to be of limited value to assess complex two hands interaction involving more realistic scenarios [36]. With the development of advanced tier 2 metrics by our group we then carried out a series of studies utilizing these metrics to evaluate the influence

of increasing tumor complexity by altering color, stiffness and stress levels on bimanual psychomotor performance (Figure 1C and E)[36,39,43]. Increasing tumor complexity impaired resident performance significantly more than that of neurosurgeons. The color of simulated tumors significantly altered resident performance. Resecting black vs glioma-colored tumors resulted in significantly higher blood loss and lower tumor percentage removed. Altering tactile cues from hard to soft also significantly decreased resident tumor resection. Regardless of tumor complexity, significant differences were found between neurosurgeons, senior and junior residents in efficiency index and ultrasonic aspirator path length index. Ultrasonic aspirator bimanual force ratio outlined significant differences between senior and junior residents, whereas coordination index demonstrated significant differences between junior residents and neurosurgeons. The NeuroTouch platform utilizing the simulated scenarios used and employing the metrics outlined differentiates novice from expert neurosurgical performance and demonstrated NeuroTouch face, content and construct validity [39].

Acute stress may result from severe bleeding during neurosurgical operations affecting operator bimanual psychomotor performance leading to surgical error and adverse patient outcome. The Neuro-Touch virtual reality simulator allows the testing of the influence of acute stress on psychomotor performance in risk free environments. The purpose of this study which involved medical students, junior and senior residents along with practicing neurosurgeons was to assess the impact of a simulated stressful virtual reality tumor resection scenario [43]. The stress scenario involved uncontrollable 'intraoperative' bleeding resulting in simulated patient cardiac arrest. The junior resident and medical student groups had decreased tumor resection and brain volume removal during stress. Since the baseline tier 1 metrics of blood loss and of total percentage tumor resected differentiated the resident and the neurosurgeon groups this provided construct validity for these metrics utilizing the NeuroTouch simulator. Psychomotor performance in advanced tier 2 metrics was altered during the stress scenario in all participant groups. The development of validated metrics in our previous study for advanced tier 2 metrics allowed the comparison of results and further validation of the NeuroTouch platform using data from this study. In the neurosurgeon group, sucker total tip path length increased while senior residents increased sucker forces. Although all participant performance decreased during the stress scenario neurosurgeons outperformed the other groups. To mitigate the possibility of injury to 'normal' tissues during the bleeding, the neurosurgery group modulated its performance by significantly increasing sucker total tip path length to improve visualization and by not increasing instrument forces applied while senior residents increased sucker forces. Performance in all advanced tier 2 metrics returned to pre-stress levels in post stress scenario tumor resections. These results confirmed that advanced tier 2 metrics were particularly valuable in the assessment of operative psychomotor performance under simulated stressful conditions associated with the uncontrollable bleeding scenario utilized [43]. Guided by these validated metrics it seems reasonable to propose the development of a series of proficiency based benchmarks helping to develop specific training curricula and self-assessment programs to maximize resident performance during the resection of cerebral tumors using the NeuroTouch simulator.

Present ongoing studies are focused on assessing the psychomotor skills of medical students applying to neurosurgery in an attempt to answer a number of issues including the differences in bimanual psychomotor performance among applicants. The tracking of residents serially during training and after graduation is critical to understanding psychomotor skills acquisition during residency and modulation of these abilities during neurosurgical practice. A number of issues need to be resolved, including which simulators to adopt, which scenarios to employ, the best validated metrics that emphasizing patient safety to assess and which proficiency-based benchmarks to utilize. To address this issue the information from these studies is being incorporated into a global multicenter longitudinal study which will follow neurosurgical resident applicants through their resident training and during their practice as independent neurosurgeons. The modulation of the force utilized during the resection of tumors appears to be critical in the development of 'expert' bimanual psychomotor performance during these procedures. The development of tier 3 metrics involving such concepts as the force pyramid, the force histogram and the ability to assess tremor may be particularly useful in understanding the bimanual utilization of force during surgery [38].

Neurosurgical simulators utilizing haptic technology provide new avenues to explore. They can be used to investigate a series of questions that relate to the concept of surgical expertise in risk-free learning environments[38]. These include how do "expert" surgeons actually perform surgical procedures? Insights into the visual, tactile, stress and/or other issues along with what forces are employed by expert neurosurgeons during the technical components of their operations can be assessed. With the proper simulators, metrics, proficiency performance benchmarks and curriculum, can virtual reality simulation impart specific bimanual psychomotor skills to residents helping to shift the goal of surgical training programs from teaching to competence to teaching to expert level? Since it is essential to have simulation scenarios that closely resemble operative realism significant further development is necessary to provide scenarios with the appropriate color (visual), stiffness (haptic) and structural realism. For neurosurgical scenarios progress is being made on these areas but the development of complex scenarios involving tumors adjacent to important motor, sensory and speech structures, epilepsy scenarios and complex scenarios involving lobectomy need to be available. The development of simulated patties to control simulated bleeding along with other adjuvants used typically during surgical procedures need further improvement. The ultimate goal must be patient specific simulation in which residents and surgeons can rehearse on the patient's tumor when it has been simulated on a virtual reality simulator and before the operative procedure.

Like any training and/or assessment tool to be implemented, rigorous efforts are necessary to provide evidence for validation and effectiveness over traditional educational methods in use.

This communication has focused on a number of studies that provide face, content and construct validity for the NeuroTouch simulation platform utilizing neurosurgical scenarios. Some of the metrics discussed are being assessed for their usefulness in other simulated neurosurgical operations and by other surgical specialties which may aid the universality of their application [44-47]. The utility of virtual reality simulators like NeuroTouch will be limited unless concurrent validity can be demonstrated which would outline that these simulators enhance resident operating room performance[48,49].

To maximize surgical expertise training, placental preparations could play important roles as hybrid models (between virtual reality simulators and the patient) by allowing participants who have obtained defined proficiency-based benchmarks on virtual reality simulators to be further tested on an ex vivo biological model before being assessed in operating room situations [50,51].

From the patients' perception, having residents pre-train with simulation increases the likelihood of success by establishing a minimum level of competency prior to clinical interaction. Graber et al. showed that patients are more willing to allow trainees to perform procedures on them after they have undergone simulation training [52]. Neurosurgical virtual reality simulators offers a safe route to carry out the sustained, deliberate, and goal-directed practice that neurosurgical expertise necessitates. Neurosurgical virtual reality simulators help learners assimilate technical skills and thereby this knowledge will ensure the development of surgical technical skills proficiency. If these promising results are validated by other researchers this will be useful in the development of further proficiency performance benchmarks for resident training, assessment and perhaps screening medical students applying for neurosurgical training programs[37,42,46].

Neurosurgical virtual reality simulation is a rapidly growing field with promising potential including the screening of medical students applying for neurosurgery programs, the training of neurosurgical residents with objective feedback metrics and the development of patient specific scenarios for rehearsal purposes.

This work was supported by the Di Giovanni Foundation, the Montreal English School Board, the B-Strong Foundation, the Colannini Foundation and the Montreal Neurological Institute and Hospital. Dr. R. F. Del Maestro is the William Feindel Emeritus Professor in Neuro-Oncology at McGill University. Dr. H. Azarnoush held the Postdoctoral Neuro-Oncology Fellowship from the Montreal Neurological Institute and Hospital.

The authors would like to thank all the previous members of our team including, Jawar Fares, Marta Baggiani, and Ibrahim Marwa and the members of the Simulation, Life Sciences Division, National Research Council of Canada at Boucherville.

1. Gélinas-Phaneuf N, Choudhury N, Al-Habib AR, Cabral A, Nadeau E, Mora V, et al. Assessing performance in brain tumor resection using a novel virtual reality simulator. Int J Comput Assist Radiol Surg. 2014;9(1):1-9. doi: 10.1007/s11548-013-0905-8.
2. Baker GR, Norton PG, Flintoft V, Blais R, Brown A, Cox J, et al. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ. 2004;170(11):1678-1686.
3. Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Vol 627: National Academies Press; 2000.
4. Ulmer C, Wolman DM, Johns MM. Resident duty hours: enhancing sleep, supervision, and safety. National Academies Press; 2009.
5. Brightwell A, Grant J. Competency-based training: who benefits? Postgrad Med J. 2013;89(1048):107-10. doi: 10.1136/postgradmedj-2012-130881
6. Bhatti NI, Cummings CW. Competency in surgical residency training: defining and raising the bar. Acad Med. 2007;82(6):569-573.
7. Cox M, Irby DM, Reznick RK, MacRae H. Teaching surgical skills—changes in the wind. N Engl J  Med. 2006;355(25):2664-2669.
8. Lave J. The culture of acquisition and the practice of understanding. Situated cognition: Social, semiotic, and  psychological perspectives. 1997:63-82.
9. Collins. "Cognitive Apprenticeship", in R.K. Sawyer (ed.). New York: Cambridge University Press; 2006.
10.   Cohen-Gadol AA, Piepgras DG, Krishnamurthy S, Fessler RD. Resident duty hours reform: results of a national survey of the program directors and residents in neurosurgery training programs. Neurosurgery. 2005;56(2):398-403.
11. Tsuda S, Scott D, Doyle J, Jones DB. Surgical skills training and simulation. Curr Probl Surg. 2009;46(4):271-370. doi: 10.1067/j.cpsurg.2008.12.003.
12. Henn JS, Lemole Jr GM, Ferreira MA, Gonzalez LF, Schornak M, Preul MC,  et al. Interactive stereoscopic virtual reality: a new tool for neurosurgical education: technical note. J Neurosurg. 2002;96(1):144-149.
13. Haluck RS, Krummel TM. Computers and virtual reality for surgical education in the 21st century. Arch Surg.      2000;135(7):786-792.
14. Hoffman H, Vu D. Virtual reality: teaching tool of the twenty-first century? Acad Med. 1997;72(12):1076-1081.
15. Wanzel KR, Ward M, Reznick RK. Teaching the surgical craft: from selection to certification. Curr Probl Surg. 2002;39(6):573-659.   
16. Gray JD. Global rating scales in residency education. Acad Med. 1996;71(1 Suppl):S55-63.
17. Turnbull J, Gray J, MacFadyen J. Improving In‐Training Evaluation Programs. J  Gen Intern Med. 1998;13(5):317-323. doi:  10.1046/j.1525-1497.1998.00097.x
18. Ericsson KA, Charness N. Expert performance: Its structure and acquisition. Am Psychol. 1994;49(8):725.
19. Kopta JA. The development of motor skills in orthopaedic education. Clin Orthop Relat Res. 1971;75:80-85.
20. Streiner D. Global rating scales. Assessing clinical competence. New York: Springer. 1985:119-141.
21. Martin J, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273-278.
22. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative "bench station" examination. Am J Surg. 1997;173(3):226-230.
23. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: a method for assessing clinical skills. Ann  Intern Med.2003;138(6):476-481.
24. Delorme S, Laroche D, DiRaddo R, Del Maestro RF. NeuroTouch: a physics-based virtual simulator for cranial microneurosurgery training. Neurosurgery. 2012;71(1):32-42.  doi: 10.1227/NEU.0b013e318249c744.
25. Jansen J, Tan L, van der Vleuten CP, van Luijk SJ, Rethans JJ, Grol RP. Assessment of competence in technical clinical skills of general practitioners. Med Educ. 1995;29(3):247-253.
26. Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998;73(9):993-997.
27. Fried GM, Feldman LS, Vassiliou MC, Fraser SA, Stanbridge D, Ghitulescu G, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg. 2004;240(3):518-525.
28. Moorthy K, Munz Y, Sarker SK, Darzi A. Objective assessment of technical skills in surgery. BMJ. 2003;327(7422):1032-1037.
29. Gurusamy K, Aggarwal R, Palanivelu L, Davidson BR. Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. Br J Surg. 2008;95(9):1088-1097. doi: 10.1002/bjs.6344.
30. Park J, MacRae H, Musselman LJ, Rossos P, Hamstra SJ, Wolman S,  et al. Randomized controlled trial of virtual reality simulator training: transfer to live patients. Am J of Surg. 2007;194(2):205-211.
31. Korndorffer Jr JR, Dunne JB, Sierra R, Stefanidis D, Touchard CL, Scott DJ. Simulator training for laparoscopic suturing using performance goals translates to the operating room. J Am Coll Surg. 2005;201(1):23-29.
32. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg.  2004;91(2):146-150.
33. Alaraj A, Lemole MG, Finkle JH, Yudkowsky R, Wallace A, Luciano C, et al. Virtual reality training in neurosurgery: review of current status and future applications. Surg Neurol Int. 2011; 2:52. doi: 10.4103/2152-7806.80117.
34. Rehder R, Abd-El-Barr M, Hooten K, Weinstock P, Madsen JR, Cohen AR. The role of simulation in neurosurgery. Childs Nerv Syst. 2015; dx.doi.org/10.1007/s00381-015-2923-z
35. Chan S, Conti F, Salisbury K, Blevins NH. Virtual reality simulation in neurosurgery: technologies and evolution. Neurosurgery. 2013;72:154-164 . doi: 10.1227/NEU.0b013e3182750d26.
36. Alotaibi FE, AlZhrani GA, Sabbagh AJ, Azarnoush H, Winkler-Schwartz A, Del Maestro RF. Neurosurgical Assessment of Metrics Including Judgment and Dexterity Using the Virtual Reality Simulator NeuroTouch (NAJD Metrics). Surg Innov. 2015. pii: 1553350615579729.
37. AlZhrani G, Alotaibi F, Azarnoush H, Winkler-Schwartz A, Sabbagh A, Bajunaid K . Proficiency Performance Benchmarks for Removal of Simulated Brain Tumors Using a Virtual Reality Simulator NeuroTouch. J Surg Educ. 2015;72(4):685-696. doi: 10.1016/j.jsurg.2014.12.014.
38. Azarnoush H, Alzhrani G, Winkler-Schwartz A, Alotaibi F, Gelinas-Phaneuf N, et al. Neurosurgical virtual reality simulation metrics to assess psychomotor skills during brain tumor resection. Int J Comput Assist Radiol.2015;10(5):603-618. doi: 10.1007/s11548-014-1091-z. 
39. Alotaibi FE, AlZhrani GA, Mullah MA, Sabbagh AJ, Azarnoush H, Winkler-Schwartz A, et al. Assessing bimanual performance in brain tumor resection with NeuroTouch, a virtual reality simulator. Neurosurgery. 2015;11(2):89-98. doi: 10.1227/NEU.0000000000000631.
40. Luciano C, Banerjee P, Lemole G, Charbel F. Second generation haptic ventriculostomy simulator using the ImmersiveTouch system. Stud Health Technol Inform. 2006;119:343-348.
41. Banerjee PP, Luciano CJ, Lemole Jr GM, Charbel FT, Oh MY. Accuracy of ventriculostomy catheter placement using a head-and hand-tracked high-resolution virtual reality simulator with haptic feedback. J Neurosurg. 2007;107(3):515-521.
42. Choudhury N, Gélinas-Phaneuf N, Delorme S, Del Maestro R. Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills. World Neurosurg. 2013;80(5):9-19. doi: 10.1016/j.wneu.2012.08.022.
43. Bajunaid KM, M. Winkler-Schwartz, A. Fares, J. Baggiani, M. Azarnoush, H. Christie, S. Alotaibi, F. Al Zharni, G. Marwa, I. Sabbagh, A. Werthner, P. Del Maestro, R.: Acute Stress Impacts Psychomotor Bimanual Performance During Simulated Tumor Resection Task. In Press J Neurosurg 2015
44.     Rosseau G, Bailes J, del Maestro R, Cabral A, Choudhury N, Comas O, et al: The development of a virtual simulator for training neurosurgeons to perform and perfect endoscopic endonasal transsphenoidal surgery. Neurosurgery. 2013; 73(1):85-93.  doi: 10.1227/NEU.0000000000000112.
45. Holloway T, Lorsch Z, Chary M, Sobotka S, Moore MM, Costa AB, et al. Operator experience determines performance in a simulated computer-based brain tumor resection task. Int J Comput Assist Radiol Surg. 2015;10(11):1853-1862.               doi: 10.1007/s11548-015-1160-y. 
46. Varshney R, Frenkiel S, Nguyen L, Young M, Del Maestro R, Zeitouni A, et al. The McGill simulator for endoscopic sinus surgery (MSESS): a validation study. Otolaryngol Head Neck Surg. 2014;43:40 doi: 10.1186/s40463-014-0040-8.
47. Varshney R, Frenkiel S, Nguyen LH, Young M, Del Maestro R, Zeitouni A, et al. Development of the McGill simulator for endoscopic sinus surgery: A new high-fidelity virtual reality simulator for endoscopic sinus surgery. Am J Rhinol Allergy. 2014;28(4):330-334. doi: 10.2500/ajra.2014.28.4046.
48. Kirkman MA, Ahmed M, Albert AF, Wilson MH, Nandi D, Sevdalis N. The use of simulation in neurosurgical education and training. J Neurosurg. 2014;121:228-246. doi: 10.3171/2014.5.JNS131766. 
49. Gelinas-Phaneuf N, Del Maestro RF. Surgical expertise in neurosurgery: integrating theory into practice. Neurosurgery. 2013;73(1):30-38. doi: 10.1227/NEU.0000000000000115.
50. Oliveira Magaldi M, Nicolato A, Santos M, Godinho JV, Brito R, Alvarenga A, et al. Face, content and construct validity of human placenta as a haptic training tool in neuro-interventional surgery.J Neurosurg. 2015:1-7.
51. Magaldi MO, Araujo AB, Nicolato A, Prosdocimi A, Godinho JV, Valle AL, et al. Face, Content and Construct Validity of Brain tumor Microsurgery Simulation using a Human Placental Model. Neurosurgery. 2015:doi.org/10.1227/NEU.0000000000001030.
52. Graber MA, Wyatt C, Kasparek L, Xu YH. Does simulator training for medical students change patient opinions and attitudes toward medical student procedures in the emergency department? Acad Emerg Med.  2005;12(7):635-639.