Research Article Open Access
Surgical Simulation Training Systems: Box Trainers, Virtual Reality and Augmented Reality Simulators
Vasileios Lahanas, Evangelos Georgiou and Constantinos Loukas*
Simulation Center, Laboratory of Medical Physics, Medical School, National and Kapodistrian University of Athens, Athens, Greece
*Corresponding author: Constantinos Loukas, Medical Physics Lab-Simulation Center, School of Medicine, University of Athens, Mikras Asias 75 str., Athens 11527, Greece, Tel: +30-210-7462437; Fax: +30-210-7462369; E-mail: @
Received: March 30, 2016; Accepted: June 30, 2016; Published: July 11, 2016
Citation: Lahanas V, Georgiou E, Loukas C (2016) Surgical Simulation Training Systems: Box Trainers, Virtual Reality and Augmented Reality Simulators. Int J Adv Robot Automn 1(2): 1-9. DOI: 10.15226/2473-3032/1/2/00109
In contrast to Virtual Reality (VR) and traditional surgical training, simulation-based training in Minimally Invasive Surgery (MIS), and Augmented Reality (AR) technology, constitute relatively new application fields in surgery. However, between simulation and AR there is a direct link: the utilization of state-of-the art computer graphics tools and techniques. Nevertheless, an integrated research system that introduces AR properties in simulation-based surgical training scenarios has not yet been proposed in the current literature. In this article we provide a survey on the simulation techniques and systems employed for surgical skills training based on traditional as well as more advanced VR technologies. Moreover, we summarize recent experimental results with regard to the design of a novel ARbased simulator developed in our lab for basic skills training in MIS.

Keywords: Simulation training; Surgery; Virtual reality; Augmented reality
An Introduction to MIS
Minimally invasive surgery (MIS), also called minimal access or endoscopic surgery has become a universally accepted method for performing certain surgical operations during the past three decades. Contrary to the traditional open surgery where operations are performed through a large incision on the area of interest, surgical procedures in MIS only require one or more tiny incisions at a size of a few millimeters (usually 0.5-1.5 cm) on the patients' skin. In general, the main benefits of minimally invasive operations compared to traditional open surgery are: shorter recovery times, less discomfort during rehabilitation and better aesthetic outcome. Amongst the numerous types of surgical procedures performed in a minimally invasive approach nowadays, the majority of them is applied in the abdominal cavity (laparoscopy). The surgeon manipulates the instruments while looking at a display monitor that shows images of the patient's abdomen acquired with the endoscope. Once the operation is finished, the carbon dioxide is expelled from the abdomen and the incisions are closed using a small number of stitches.

The patient's benefits of MIS may be summarized as:
1. Minimized risks of blood loss.
2. Reduced need for blood transfusion.
3. Smaller chance for post-operational bleeding implications.
4. Reduced need for long-term pain relief medication, which is commonly required in open surgery for relieving patients from stitch-healing pain.
5. Reduced exposure of the internal organs to external contaminants.
6. Smaller scar after surgery, which translates to fewer chances of post-operational infection.

Healthcare providers also benefit in many ways from laparoscopic surgery. First and foremost, the improved patient care, originating from the reduced medical risks, improves the overall efficiency of healthcare delivery. In addition, the shorter rehabilitation time leads to more surgical procedures per year and consequently to higher revenues. Although the latter might seem insignificant compared to patient safety, it is also a parameter of great importance, since healthcare revenues translate to more investments, and in long terms may lead to further improvement of the healthcare system.
Training and assessment of MIS skills
Laparoscopic surgery is performed using a rigid endoscope, introduced into the patient's abdomen through a small incision. Using conventional laparoscopic imaging systems, instead of the direct 3D view of open surgery, surgeons obtain only a 2D view of the operating area through a display monitor. This monitor is usually positioned in front and on top of the surgeon's head, and consequently he/she must learn to operate while looking at the monitor and not at the patient's operated site. Furthermore, the restricted 2D vision combined with the limited field of endoscopic view creates a lack of depth perception. To cope with this, surgeons must learn to depend on other cues in order to enhance their overall sense of depth. These cues are primarily the sense of touch and the interpretation of lights and shadows.

Indirect vision is not the only issue in MIS. The standard surgical tools involved in open surgery are also not applicable in laparoscopy. Instead, long-shaped surgical instruments are utilized, inserted into a patient's abdomen through trocars. The use of such instruments increases the operation difficulty for several reasons, the most important of which is the lack of adequate force and haptic feedback. Kinesthetic feedback, providing indications regarding the forces exerted on tissue, is also significantly reduced, thus making it more difficult for surgeons to operate on subtle anatomies such as vessels.

In addition, the long shape of laparoscopic instruments also produces a number of additional challenges. Firstly, their long length amplifies tremor and makes harder to control the instruments. Secondly, the use of trocars reduces the degrees of freedom of movement. Thirdly, the poor ergonomic design of laparoscopic instruments' handles makes them difficult to manipulate. Another difficulty of laparoscopic surgery arises from what is called the fulcrum effect: the abdominal wall acts as a fulcrum where hands movements towards one direction result in opposite instruments movements. Surgeons need to familiarize with this effect and perform actions in a totally different way compared to standard laparotomy. Due to the combination of all the aforementioned factors, highly trained and experienced specialists are required for successfully performing laparoscopic interventions, mastering a different set of cognitive, psychomotor and visio-spatial skills. Traditional methods of surgical skill acquisition, cannot accommodate the new skills required for MIS. Criticizing this lack of established curricula and its unstructured expansion, Cuschieri and Shapiro stated in 1995 [1] that laparoscopy was: "the biggest unaudited free-for-all in the history of surgery".
Laparoscopic box-trainers
Based on the aforementioned considerations, in the years that followed the initial expansion of laparoscopy, the surgical community was prompted to reconsider the training strategy to facilitate the skills required for MIS skills acquisition. In MIS, the supervising surgeon cannot directly guide the trainee's hands, nor intervene to rectify a complication during the procedure. Hence, the surgical community looked for new teaching methods that would allow surgeons to acquire fundamental skills outside of the operating theater, aiming to a safe introduction of new techniques into surgical practice. Societies, regulatory bodies and organizations, with the most important being the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES [2]) and the European Association of Endoscopic Surgeons (EAES [3]), defined minimum requirements for those are performing laparoscopic surgery.

Training in a laboratory setting was not a new concept in surgery and medicine. Human cadavers and animals were already utilized for teaching and practicing skills in open surgery. Both options however are criticized for a number of reasons: expensive way of training, the animal's anatomy is differed to that of humans, increased maintenance costs, and training on animals creates ethical considerations. Hence, surgical community searched for alternative solutions, utilizing synthetic training modalities and stepping towards simulation-based training.

In order to alleviate the aforementioned drawbacks, several manufacturers have developed laparoscopic training platforms, called box-trainers. This is a platform with a very simple design that includes a box with holes for trocar insertion, simulating the insufflated abdominal cavity. Surgeons can practice their skills using real laparoscopic instruments and a camera that simulates the endoscope, while manipulating simple objects such as pegs and inanimate models of human organs. A great leap towards the establishment of box-trainers in the standard surgical training curricula was the development of the Fundamentals of Laparoscopic Surgery (FLS) program, a joint educational program between SAGES and the American College of Surgeons (ACS). The FLS, adopting a commercial training system called McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS), was introduced to systematize training and evaluation of both cognitive and psychomotor skills required to perform MIS. This program included both a cognitive component as well as a manual (psychomotor) component. The psychomotor component of the FLS was a trainer toolbox that allowed testing of five pre-defined tasks: peg transfer, pattern cutting, ligation loop and suturing with intracorporeal as well as extracorporeal knot tying, illustrated in figure 1. Performed in a box-trainer, these tasks were designed for acquiring essential psychomotor skills such as picking, transferring between dominant and nondominant hands, knot tying, precision cutting and ligating slender structures. In addition to skill training, the FLS committee also developed task-specific metrics for subjective performance assessment and evaluation.

Over the years, the primitive versions of the first generation of box-trainers evolved into sophisticated training platforms that are characterized as Physical Reality (PR) surgical simulators, since they require trainees to operate standing up within the confines of simulated anatomy such as the pelvis and upper abdomen (Figure 2). On their advanced form, the simple boxes were substituted by human torso models while some high-end platforms consisted of the exact hardware setup of an operating room, including a real endoscope with accompanying fiber-optic light source and display (Figure 2).

Except from the aforementioned benefits, there were some limitations in training with physical models. First of all, training models require replacement often, hence increasing the cost of training. In addition, training individual skills with basic FLS drills did not provide sufficient experience to trainees regarding actual surgical operations, which would yet require practicing on a real operating theater. Finally, physical reality simulators lacked efficient objective assessment, since the overall process of supervising the training sessions, obtaining the scores and evaluate trainees' performance entailed a considerable amount of labor, time and cost. These considerations, along with the advancement of computer science and technology during the past decades, motivated the development of computer-based laparoscopic simulation that would allow automation of training
Figure 1: The collaborative AFP machine.
Figure 2: Box-trainers evolved into high-end training platforms, providing simulation in a physical reality (PR) environment.
Figure 3: Laparoscopic cholecystectomy performed in a purely virtual environment of a commercial VR trainer.
and assessment curricula and would exploit the limitless potentials of Virtual Reality (VR).
Virtual reality laparoscopic simulation
An initial approach towards the utilization of VR technology for surgical simulation was the work of Delp, et al. [4] who described an interactive computer system that simulated the effects of surgical intervention in the musculoskeletal system of human lower extremity. A pioneer work however, first discussing a "virtual reality surgical simulator", was presented in 1993 by Satava [5]. Inspired by the aviation paradigm, Satava discussed the implementation of a primitive VR surgical simulator that utilized VR hardware and software, creating a computergenerated 3D abdomen that included models of human organs (pancreas, stomach, etc.).

In the following years, and especially during the mid-1990s, the field of computer graphics demonstrated tremendous progress. Powered by revenues of the PC-gaming industry, faster graphics accelerators and CPU technology were designed, resulting in constantly increasing levels of visual realism regarding computer-generated graphics. In 1997 the first commercially available surgical simulator was presented [6]. It was VR laparoscopic simulator called MIST-VR (Minimal Invasive Surgery Trainer-Virtual Reality), developed as a joint venture between the Wolfson Center for Minimally Invasive Therapy in Manchester and the company VR Solutions. MIST-VR provided a realistic and assessable VR environment where trainees could practice in six purely virtual training tasks, simulating some of the basic maneuvers performed during laparoscopic cholecystectomy.

Nowadays, VR laparoscopic simulators have become high-end platforms that combine cognitive and motor skills training into an integrated VR learning experience, providing unique training opportunities in a highly realistic, purely virtual environment. Except from a PC, the mechanical setup of VR consists of custom laparoscopic instruments equipped with a number of sensors that record instrument's movements, a custom device simulating the laparoscopic endoscope and a display monitor. In addition, some trainers are equipped with diathermy foot pedals. Finally, the most high-end VR trainers are equipped with mechanical feedback devices connected to the laparoscopic instruments. This is a crucial component of VR platforms since it provides real-time haptic feedback during training, hence enhancing the overall sense of simulation realism.

In VR trainers, surgeons can be individually guided through a series of training scenarios of progressive difficulty and complexity. This way, novice surgeons are allowed a smooth skill development as well as transition of skills from training to clinical practice. In VR trainers, a large set of different types of basic procedures can be performed, including endoscope navigation, cutting and suturing, needle driving, diathermy and other essential exercises similar to the FLS tasks described earlier. The virtual models utilized for the aforementioned training tasks can vary from simple geometrical shapes such as virtual pegs and cubes to complex ones such as virtual models of human anatomical structures (see Figure 3). In addition to skill learning and exercising, VR trainers also provide automated objective assessment of the performance, based on specific metrics recorded during training. Such metrics can be the task completion time, the number of errors committed as well as a variety of additional parameters such as the instruments' path length. Utilizing validated algorithms, VR trainers objectively assess the performance and efficiency of trainees' in each scenario, providing important feedback for their individual psychomotor and cognitive skills, as required for performing a real laparoscopic surgery. Some popular VR surgical simulators are depicted in figure 4.
Augmented Reality
In the previous sections, we discussed the concept of simulation-based training for laparoscopic surgery, focusing on the two simulation modalities used in MIS training: Physical Reality (PR) box-trainers and Virtual Reality (VR) simulators. A new technology bridging the gap between PR and VR is Augmented Reality (AR), also called Mixed Reality (MR). AR supplements the real world by introducing virtual objects, which appear to coexist in a common environment with objects of the real world. Nowadays AR is gradually evolving into a valuable alternative of VR. Several applications introducing AR as the new alternative have emerged in various fields, the most important of which being the game of 3D gaming. Since VR and AR essentially depend on the same building blocks (computer graphics), a transition from one technology to the other would be a natural progress. And indeed, as later sections will present, this is the case for many applications aiming to introduce AR in the medical context. Despite the fact that AR is constantly gaining ground in other medical fields however, its potential utilization in the field of laparoscopic simulation training has not yet been exploited.

In order to introduce AR and allow readers to identify the aforementioned observations, which motivated the fulfillment of the present thesis, the following sections present an introduction to AR, including a brief historical review, the technologies involved and finally a review regarding the use of AR in surgical applications, primarily focusing on MIS and laparoscopy.
History and applications
AR was coined in 1990 from Tom Caudell [7]. In a search to find an easier way to help the aviation industry manufacturing and engineering process in assembling aircraft wiring, Caudell invented a complex system that could overlay the positions of where certain cables in the building process were supposed to go, described it as "augmented reality". While however AR firstly
Figure 4: Commercially available laparoscopic VR trainers: (a) LapVR (CAE), (b) LapMentor (Simbionix), (c) LapSim (Surgical Science).
appeared as a term during the early 1990's, some applications involving machine generated enhancements of the real world have already been present many decades before. The first such application was a device called Sensorama, developed in 1957. Sensorama was designed as a cinematic experience that enhanced the senses of viewers. Since the projected video illustrated images of the real world enhanced with simulated effects, Sensorama is considered as the first AR application system. In 1966, Ivan Sutherland and his research group invented the first Head- Mounted Display (HMD) device allowing the mixture of real-world images with (virtual) visual enhancements. This invention was an innovation for the computer graphics community, initiating a whole new field of research around wearable/portable display systems and it is considered the first AR display.

The true potentials of AR began to appear during the early 1990s. Louis Rosenberg created what is widely recognized as the first fully functional AR system for the US Air Force known as Virtual Fixtures. Another group of researchers brought out an idea of a prototype system called Knowledge-based Augmented Reality for Maintenance Assistance (KARMA) [8,9]. The idea was to develop a system that would supply visual instructions for loading and servicing a laser printer, using virtual graphics and HMD glasses instead of a written manual.

Despite the increasing frequency of AR appearance in the literature however, not a commonly accepted description of what the term "augmented reality" stands for, existed before 1994. In an effort to derive with a consistent definition, Milgram, et al. [10] described AR as "a form of virtual reality where the participant's head-mounted display is transparent, allowing a clear view of the real world". However, this definition suffered from an important restriction; it narrowed AR to the use of HMD devices which were a trend at that time, excluding a wide range of alternative display devices, such as monitor-based interfaces, monocular systems, mobile phones and various other combining technologies. To overcome this limitation and avoid relating AR to a specific technology, Azuma, et al. [11] contributed a broader definition. Specifically in their work, the authors stated that in order for an application to be characterized as AR, it must fulfill the following requirements:

1. Real and virtual objects must be combined in a common environment.
2. The system must run interactively and in real time
3. Real and virtual objects must be spatially registered

Through the years, a compact definition of AR based on the aforementioned rules prevailed in the literature: Augmented reality is the introduction of superimposed graphics, audio and other sense enhancements over a real-world environment that is interactive in real-time.

Although the interest around the employment of AR in everyday activities was growing, AR remained mostly a subject of scientific research for many years. In 2000, Hirokazu Kato from the Nara Institute of Science and Technology released the ARToolkit library as an open source project to the internet community. The ARToolkit library provided a robust software solution for real-time camera pose tracking, but also yielded a tool for the development of AR applications using a standard web camera instead of the costly display apparatus used at the time. The public distribution of ARToolkit, followed by a number of similar software libraries such as ARTag, along with the "invasion" of smart phones in our everyday lives, significantly contributed in AR becoming a widespread technology, earning its position as an alternative to VR.
Augmented reality in open surgery and MIS
In an era that pre-operative medical information became digitized through the use of MRI/CT equipment etc. [12], AR constituted the most suitable way of introducing computer generated information into a surgical environment. When applied to the surgical context, the most important requirement of AR is registration; any virtual element introduced into a scene must be spatially registered with respect to the real objects involved in this scene. Given the fact that a surgical environment is highly unpredictable, involving both rigid movable objects such as surgical instruments as well as deformable structures of the human anatomy, implementation of real-time AR requires overcoming a series of technical challenges. These challenges have to mainly deal with hardware limitations arising from the special conditions that apply in an operating theater and the high levels of accuracy required in surgical procedures. Especially the latter poses a major obstacle, since sub-millimeter accuracy on the registration of virtual objects is a perquisite.

The first medical application that included spatial registration between real anatomical structures and virtual elements was introduced in the middle of the 1980s in the field of neurosurgery. The proposed system had clear characteristics of AR since it spatially registered CT images with patient anatomy using fiducial markers attached on a human skull. Another pioneer work was proposed by Bajura, et al. [13]. The authors described a system for real-time augmentation of ultrasound images on top of a pregnant human subject where real-time visualization was achieved using a small video camera mounted in front of a conventional head-mounted display.

The following years, introduction of AR in the Operating Room (OR) became a popular subject of research amongst scientists both from the field of medicine as well as computer science and a large number of applications focusing on different subfields of surgery were presented. The main focus between researchers has been primarily put on the use of AR for guidance. Mischkowski, et al. [14] presented an AR tool for maxillary positioning in orthognathic surgery using a portable see-through device for overlaying pre-segmented CT and MRI 3D images on top of the patient's skull. The position of the display device with respect to the patient is being tracked using a system of infrared optical cameras positioned on the center of the operating theater. Nicolau, et al. [15] described an AR guidance system for liver punctures, helping surgeons to reach tumors and perform Radio-Frequency (RF) treatment via the superimposition of preoperative MRI information. Their system utilizes a single camera display system and radio-opaque pattern markers for real-time tracking of surgical instruments. Magee, et al. [16] presented a navigation system for ultrasound guided needle placement training. Their proposal included generation of ultrasound anatomic images on top of an inanimate model of a human torso, using Electromagnetic (EM) motion sensors for tracking the ultrasound probe and the needle. This concept has been proposed for several medical applications requiring needle intervention, such as for instance MRI-guided needle placement in spinal biopsies.

Up to date, ideas employing AR for the visualization of preoperative data on top of the patient during a surgery have been widely proposed in the literature, extensively investigating the potentials of various displays and tracking technologies, both in vitro and in vivo. Except from surgery though, AR has also been employed in non-surgical medical applications. Talbot, et al. [17] for instance, proposed a system for patient positioning and monitoring in radiotherapy. Their system utilizes AR by acquiring live images of the LINAC and superimposing a virtual representation of the patient body contour onto the correct position on the treatment couch.

Augmented reality may also be utilized for enhancing the surgeons' field of view and minimize the lack of visual perception by augmenting pre-operative patient data on top of the patient's anatomical structures. A system achieving this in a robust and accurate way would be a tool of major importance in the hands of surgeons, leading to safer operations and minimized implication risks. The first such approach however did not appear before the late 1990s when Freysinger et al. developed an image guidance system for endoscopic ENT surgery. Their system achieved realtime superimposition of a 3D path towards predefined targets on top of endoscopic images. On the same period, Fuchs, et al. [18] proposed a system that obtained the 3D surface of patient's anatomy using a 3D endoscope and superimposed its virtual reconstruction on top of the real anatomy using HMD glasses instead of the typical monitor involved in laparoscopic surgery. Also targeting in MIS, Traub, et al. [19] proposed a system that would help surgeons in achieving optimal port placement and intra-operative navigation in robotically assisted minimally invasive cardiovascular surgery.

Despite however the constantly increasing number of relative scientific publications, the introduction of AR into MIS still faces significant difficulties. Nicolau, et al. [12] characterized the possibility of providing augmented reality information in the endoscopic view as the holy grail for surgeons and a great challenge for augmented reality researchers. In that work, they focused on three critical issues and potential solutions; display technologies, real-time tracking methods and virtual objects registration techniques. These are the main technical challenges for any researcher aiming to introduce AR in MIS, while almost each possible combination of existing solutions has been proposed in the current literature.

Regarding the display technologies, although a wide range of AR displays are available, not all of them are applicable in MIS. Mobile devices and portable projectors for instance are obviously excluded since occupying surgeons' hands during an operation is not an option. Alternative solutions such as the use of HMDs worn by surgeons while operating on a patient, or see-through displays where pre-operative data are projected onto a semitransparent surface laying between the surgeon and the patient have been proposed in the literature. These options however, alter the typical way MIS is performed, introducing additional apparatus in the OR.

The additional technical challenges, spatial registration and 3D tracking, are common in any surgical application involving augmentation of virtual information. In MIS though, both issues are of critical importance since surgeons obtain a close view of the operating area and hence even the smallest inaccuracies heavily distort their visual perception. Spatial registration in endoscopic soft tissue surgery poses a special challenge since established optical and electromagnetic tracking devices only deal with rigid structures and are not readily able to track soft tissue deformations. A number of approaches have emerged aiming to overcome these challenges including intraoperative image acquisition with ultrasound, MRI or CT, or approaches using endoscopes as sensors to track artificial or natural features, as reviewed by Baumhauer, et al. [20] and Mirota, et al. [21].

In addition, utilizing 3D tracking techniques that apply in other surgical procedures is also a more challenging issue in MIS. The operating space is limited and consequently, inserting devices such as electromagnetic sensors or infrared-optical sensors into the patients' abdomen is not always practical. On the other hand, a typical MIS operation involves a limited number of movable objects (surgical instruments), which perform controlled movements and appear in specific shapes and dimensions with respect to the endoscopic camera. Thereby, an ideal approach would implement vision-based techniques for real-time tracking of the instruments kinematics. Several instrument tracking techniques have been proposed, either targeting on AR laparoscopic applications, or other types of applications that require knowledge of the instruments' kinematics. Despite the spatial limitations of MIS described earlier, other instrument tracking methods involving the use of optical sensors, electromagnetic sensors or pattern markers attached on the instruments' handles have been proposed.
Augmented reality in MIS simulation-based training
Although AR has been applied in real operating environments, in MIS simulation training there is a clear gap in the current literature for AR laparoscopic simulation platforms. The first appearance of a scientific publication focusing on AR-based laparoscopic simulation, was published in 2005 [22], evaluating the construct validity of the ProMIS laparoscopic simulator [23]. ProMIS is a commercial laparoscopic simulation platform that according to its specifications provides an AR training environment, enables users to interact with virtual and physical models in the same unit while providing accurate, comprehensive feedback on performance. Indeed this platform is the first, and up to date the only commercial simulator to introduce VR elements in a PR training environment. However, virtual augmentations are employed for guidance purposes (e.g. indicating where a tissue cut should be performed), or for achieving enhanced visual effects (e.g. adding bleeding effects at points where a cut has been performed), without though implementing any interaction between the instruments and virtual objects whatsoever.

Several studies have been performed to evaluate the construct and face validity of ProMIS in comparison to other available modalities (VR or PR), presenting promising findings regarding the advantages of AR compared to other modalities used in laparoscopic simulation training, as illustrated in table 1. In the most cited amongst these studies, Botden, et al. [7,24] compared ProMIS with a state-of-the-art VR laparoscopic simulator (LapSim). The outcome of this study allowed the authors to conclude that due to better realism and improved haptic feedback, ProMIS AR laparoscopic simulator outperformed LapSim VR platform in terms of didactic value and construct validity. Based on their results, the authors recommended the implementation of ProMIS in the training curricula in laparoscopic skills for surgical residents.

In another work, Botden, et al. [25] presented a review and classification of the existing AR simulators. Aiming to provide an overview of the existing AR laparoscopic simulation market, the authors compared commercially available training platforms that according to their claim fitted the characteristics of AR; ProMIS, the Blue Dragon [26], the LTS3-E [27] and the CELTS [28]. However, the authors considered as AR any system that enhanced purely physical simulation training with any kind of virtually generated information, such as performance assessment results automatically obtained and processed by a computer. And although these systems indeed target on computer-enhanced laparoscopic simulation, only ProMIS suits the fundamental requirement of AR, which is the combination of virtual and real elements in a common environment [11].
A novel AR simulator for surgical training
Over the last few years, our research group in the Medical Physics Laboratory-Simulation Center (MPLSC) has experimented with the implementation of technologies and techniques for laparoscopic simulation training using AR technology. Our first
Table 1: Advantages and disadvantages of PR, VR and AR in MIS simulation training

Table 1: Advantages and disadvantages of PR, VR and AR in MIS simulation training






- Realistic haptic feedback-

- Cost-effective setup

- Objective assessment


-Realistic haptic feedback-Objective

- assessment



- Subjective    assessment

-Lack of interactivity

- Unrealistic haptic feedback

-Lack of assessment protocol



-Lack of assessment protocol


approach comprised the implementation of simple AR-based training scenarios for psychomotor skills assessment using utilizing pattern-markers for laparoscopic instrument tracking [29]. Based on the observations from our initial efforts, we have developed and evaluated an image processing algorithm that allowed real-time tracking of the laparoscopic instruments' pose within a typical box-trainer. The aforementioned technique allowed the creation and evaluation of additional AR scenarios for basic skills training in MIS [30]. Moreover, our contribution included the development of a calibration method for acquiring the spatial relationship between laparoscopic instruments and Electromagnetic (EM) sensors attached on the instrument's handles. The aforementioned calibration method proved a significant tool for the implementation of more complex AR training scenarios [31].

The next step of our research efforts was the development of a novel AR simulator for basic skills training in MIS [32]. The proposed system allows the implementation of training scenarios for technical skills acquisition such as perception of depth of field, hand-eye coordination and bimanual operation. Based on this system, the trainee is able to interact with various virtual elements introduced into the box-trainer, using the actual laparoscopic instrumentation (camera and tools). The following section provides a summary of the main components, the functionality and the evaluation of the AR surgical simulator developed in our lab.
Hardware setup: As shown in Figure 5, the main components of the system include: a standard PC with a monitor, a fire-wire camera with appropriate wide-angle lenses, a boxtrainer, laparoscopic tools, and three different types of sensors attached to the laparoscopic tool. Eight Degrees of Freedom (DOF) information about the tool kinematics was provided by the sensors: 3D poses of the tool, shaft rotation, and opening angle of the tooltip. A fiducially pattern marker is placed on the bottom surface of the box trainer to define the global coordinate system of the simulation environment. The pose of the tool was obtained via trakStarTM (Ascension Tech Corp., Burlington, VT) electromagnetic position-orientation sensors. The EM transmitter is placed at a fixed position within the box-trainer, (Figure 5) whereas the sensors are attached to the tool handles.

A custom-made rotary encoder controlled via an Arduino(http:// microcontroller board is employed to measure the rotation of the shaft. The encoder consists of a magnetic rotor firmly attached to the shaft, and a plastic stator attached to the instrument's handle. Inside the stator, two Hall Effect sensors, positioned with an approximate 90 degree angle separation, detect changes on the sinusoidal waveform that the magnetic rotor generates during rotation. The voltage output of the HFs is digitally converted using a microcontroller's analog to digital converter. Calculation of the shaft rotation angle is obtained with the CORDIC algorithm. This setup allows us to obtain angular measurements for the full 360° range of the shaft revolution.

Simulation engine: The development of the VR-based laparoscopic tasks was based on the Ogre3D (
Figure 5: Experimental setup of AR simulator, consisting of a box-trainer, two laparoscopic instruments equipped with appropriate sensors, an electromagnetic (EM transmitter), and a fire-wire camera.
graphics library. A flowchart and the main components of the simulation engine are presented in figure 6. The ARToolkitPro( library was used to calculate the spatial relationship between the camera and the boxtrainer reference frame in terms of rotation and translation, by tracking the pose of the pattern marker described earlier. The 3D models of the virtual objects employed in the simulation tasks, were performed in Blender3D( The physics-based modelling of the virtual objects (collision detection, response between tools and virtual objects, and soft body deformations), was implemented via the Bullet(http:// library. In order to meet the specific needs of each task and achieve realistic behavior of the virtual objects, several modifications and supplementary algorithms to the Bullet source code were applied.

Task description: Three training tasks were developed, targeting different technical skills in laparoscopic surgery (Figure 7):

Task 1: Instrument Navigation (IN). Eight rounded buttons, each one enclosed in a black cylinder, is introduced to the center of the platform (Figure 7A). The task requires the user to hit the buttons in a sequential order, when each one of them is highlighted in green. The order on which the buttons get highlighted is random. The user is given a time limit to hit each button. Two types of error were recorded during task performance: the number of missed targets due to time lapse, and the number of tooltip collisions with either the base of the box-trainer or the black cylinders.

Task 2: Ring Transfer (RT). Four cylindrical targets and an equal number of rings are introduced sequentially into the scene (Figure 7B). Each target has a distinctive color. The user has to lift and transfer each ring into the target with the same color. When
Figure 6: Flowchart and the main components of the simulation engine of the AR laparoscopic simulator shown in Figure 5.
Figure 7: Screenshots of the three AR training tasks: (a) Instrument navigation, (b) ring transfer, (c) clipping.
a ring has been transferred, another one (of different color) is introduced. Two types of error were recorded: unsuccessful transfers and ring drops.

Task 3: Clipping (CL): Initially, a virtual vein is shown (Figure 7C). The center of the vein is highlighted in green and two different locations on either side of its center are highlighted in red. The task goal is to apply a clip at the center of each of the red regions using a virtual clip applicator. Once clipped, the user has to use a pair of virtual scissors to cut at the center of the green region. Two different metrics were recorded: The distance error during clipping, and cutting, with respect to the center of the corresponding region, and the number of unsuccessful clipping and cutting attempts, which are recorded when the user clips/ cuts outside the highlighted area.
We have collected data from subjects with two different levels of expertise: ten experienced surgeons (experts), and ten individuals with no experience in laparoscopic surgery (novices). Each participant performed two trials of each task. To evaluate the construct validity of the proposed system (i.e., whether the system can distinguish different levels of expertise), a statistical comparison of the two experience groups was performed with regard to the three tasks described previously (see Table 2). The performance metrics included the two types of errors for each task, the task completion time, and the total path length of the laparoscopic tools. Our results showed a statistical significance in terms of recognizing experts from novices across all metrics and tasks attempted. Moreover, experts outperformed novices in all tasks and for all metrics assessed.
Creating a computer-based surgical simulator from the beginning is a challenging and demanding task, requiring involvement and knowledge of a wide range of topics. In general, the main design requirements for developing such simulator can be generally categorized in five fields; computer graphics, physicsbased modeling, sensory equipment, and finally hardware/ software design. In addition to the technical challenges involved, building and validating a surgical simulator for laparoscopic training and assessment also requires background knowledge of the medical and surgical aspects; the psychomotor and cognitive skills that a simulator has to support, and how these skills will be trained and assessed via appropriate training scenarios.

The box trainers provide an affordable solution for surgical skills training allowing the trainee to use the actual surgical instruments and interact with physical models, thus allowing the real feel of force feedback. On the other hand though, assessment is difficult since an expert has to review the videos of the training tasks. Virtual reality simulators are more expensive and allow the practice of various scenarios in one machine, without the need for model replacement. Objective assessment is a straightforward process since the sensory equipment allows acquisition of various kinematic parameters during task performance. However, the graphics and force feedback sense are not always realistic. An augmented reality simulator is a new direction that attempts to combine the advantages of the aforementioned training modalities. Further experimentation, validation studies, and system development with force feedback capabilities are required in order to demonstrate the training potential of AR technology with regard to VR and physical reality systems.
Table 2: Statistical comparison (p values) between the two experience groups for the four metrics (columns) and the three tasks (rows) performed on the AR simulator.

Table 2: Statistical comparison (p values) between the two experience groups for the four metrics (columns) and the three tasks (rows) performed on the AR simulator.




Error 1

Error 2

Instrument navigation





Ring transfer










The views expressed in this article are the authors' views and not an official position of the institution.
  1. Cuschieri A, Shapiro S. Extracorporeal pneumoperitoneum access bubble for endoscopic surgery. Am J Surg. 1995;170(4):391-4.
  2. SAGES-Society of American Gastrointestinal and Endoscopic Surgeons. Accessed 1 Oct 2015:
  3. EAES-Home. Accessed 1 Oct 2015:
  4. Delp SL, Loan JP, Hoy MG, Zajac FE, Topp EL, Rosen JM. An interactive graphics-based model of the lower extremity to study orthopaedic surgical procedures. IEEE Trans Biomed Eng. 1990;37(8):757-67.
  5. Satava RM. Virtual reality surgical simulator. The First steps. Surg Endosc. 1993;7(3):203-5.
  6. Wilson MS, Middlebrook A, Sutton C, Stone R, McCloy RF. MIST VR: a virtual reality trainer for laparoscopic surgery assesses performance. Ann R Coll Surg Engl. 1997;79(6):403-4.

  7. Wilson MS, Middlebrook A, Sutton C, Stone R, McCloy RF. MIST VR: a virtual reality trainer for laparoscopic surgery assesses performance. Ann R Coll Surg Engl. 1997;79(6):403-4.
  8. Surgical Science - The global leader in medical simulation training.Accessed 22 Sep 2015:
  9. Höllerer TH, Feiner SK. Mobile Augmented Reality. 2004;1-39.
  10. Krevelen DWF Van, Poelman R, A Survey of Augmented Reality Technologies, Applications and Limitations. 9.2010.
  11. Paul Milgram, Haruo Takemura, Akira Utsumi, Fumio Kishino. HTAUFK Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum.   
  12. Azuma RR, Azuma RR. A survey of augmented reality. 1997; Presence 4:355–385. doi:
  13. Nicolau S, Soler L, Mutter D, Marescaux J. Augmented reality in laparoscopic surgical oncology. Surg Oncol. 2011;20(3):189-201. doi: 10.1016/j.suronc.2011.07.002.
  14. Bajura M, Fuchs H, Ohbuchi R. Merging virtual objects with the real world. ACM SIGGRAPH Comput Graph. 1992;26:203–210. doi: 10.1145/142920.134061.
  15. Mischkowski RA, Zinser MJ, Kübler AC, Krug B, Seifert U, Zöller JE. Application of an augmented reality tool for maxillary positioning in orthognathic surgery - a feasibility study. J Craniomaxillofac Surg. 2006 Dec;34(8):478-83.
  16. Nicolau SA, Pennec X, Soler L, Ayache N.A complete augmented reality guidance system for liver punctures: First clinical evaluation. Med Image Comput Comput Assist Interv. 2005;8(Pt 1):539-47.
  17. Magee D, Zhu Y, Ratnalingam R, Gardner P, Kessel D. An augmented reality simulator for ultrasound guided needle placement training. Med Biol Eng Comput. 2007;45(10):957-67. doi: 10.1007/s11517-007-0231-9.
  18. Talbot J, Meyer J, Watts R, Grasset R. An augmented reality application for patient positioning and monitoring in radiotherapy. IFMBE Proc. 2009;25:620-623. doi: 10.1007/978-3-642-03474-9-174.
  19. Fuchs H, Livingston M, Raskar R. Augmented reality visualization for laparoscopic surgery. Medical Image Computing and Computer-Assisted Interventation. 1998;934-943.
  20. Traub J, Feuerstein M, Bauer M, Schirmbeck E, Najafi H, Bauernschmitt R, et al. Augmented reality for port placement and navigation in robotically assisted minimally invasive cardiovascular surgery. Int Congr Ser 1268:2004:735–740. doi: 10.1016/j.ics.2004.03.049.
  21. Baumhauer M, Feuerstein M, Meinzer HP, Rassweiler J. Navigation in endoscopic soft tissue surgery: perspectives and limitations. J Endourol. 2008;22(4):751-66. doi: 10.1089/end.2007.9827.
  22. Mirota DJ, Ishii M, Hager GD. Vision-based navigation in image-guided interventions. Annu Rev Biomed Eng. 2011;13:297-319. doi: 10.1146/annurev-bioeng-071910-124757.
  23. Van Sickle KR, McClusky DA, Gallagher AG, Smith CD, McClusky  3rd DA, Gallagher AG, et al. Construct validation of the ProMIS simulator using a novel laparoscopic suturing task. Surg Endosc. 2005;19(9):1227-31. doi: 10.1007/s00464-004-8274-6.
  24. CAE- Accessed 22 Sep 2015:
  25. Botden SM, Buzink SN, Schijven MP, Jakimowicz JJ. Augmented versus virtual reality laparoscopic simulation: what is the difference? A comparison of the ProMIS augmented reality laparoscopic simulator versus LapSim virtual reality laparoscopic simulator. World J Surg. 2007;31(4):764-72.doi: 10.1007/s00268-006-0724-y.
  26. Botden SM, Jakimowicz JJ. What is going on in augmented reality simulation in laparoscopic surgery?.Surg Endosc. 2009;23(8):1693-700. doi: 10.1007/s00464-008-0144-1.
  27. Rosen J, Brown JD, Barreca M, Chang L, Hannaford B, Sinanan M. The Blue DRAGON--a system for monitoring the kinematics and the dynamics of endoscopic tools in minimally invasive surgery for objective laparoscopic skill assessment. Stud Health Technol Inform. 2002;85:412-8.doi: 10.3233/978-1-60750-929-5-412.
  28. Soyinka AS, Schollmeyer T, Meinhold-Heerlein I, Gopalghare DV, Hasson H, Mettler L. Enhancing laparoscopic performance with the LTS3E: a computerized hybrid physical reality simulator. Fertil Steril. 2008;90(5):1988-94.doi: 10.1016/j.fertnstert.2007.08.077.
  29. Stylopoulos N, Cotin S, Maithel SK, Ottensmeye M, Jackson PG, Bardsley RS, Neumann PF, Rattner DW, Dawson SL, Ottensmeyer M, Jackson PGG, Bardsley RSS, Neumann PFF, Rattner DWW, Dawson SLL. Computer-enhanced laparoscopic training system (CELTS): bridging the gap. Surg Endosc. 2004;18(5):782-9.doi: 10.1007/s00464-003-8932-0.
  30. Lahanas V, Loukas C, Nikiteas N, Dimitroulis D, Georgiou E. Psychomotor skills assessment in laparoscopic surgery using augmented reality scenarios. In: 2011 17th Int. Conf. Digit. Signal Process. IEEE. p. 1–6.
  31. Loukas C, Lahanas V, Georgiou E. An integrated approach to endoscopic instrument tracking for augmented reality applications in surgical simulation training. Int J Med Robot. 2013;9(4):e34-51. doi: 10.1002/rcs.1485.
  32. Lahanas V, Loukas C2, Georgiou E2. A simple sensor calibration technique for estimating the 3D pose of endoscopic instruments. Surg Endosc. 2016;30(3):1198-204. doi: 10.1007/s00464-015-4330-7.
  33. Lahanas V, Loukas C, Smailis N, Georgiou E. A novel augmented reality simulator for skills assessment in minimal invasive surgery. Surg Endosc. 2015;29(8):2224-34. doi: 10.1007/s00464-014-3930-y.
Listing : ICMJE   

Creative Commons License Open Access by Symbiosis is licensed under a Creative Commons Attribution 4.0 Unported License