Review article Open Access
Applying Predictive Processing and Functional Redeployment to Understanding Embodied Virtual Experiences
Shulan Lu1*, Derek Harter2
1Department of Psychology, Texas A&M University-Commerce
2Department of Computer Science, Texas A&M University-Commerce>
*Corresponding author: Shulan Lu, Department of Psychology, Texas A&M University-Commerce, Commerce, TX 75429,Tel: 903-468-8628;Fax: 903-886-5510; E-mail: @
Received: September 23, 2016; Accepted: October 07, 2016; Published: October 19, 2016
Citation: Howard RC (2016) The Link between Early Adolescent Alcohol Abuse and Adult Antisocial Behaviour: A Hypothesis Revisited. SOJ Psychol 3(2): 1-6. DOI:
Recent theories of cognition have begun to go beyond embodiment to think about general mechanisms that may help explain many of our observations about the nature of embodied cognition, and organize how we predict people will perform and react in situations ranging from everyday systems we use, to simulated and fictional experiences both using technology and in more traditional settings. These organizing theories include ideas about the hierarchical predictive nature of cognition, and how brains fundamentally use redeployment of function in order to build new capabilities. In this paper we summarize some of these new ideas in cognitive theories of high-level cognition. Predictive and functional redeployment theories of cognition have implications on how we process and experience presence in virtual environments, how we interact with technological systems in general, and even how and why we immerse ourselves in fictional experiences.

Keywords: Predictive Processing; Functional Redeployment; Neural Reuse; Virtual Reality; Presence; Processing Fluency; Training Of Complex Skills; Situated Learning;
Human beings have recently (evolutionarily speaking) been exposed to a wide range of virtual or fictional experiences, manufactured by our storytellers using technology for many purposes: to teach lessons and skills, to entertain and escape from real life, to motivate people and to impart moral and ethical lessons are a few among many purposes. In this article we are looking at fictional experiences that people participate in, that range from the simple telling of a story using spoken words alone, to plays and novels, up to more recent, immersive technologies for sharing stories and experiences, including virtual and augmented reality. This paper is not intended to propose an all-encompassing cognitive architecture. Instead this paper aims to point out some general underlying processing mechanism, and thus provide a framework for conceptualizing the commonality and differences there are in representations constructed via different media with an emphasis on computer mediated environments. Our brains have an amazing capability to invoke a feeling of being there, and becoming immersed in the virtual worlds, no matter the medium that is used to deliver the experience. This ability to “get into” the story is at the root of how such fictional experiences can affect us emotionally. And in turn, the ability of fictional experiences to engage us is directly responsible for their usefulness in teaching, learning and passing on ideas that we listed as some of the purposes such fictional stories have for humans and society.

Avariety of media have been developed that allow creators to direct and support humans to construct representations of virtual experience. On the one end of the spectrum, humans can immerse themselves in virtual experiences even with a few simple well-chosen words. For example, it is not uncommon to see people shed tears when they read novels and poems after they have become deeply invested in the stories. On the other end of the spectrum, humans have been building ever more realistic technology-based environments for the purpose of triggering brains to feel what is happening in the virtual world is real, using increasingly sophisticated tools and techniques. This begs the question as to how these representational media are used in supporting our cognition and imagination. The more concrete and the less abstract the medium is, the more similar the representations humans are directed to construct [47]. From the viewpoint of information processing, it makes the subsequent construction of the representations more spontaneous and less subject to disruptive discussion. There is less negotiation of what the intended representation is. This indicates that we recruit previous experiences in building representations when experiencing virtual worlds. These media direct and support the construction of the representations, and enhance the flow of virtual experiences.
In this article, we wish to examine the ability of such fictional experiences to immerse us in the story in light of modern theories of embodied cognition. Our goal in this article is to discuss the embodiment theories, and understand the implications for how and why they might enable people to successfully immerse themselves in fictional experiences. Better understanding this relationship should give many beneficial insights into storytelling. In particular, better understanding theories of cognition and how they affect immersion may enable us to more reliably and more strongly create immersive experiences in our fictional storytelling. This is a desirable goal for many reasons, but for us we will show and argue that higher rates of immersion in the story lead to better outcomes when such fictions are used to teach or educate. Thus more fully understanding how our brains become presents in fictional settings help us to better craft such experiences to enhance their beneficial aspects.
Predictive Processing: Hierarchical and Predictive Views of Cognition
Recent theories of embodied cognition have begun to emphasize the hierarchical and predictive nature of cognition [12]. Cognition is hierarchical in nature because it is organized with multiple levels, or layers, with earlier layers typically processing what we would think of as the “raw” or low-level signals from our sensory systems. Or conversely, on the action side of cognition, dealing with the low-level motor command signals sent to our motor muscle systems. Later or “higher” layers appear to be involved with creating and working with ever more abstract embodied representations Fig 1. These represent concepts of the world, and higher-level goals and actions that guide our cognition [20, 21]. Predictive theories of cognition propose that these abstractions are built up using anticipation and prediction to form abstractions, and to test and refine the abstractions against reality.
Figure 1: Functional diagram illustrating principles of hierarchical predictive coding. The top portion represents hierarchical layers across three cortical regions, with the ‘lowest’ layer of the hierarchy on the left, and the highest on the right. Information from senses is projected into the lowest layers. Higher layers send prediction signals to lower layers (blue). Differences between predictions and the current state of the layer are computed, and propagated back up as prediction error signals (red). Top-down projections (dashed lines) regulate the gain and weights of the prediction units, thus allowing predictions to be refined and learned over time. In the lower part of the figure, the curves represent probability distributions over the value of a signal. Inputs, such as from sensory signals, or projections from lower layers, will always have some uncertainty, represented as probability density functions. Input can be more or less noisy. Likewise, predictions are probabilistic, and can be more or less likely given the inputs and past experience [52].
Cognition appears to be fundamentally predictive because within the hierarchical and bidirectional system of signals, so called higher layers appear to encode a probabilistic model of the activities of the lower layers they interact with. In particular, it seems as if such processing is a type of hypothesis testing, in that the higher levels generate predictions of the activity it expects from the lower levels, and differences in these expectations or predictions by the higher-level layer generate error signals [18, 21]. Subsequently the higher levels will adjust and learn in an effort to meet the actual inputs when experiencing similar context in the future.

Predictive models of cognition are diametrically opposed to symbolic cognition. In classical symbolic approaches, information is gathered by perceptual systems, then converted to symbolic information and processed, the results of which are converted into action plans and actions. In predictive cognitive models, the flow of information is fundamentally more interactive and bidirectional. Also, in predictive models, it is assumed that higher layers send prediction signals down to lower layers they are building abstractions from. Then, prediction errors are propagated from these layers back up to the abstraction/prediction layers, signaling the closeness of the fit of the prediction to the real world phenomenon. This monitoring of errors and propagation of error signals in response to the real world, makes predictive models fundamentally enactive in nature [28,7].

Contrary to symbolic views of cognition, the brain appears to be performing a type of Bayesian inference and prediction, as one of its most basic functions. That is to say that the fundamentally probabilistic nature of the encoding that the brain makes appears to be similar to Bayesian statistical inference [44, 45]. In Bayesian inference, expectations, prior experience and context, as well as current perceptual inputs and motor action outputs are continually combined, weighed and interpreted in order to make fundamental judgments about the current state of the world, and guide actions in order to carry out and achieve desired goals. Prior probabilities and expectations, as well as judgments about the current reliability of sensory information are all considered when forming hypothesis about the current situation.

In this probabilistic view of cognition, knowledge of the world is provisional. It always represents our current best hypothesis of the real state of the world, given our current perceptions and evidence, combined with our past experience and expectations. In such a system, pieces of evidence are continually competing and cooperating to form a coherent picture explaining the current situation. In this view, the top-down flow of probabilistic inference (or predictions) is attempting to “explain away” any discrepancy or error (propagated back up through error signals) it notices in the predictive models it makes of the lower-level signals [22,15, 22]. In states of great uncertainty, many such competing hypotheses can live and cooperate and compete, until evidence allows the burden of proof to shift in favor of one view or another. When (as will happen more often in novel situations and hopefully less often in familiar ones) an incorrect conclusion is reached, the error signals can sometimes persist, and drive the system out of the equilibrium it has reached, in search of a better set of predictions that encompass and explain more of the current situation.

Take the phenomenon of perceptual illusions of all types (visual, auditory, proprioceptive), say for example the Necker cube illusion [16, 48]. The perception that the two orientations of the Necker cube coexist simultaneously occurs because there is in actuality no real three dimensional object being perceived. The perceptual cues interact with our past experience of three dimensional objects, but the evidence is equally supportive of two competing orientations of the perceived object. Thus topdown predictive processes, which attempt to relax the system into a (hopefully) correct interpretation of the existing evidence, can never successfully explain away the error signals generated in the predictions. Therefore the system can flip between the two incompatible conclusions, since there is in fact no real ground truth, and both hypotheses are fully compatible with the available evidence. These types of unreal situations, or perceptual illusions, are examples of experiencing fictional realities, and how our brains drive past experience and memory in order to interpret such fictional realities.

The common view of such illusions has been that they provide insights into the errors that our cognitive systems make when perceiving the world. This view is only partially correct. Such illusions give us insights into the types of assumptions and prior probabilities our cognitive systems make when trying to understand perceptual/motor signals. However, under the assumption that brains are Bayesian inference machines, it would be more correct to say that such illusions are the result of an optimum probabilistic judgment, where the prior probabilities and assumptions, which are perfectly reasonable (and optimal) to make in most ordinary circumstances, lead to (consistently reproducible) erroneous conclusions from the data [24, 48].

Predictive views of cognition are important because they allow us to unify and begin to explore common mechanisms that seem to underlie perception, action and attention. In the predictive model, all of these processes are performing the same type of calculation under different contexts. In all these cases, the main task is to reduce prediction error, error that results from active and embodied interactions with the environment. The longer-term goal is to learn environmental structures through the basic mechanism of prediction and correction from the errors of the predictions. In short, the predictive cognitive model is a deeply embodied and enactive view of cognition, and it proposes that we structure our worlds and actions so that our sensory predictions come true [12, 22, 23].

In the context of experiencing virtual realities, hierarchical predictive models of cognition are important tools to structure our understanding of why some narratives or virtual realities work, while others fall apart. Failure of prediction can happen at any level of abstraction, from failures of very abstract goals and motivations, down to failures to predict very fundamental sensory inputs. For example, on the most abstract end, many fictional stories fail to engage audiences when the characters or protagonists act in ways that are not supported by the story. When characters perform actions that don’t seem in line with their character (their goals and motivations), or seem out of place from what we would expect, we break our immersion and engagement with the story. On the other end of the spectrum, very small inconsistencies in visual input can be enough to ruin the illusion of presence in virtual environments. For example, even very small time lags in responding to head movements when rendering a visual scene in an immersive virtual reality will easily break the flow of the experience and cause problems in interacting with the virtual environment [1].
Functional Redeployment
Hierarchical probabilistic views of cognition indicate that functional reuse of cognitive machinery must be the norm, and not the exception, in how our brains deal with the world Fig 2. In functional redeployment, the general processing mechanism being proposed means that reuse and redeployment of circuits would both be expected and would be a common and highly effective method to make (re)use of scarce cognitive resources [3]. Since higher-level layers are simply modeling and predicting an abstraction of the activity of a lower-level (and generating an error signal), it is most economical that other layers reuse an existing layer when dealing with a novel situation or in a new context. This can take the form of a new type of model or prediction being created that reuses an existing lower-level layer. Or it could take the form of lower-level signals being generated
Figure 2: Logical possibilities for the functional organization of brain circuits. a) illustrates a modular functional organization of two tasks in the brain. In a modular organization, we would expect little to no overlap between brain circuits, functions would be largely segregated. b) illustrates a holistic organization. Holistic views of organization propose that functions are not localized at all, all brain areas will be used for all functions. c) illustrates the functional pattern we would expect to see if neural reuse is common. There is high overlap between areas used for the two tasks (reuse), but for each task they cooperate in different patterns with different partners. Network analysis of brain functional connectivity appears to support reuse as the primary functional organization used by the brain [4].
in a new context, one in which they were not originally evolved for. Whenever a neural circuit evolves for one task, but is of use to a later evolving feature, it will be reused or exapted by the new feature. Because of the ubiquitous reuse and conservation of neural circuits, we expect and in fact see patterns of cognitive functions that are best viewed as the result of this sharing of common circuits across differing domains such as perception, action, and language.

This type of reuse and sharing is in some ways similar to the sharing of code for software engineering systems. When code is first created, it is often with a particular purpose in mind. But as the code is used and matures, useful parts of the code are often identified. These are then refectories into libraries and packages. These reusable modules can then serve the original function, but can be reused by new functions and new systems. And this process can continue and accelerate for several levels of complexity (abstraction). Some of the biggest software systems are constructed in this manner, as hierarchical reconstruction and exaptation of small software modules into larger and larger functional systems. It seems reasonable to propose such functional redeployment of the existing cognitive repertoire when users interact with computer mediated and other fictional environments. Such a view of reuse is a significant help in understanding perception and performance in virtual experiences, where there is sometimes overlap with existing neural circuit expectations, and sometimes mismatches in those expectations.

Neural reuse differs from the normal idea of neural plasticity in brain organization and development. Plasticity, as understood, is the idea of neural circuits being able to be repurposed or relearn new functions, if their original function is lost or damaged. For example, a blind person may repurpose neural circuitry that originally evolved for visual processing in order to do more auditory processing of their environment. Neural reuse, on the other hand, is when existing neural circuitry acquire a new use or function that differs from their original purpose. Interesting examples of forced neural reuse are the sensory substitution experiments of Bach-Y-Rita [6]. For example, electrical stimulation of the tongue or chest in a crude visual image can be learned to be perceived as visual images through these unorthodox sensory channels. In this case, an existing neural circuit is used for new functionality. Presumably the higher level abstractions used in recognizing visual objects make new connections (predictions) to the low level tactile sensors being stimulated by the electrical devices, in order to predict object perception from this novel sensory input. In these experiments, the goal is usually to learn to navigate the environment using this novel sensory input. Thus, error signals are propagated to the newly formed predictive connections through attempts to navigate the environment, and identify and not bump into objects being encountered.
There are many examples of reuse of neural circuits in the literature that help shed some light on why understanding overlap or reuse of function is important for theories of experiencing virtual or fictional realities. For example, there are many studies showing overlap of functions in the domain of understanding and using numbers. There is increased activation in the areas used to control the fingers when performing arithmetic operations verbally, such as addition and multiplication [5, 50].

Theories that don’t propose reuse of function as a fundamental concept in cognition are hard pressed to predict such types of activation. However, when we consider how in early learning we use fingers as markers to aid in understanding counting and basic mathematical operations, we can see how such motor circuits may be exapted to become involved in higher level cognitive numerical processes. Relations like these, then, can illustrate some of the ways designed systems and virtual realities can run into problems. When performing tasks that might need fine finger motor control, if mathematical calculations (even of simple arithmetic) are needed, there is likely to be significant interference in the tasks.

In the context of virtual or fictional experiences, the concept of neural reuse obviously has significant implications. Our training through society to be able to be immersed in stories relies on learning to build connections that can be invoked by words, so that we imagine ourselves in the skin of the protagonist. In technological systems, like virtual reality, the connections are even stronger. Designers of virtual reality systems, within the limit of the technology, strive to present visual, auditory and other stimuli that mimic or approximate those we normally receive through our eyes and ears (within the limits of the current state of the technology). Of courses, differences in the abilities and capabilities of the technology to match real life put a limit on what can be achieved, and thus we are forced to consider what effects can be achieved in immersing the person in the experience, given lower than real life capabilities for the visual, auditory or other input stimuli that we can present.

Neural reuse has important implications for interference that may occur when designing human-computer interfaces or in general when using any system in which we want to minimize cognitive overload. Information about the overlap of neural functions is an important tool that must be kept in mind by the designers of systems with which people will interact. Why some types of information can be delivered with ease simultaneously, while others will invariably cause attention to be split and interfere with one another are often consequences of the reuse of cognitive circuits being utilized for different types of tasks. This reuse and interference is also a possible explanation for why immersion sometimes fails for virtual environments for given technologies, but is much better in other instances. Increasing attention to the overlap and reuse of neural circuits is an important component for cognitive models if we wish to apply them to building less cognitively taxing and more immersive technological systems [30].
Virtual Environments and Feelings of Presence
Researchers in the virtual environments and telepresence community have developed and tested various theories of VR and how users experience and interact with such virtual systems. Current theories tend to split user experiences into two parts. Immersion refers to a system’s technical capability in engaging the users’ attention in the virtual experience while presence refers to the users’ feeling of their bodies being transported into the virtual space [31, 32]. Presence, as a state of conscious experience, is multidimensional, including spatial and social aspects, among others. Earlier work in the VR community focused on spatial presence. As the technical capability of virtual environments advances, instead of looking at various categories of presence, some researchers proposed the distinction between place illusion (i.e., spatial presence) and plausibility illusion (i.e., treating events as if they are real) [37]. To experience presence, users need to take on a virtual body, perform actions from the first person perspective in the virtual environment, and be provided with the sensorimotor contingency of their actions.

A variety of measures have been used in determining the level of presence experienced by users in a virtual experience: (a) the subjective reports of users’ conscious reflections of their experiences in the virtual environments; (b) the behavioral measure of user responses to the events, and (c) physiological measurements of users’ bodily reactions to the experience. Presence is perceived to be a multilevel representation, from unconscious arousal to conscious volitional responses [32]. The more that measure such as those just mentioned are consistent at different levels and point in the same direction, the greater the presence that will be experienced by the user of a virtual environment. For example the Milgram obedience study, where people would agree to deliver a lethal electric shock to strangers upon the request of an authority figure, was replicated in a virtual environment setting [38, 39]. As in the original study, participants administered word association learning and memory tasks to a virtual learner in the virtual environment reprisal of the study. Some participants saw and heard the avatar’s protestations of pain, whereas others communicated with the virtual learner through a text interface only. As the virtual learner made mistakes, the voltage of electric shocks progressively went higher and higher. In both conditions, participants delivered the highest voltage (450 volts) to the virtual learner, an amount that would kill a real human. More participants did so in the text only interface condition than the virtual learner visible condition. A significantly greater proportion of participants in the learner visible condition reported having the thought of terminating the study than the text only condition.

In this virtual reprisal of the Milgram study, the skin conductance level (i.e., overall sympathetic arousal) and skin conductance responses (i.e., transient arousal) were measured and were significantly different between the two conditions. Even though it was not necessary to wait after hearing the learner’s wrong answers and delivering the shock to the learner, participants in the learner visible condition waited significantly longer. Participants in the learner visible condition also reported greater self-awareness of physiological responses (e.g. ‘trembling or shaking’, ‘face becoming hot’). Knowing that the virtual learner cannot be truly harmed, people acted as if their actions had exerted effects on another human. The visibility of the virtual learner’s pain and frustration was very important for triggering the plausibility illusion.

Place illusion, the feeling of being transported to a location when experiencing a virtual reality, was found to be highly reliant on natural sensorimotor contingencies delivered through head tracking. Plausibility illusion, the feeling that what appears to be happening is really happening, depends on the correlation of self-actions with the expected events that should co-occur in the virtual environment. The user possessing a virtual body is critical to both types of presence experiences. A direct comparison of the impact of immersion on presence was reported in a study where users were first placed in the highest level immersive system and progressively transitioned to a slightly lower immersive system [42]. By changing a single parameter at a time, such as the virtual body possessed by the user, the study showed that plausibility is critically affected by such co-occurrence relations. Even though the issue of how plausibility contrasts with spatial presence remains to be fleshed out, subsequent studies appear to converge on finding that the physical basis of presence depends on: (a) the user having a virtual body; and (b) the user experiencing appropriate sensorimotor contingencies. Slater and colleagues subsequently conducted various studies testing this sense of embodiment [38] and found that embodiment in a virtual environment needs both strong place and plausibility illusion in order to occur. Some other group focused on the virtual world’s simulation training. The most familiar examples of these, both in current and past systems, include simulations of real world tasks, such as flight simulators or surgical training [35].

There have been many studies performed that link more immersive experiences to better learning and transfer of skills. Immersion in a virtual environment allows people to experience situations from multiple perspectives. A virtual experience allows one to learn skills or lessons in a situated learning or context dependent manner, which has been shown to be one of the most effective ways of gaining experience. We learn best when we practice things in the actual context in which we will have to apply them. Further, virtual environments allow for the transfer of learned skills and lessons, because we can experience things in a relatively safe way but still have the situated context that improves retention and transfer. All of these ways of enhancing the learning process have been shown to be positively tied to the level of immersion of the virtual experience [13].

Compared with verbal learning, situated learning capitalizes on the developed neural circuitry more fully. It is a powerful pedagogical technique, but one that is not widely used in the classroom. In the physical sciences, laboratory sections of courses are examples of situated learning, where experts in procedural techniques lead students through experiments and procedures with well-known results but that pedagogically demonstrate lessons about the subject while simultaneously exposing students to techniques needed to carry out and understand such results. But in general it is well accepted that situated learning provides many benefits, and is a very effective techniques for learning and transfer of skills [2]. In many cases, a controlled situated learning experience (like a course laboratory section) is not possible, economically or because of safety reasons. Thus virtual or computer mediated environments have been explored, to address both of these points and to provide a simulation of a situated experience. The benefits of virtual environment for transfer and serious purposes have been well shown and are familiar for task learning situations, such as flight simulators and driving simulators. In the case of these complex motor skill task simulators, the big benefit is that the learned skills can be successfully transferred from the simulation to the real world tasks. This has been successfully done for many such psychomotor digital skill simulators. But, even for more abstract types of tasks, immersive environments with some sort of situated context have been shown to provide some level of transfer to the real world to some degree [33].

So far we have reviewed some of the important concepts in computer simulated environments, and how presence and immersion in these environments can vary and affect various aspects of performance and training. This raises the question of how embodied views of cognition, especially in light of hierarchal predictive models and neural reuse perspectives of cognition, affect and inform ideas of performance in fictional environments and realities. In the next section, we look at prediction error and processing fluency concepts, and relate these to experiences in virtual environments.
Prediction Error, Processing Fluency and Presence
In the hierarchical predictive view of cognition, upper levels of the neural circuitry have the goal of reducing prediction error on the generated error signals. This can be accomplished through several mechanisms, such as refining the models with more specialized cases, or generating new classes of models, if necessary, to capture context or concepts the system can now infer given its existing history of inferential machinery. The success of the predictive experiences can be viewed on a spectrum. On the one end, we have a model that achieves perfect prediction, where all inputs are explained and no error signals generated. On the other end of the spectrum, we have no model whatsoever of what we are experiencing, and thus cannot explain or predict any of the information being received.

Learning theory says that there is an optimal location on this spectrum in which we can maximally learn new information (or maximally learn and build better inferential predictive models) [27]. This zone for optimum learning (Zone of Proximal Development, see a review in [10, 46] occurs when the error signals from our predictions are somewhere in the middle of the spectrum. When we do well at predicting and performing, there is little new information being generated and little opportunity for us to learn new things. Learners in such a situation are not being challenged by the task, and thus are prone to become bored, and possibly let their mind wander. At the other end of the spectrum, if we are too far over our head in terms of the task difficulty, we simply do not have enough predictive machinery in place in order to successfully model and make new predictive inferences from what we are experiencing. Only when the error signal is high enough, but not too high, are we being significantly challenged such that we can maximize our learning gains, but not be overwhelmed with too much new information that needs to be integrated.

Hierarchical predictive learning systems must have as a fundamental motivation the desire to try and minimize the prediction errors being experienced. But this fundamental motivation must also be balanced by other goals that seek to maintain the health and well-being of the organism. A sensory deprivation environment would provide an ideal situation to achieve zero prediction errors, but organisms that only seek this type of error minimization would not be able to maintain living and other bodily and reproductive needs for long. At the other end of the spectrum, we can imagine experiencing almost perfect prediction (zero prediction errors) but while performing a very complex and important task. In such a state, the processing fluency is very high. World class athletes and performers are most often thought of as examples of such high fluency in task performance, though almost all of us who attain some level of expertise in a field can and do experience this state of almost perfect cognitive functioning and very low prediction errors in performing tasks from time to time

In the context of using virtual environments or fictional training scenarios for serious purposes, our goal is to better understand how the predictive machinery is harnessed in order to reuse existing cognitive machinery in order to learn from the experience. Certain guidelines thus become apparent in the context of modern theories of embodied cognition on how we can improve the sense of presence and immersion in such virtual experiences to thus increase engagement and interaction within them. We propose the following ideas or guidelines in this context:
• Users naturally and automatically lower their expectations at the onset of interactions with a virtual or computer mediated environment.

• In the process of interacting with a simulated environment, mismatches between the expected and actual signals produce prediction errors that lead to jagged processing fluency.

• Processing fluency is not presence, but presence entails processing fluency.

First, we propose that users automatically lower their expectations at the onset of interactions with virtual environments or other less realistic media. This reduces the prediction errors and enhances the processing fluency. This adjustment explains why stimuli with minimum realism can trigger the experience of immersion. Second, in the process of interaction, mismatches between expected and actual signals produce prediction errors and lead to jagged processing fluency. When the jagged processing fluency reaches a certain point, the experience of presence will be lowered or breaks down. These prediction errors are usually more of an issue with low-level perceptual/ motor cues that can sometimes be improved by increasing the fidelity of the virtual environment we are using for the training. But improving the technology is not always reasonable, or even possible, so we might want to influence top-down processes to maintain or enhance the processing fluency.
Third, processing fluency is not presence, but presence entails processing fluency. Presence in a virtual environment is achieved when the user forgets the unreality of the situation, and feels as if they are experiencing a real event. People can experience processing fluency but still not forget the unreality of the situation they are experiencing. For example, people can be in a flow while playing with a simple and fun game. When we control the range of the prediction errors of our cognitive machinery, and thus are performing the task more naturally, without having to consciously think about the details of how we are performing the task and interacting with the environment, then we are in a state of flow. Training and engagement in computer simulated environments will need to be able to reliably reach such levels, if they are to be successful in achieving their learning goals.
How Predictive Processing Fluency Effects Experiences in Virtual Environments
Many of our existing cognitive circuits that encode prior assumptions of experiences and events are retooled and redeployed when we experience a simulated or fictional experience of many different kinds, from simple story telling to experiences in virtual reality simulations. The prior assumptions our cognitive systems make about our actions and perceptions should be greatly affected by the semantic, visual and proprioceptive properties of objects we are holding. Consider the effectors we use inside of a VR experience in order to control our interactions in the environment. For example, the Microsoft Kinect (TM) interface allows for users to interact with a computer mediated environment using hand, arm and leg gestures. Users can interact with the environment when their hand is empty, or they can instead interact while holding a tool, such as a screwdriver or knife. If the user is performing tasks within the computer mediated environment that are “screwdriver like” or “knife like”, the different controllers cue up somewhat different expectations about the subsequent processing hierarchy and lead to differences in predictions from the top down layers. For example, if there is an inconsistency in the tool being held (a knife) and the task it is being used for (hammer a nail) then the signals generated by the low-level sensory motor system will yield more prediction mismatches. Thus predictions will yield mismatches, which will generate error signals bi-directionally from the sensory motor systems. When such incongruence exists between the user’s controller and the tool the avatar representing the user in the VR appears to be using, greater adjustments for the next predicted perceptual cue are needed in the processing hierarchy, which can effect or break the feeling of immersion in the task.

There is evidence for such functional retooling and redeployment. Experiences of users in virtual environments are known to be very sensitive to temporal time lags [1, 26, 36]. In fact, even with a very high definition system, unacceptable time lag of feedback signals in response to head tracking or arm/ limb positioning are known to immediately throw the user out of feeling immersed in the experience [26]. And vice-versa, even low-fidelity environments can be highly immersive, when they give immediate responses to actions, with no perceivable lag in the system responding to the users actions. Such sensitivity to temporal consistency and inconsistency with expected results should not be surprising given a predictive and enactive view of cognition.

More evidence for such retooling and redeployment comes from the rubber hand illusion (RHI). By simultaneously tapping and stroking a rubber hand and a person’s hidden real hand, the person experiences the illusion as if the rubber hand were his / her own [8]. This illusion has been demonstrated with a virtual arm [41] and a whole manikin body viewed from the first-person perspective [29]. The setup for such illusions does not occur naturally, so the cognitive machinery has no prior probabilities that would allow for such artificial manipulations. The urge of processing fluency is so powerful that we retool and redeploy our body representations even with the conscious awareness of what is actually happening. This retooling and redeployment effect has been extended to the marble-hand illusion [34]. A person’s hand was gently hammered, and gradually the sound of a hammer hitting a piece of marble was synchronized to co-occur with the hammer taps. Five minutes later, the person began feeling a stiffening of their hand, as shown by Galvanic skin responses.

The simplest and most obvious predictions to develop are those where motor actions should result in immediate effects. These tightly-coupled perception/action immediate cause and effect predictions are important building blocks in our cognitive machinery. Because of the relative ease with which such predictions can be developed, we would expect them to occur relatively low down, at lower levels, in the hierarchy of models and predictions. Thus, failures at these levels have many more opportunities to disrupt higher-level conceptual models that are using them to maintain coherent predictions. That is to say, we would expect that when prediction errors occur at more fundamental levels of the hierarchy, they will possibly disrupt many higher-level theories or conceptual frameworks the brain might be trying to maintain. Lower-level prediction circuits would be expected to be making more fundamental predictions, and to be used by many more high-level conceptual frameworks for constructing reality. When errors at these lower levels cannot be mitigated by the machinery, the result is a quick breakdown of being able to work within the virtual environment, and maintain feelings of presence with what is being experienced. For example, in the context of virtual environments, it can be as important to induce motor movements that are compatible with expectations of the user when performing a task, as it is to produce realistic visual inputs. Motor movements are a very powerful trigger (or disruptor) of feelings of presence in experiencing a virtual environment [49].
We discussed how the hierarchical predictive coding theory can be used as an overarching cognitive processing framework for understanding the minds of virtual media users. Given that humans weave their previous experiences into their virtual interactions, the brain as the hypothesis generation machine would deploy and adapt if necessary the neural circuits that are functionally closest to the ongoing virtual worlds. The design of a natural and productive human computer interaction lies not only at the alignment of the virtual and real worlds but also at the invoking of the relevant personal history.

Many other factors can influence how involved we get in a virtual experience. Our motivation, valence systems, involvement, personal history and personal differences can all factor into the experience. Top-down processes drive the cognitive experience to a coherent and meaningful interaction with the task environments we are consuming. When we are successfully immersed in the experience, the top down predictive models successfully suppress, ignore or cordon off inconsistent or contradictory information that can threaten to take us out of the experience. Conversely, the top-down processes can also work in the other direction, in order to enhance and detect such inconsistencies, thus allowing us to pick apart the experience and detect the unrealistic aspects of the computer mediated or fictional environment.
Given this understanding of how internally our goals and personal differences can influence the immersive experience, and given that immersion is a desirable quality to achieve for many reasons, we should conclude that we need to take such factors into consideration when designing a virtual experience, in order to maximize the effectiveness of the scenario we wish to create. The better designs and devices are at cueing up users’ personally, hedonically or motivationally marked experiential basis, the more natural and productive the interaction experiences are. The better designs and devices are at aligning users’ mental states, the greater audience potential the design will enjoy.” completely here.
  1. Allison RS, Harris LR, Jenkin M, Jasiobedzka U,  Zacher JE. Tolerance of temporal delay in virtual environments. Proceedings of the Virtual Reality 2001 Conference (VR'01). 2001;247.
  2. Anderson JR, Reder LM, HA Simon. Situated learning and education. Educational Researcher. 1996;25(4):5-11.
  3. Anderson ML. Neural reuse: A fundamental organizational principle of the brain. Behavioral and Brain Sciences. 2010;33(4):245-266. doi:10.1017/S0140525X10000853.
  4. Anderson ML, Penner-Wilger M. Neural reuse in the evolution and development of the brain: Evidence for developmental homology? Deve lopmental psychobiology. 2013;55(1):42-51.  doi:10.1002/dev.21055.
  5. Andres M, Seron X, Oliver E. Contribution of hand motor circuits to counting. Journal of Cognitive Neuroscience. 2007;19(4):563-576. doi:10.1162/jocn.2007.19.4.563.
  6. Bach-y-Rita P, Kercel SW. Sensory substitution and the human-machine interface. Trends in Cognitive Science. 2003;7(12):541-546.
  7. Beer RD. Dynamical approaches to cognitive science. Trends in Cognitive Science. 2000;4(3):91-99. doi:10.1016/S1364-6613(99)01440-0.
  8. Botvinick M, Cohen J. Rubber hands 'feel' touch that eyes see. Nature. 1998;391:567. doi:10.1038/35784.
  9. Bowman DA, McMahan RP. Virtual reality: How much immersion is enough? Computer. 2007;40(7):36-43.
  10. Chaiklin S. The zone of proximal development in Vygotsky's analysis of learning and instruction. In Vygotsky's Eductional Theory in Cultural Context. 2003;39-64. Doi:10.1017CBO9780511840975.
  11. Chisholm JD, Risko EF, Kingstone A. From gestures to gaming: Visible embodiment of remote actions. Quarterly Journal of Experimental Psychology. 2013;67(3):609-624. doi:10.1080/17470218.2013.823454.
  12. Clark A.  Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences. 2013;33(4):181-204. doi:10.1017/S0140525X12000477.
  13. Dede C. Immersive interfaces for engagement and learning. Science. 2009;323(5910):66-69. doi:10.1126/science.1167311.
  14. Draper JV, Kaber DB, Usher JM. Tele presence. Human Factors. 1998;40(3):354-375. Doi:10.1518/001872098779591386.
  15. Friston K. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 2010;11:127-138.  doi:10.1038/nrn2787
  16. Geisler WS, Kersten D. Illusions, perception and Bayes. Nature Neuroscience. 2002;5(6):508-510. doi:10.1038/nn0602-508
  17. Giese MA, Rizzolatti G. Neural and computational mechanisms of action processing: Interaction between visual and motor representations. Neuron. 2015 ;88(1);167-180. doi:10.1016j.neuron.2015.09.040
  18. Gregory RL. Perceptions as hypotheses. Philosophical Transactions of the Royal Society of London B. 1980;290(1038):181-197.
  19. Hendrix C, Barfield W. Presence within virtual environments as a function of visual display parameters. Presence-Teleoperators and Virtual Environments. 1996;5(3):274-289. doi:10.1162/pres.1996.5.3.274
  20. Hohwy J. Functional integration and the mind. Synthese. 2007;159(3):315-328.
  21. Hohwy J. Attention and conscious perception in the hypothesis testing brain. Frontiers in Psychology. 2012;3(96):1-14. doi:10.3389/fpsyg.2012.00096.
  22. Huang Y, Rao PNR. Predictive coding. Wiley Interdisciplinary Reviews: Cognitive Neuroscience. 2011;2:580-593. DOI:10.1002/wcs.142.
  23. Jehee JF, Ballard DH. Predictive feedback can account for biphasic responses in the lateral geniculate nucleus. PLoS (Public Library of Science) Computational Biology. 2009;5(5):e1000373. doi:10.1371/journal.pcbi.1000373.
  24. Knill DC, Pouget A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends in Neuroscience. 2004;27(12):712-719. doi:10.1016/j.tins.2004.10.007.
  25. Lu S, Rawlinson T, Harter D. Embodiment in low cost virtual environments: The role of working memory in experiencing presence as revealed via eye tracking.
  26. Mania K, Adelstein BD, Ellis SR, Hill MI. Perceptual sensitivity to head tracking latency in virtual environments with varying degrees of scene complexity. Proceedings of the 1st Symposium on Applied Perception in Graphics and Visualization. 2004;39-47. ACM. Doi:10.1145/1012551.1012559.
  27. Miller HE, Schumann PL, Anderson PH, Scott TW. Maximizing learning gains in simulations: Lessons from the training literature. Developments in Business Simulation and Experiential Learning. 1998;25-32.
  28. Noë A. Action in Perception. MIT Press. 2004.
  29. Petkova VI, Khoshnevis M, Ehrsson HH. The perspective matters! Multisensory integration in ego-centric reference frames determines full-body ownership. Frontiers in Psychology. 2011;2:35. doi: 10.3389/fpsyg.2011.00035.
  30. Ritter FE, Young RM. Using cognitive models to improve interface design. International Journal of Human-Computer Studies.  2001;55(1):1-107.
  31. Riva G, Mantovani F. From the body to the tools and back: A general framework for presence in mediated interactions. Interacting With Computers. 2012;24(4):203-210. doi: 10.1016/j.intcom.2012.04.007
  32. Sanchez-Vives MV, Slater M. From presence to consciousness through virtual reality. Nature Review Neuroscience. 2005;6(4):332-339. DOI:10.1038/nrn1651
  33. Schwartz DL Sears, D, & Bransford, J. D. Transfer of learning from a modern multidisciplinary perspective. J Mestre, Ed. Greenwich, CT: Information Age. 2005.   
  34. Senna I, Maravita A, Bolognini N, Parise CV. The marble-hand illusion. PLoS One. 2014;9(3):e91688. doi:10.1371/journal.pone.0091688.
  35. Seymour, N. VR to OR: A review of the evidence that virtual reality simulation improves operating room performance. World Journal of Surgery. 2008;32(2):182-188.
  36. Sheridan TB. Musings on telepresence and virtual presence. Presence: Teleoperators and Virtual Environments. 1992;1(1):120-126.
  37. Slater M. Place illusion and plausibility can lead to realistic behavior in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences. 2009;364(1535):3549–3557.  doi:10.1098/rstb.2009.0138
  38. Slater M, Sanchez-Vives MV.  Transcending the self in immersive virtual reality. Computer. 2014;47(7):24-30. doi:10.1109/MC.2014.198
  39. Slater M, Antley A, Davison A, Swapp D, Guger C, Barker C, et al. A virtual reprise of the Stanley Milgram Obedience Experiments. PLoS One. 2006;1:e39. DOI:10.1371/journal.pone.0000039
  40. Slater M, Khanna P, Mortensen J, Insu Y. Visual realism enhances realistic response in an immersive virtual environment. IEEE Computer Graphics and Applications. 2009;29(3):76-84. Doi:10.1109/MCG.2009.55
  41. Slater M, Perez-Marcos D, Ehrsson HH, Sanchez-Vives MV. Towards a digital body: The virtual arm illusion. Frontiers in Human Neuroscience. 2008;2:6. doi:10.3389/neuro.09.006.2008.
  42. Slater M, Spanlang B, Corominas D. Simulating virtual environments within virtual environments as the basis for a psychophysics of presence. ACM Transactions on Graphics (TOG). 2010;29(4):92. Doi:10.1145/1778765.1778829
  43. Sowndarajan A, Wang R,  Bowman DA. Quantifying the benefits of immersion for procedural training. Proceedings of the 2008 Workshop on Immersive Projection Technologies / Emerging Display Technologies  ACM. 2008;2. Doi:10.1145/1394669.1394672
  44. Todorov E. Parallels between sensory and motor information processing. In M. Gazaniga Ed. The Cognitive Neurosciences 4th ed. MIT Press. 2009;613-624.
  45. Toussaint M. Probabilistic inferences as a model of planned behavior. Künstliche Intelligenze. 2009;3:23-29.
  46. Vygotsky L. Mind in society: The development of higher psychological processes. Harvard university press. 1980.
  47. Walton KL.  Mimesis as make-believe: On the foundations of the representational arts. Harvard University Press 1990.
  48. Weiss Y, Simoncelli EP,  Adelson EH. Motion illusions as optimal percepts. Nature Neuroscience. 2002;5(6):598-604. DOI:10.1038/nn858
  49. Won AS, Bailenson J, Lee J, Lanier J. Homuncular flexibility in virtual reality. Journal of Computer-Mediated Communcation. 2015;20(3):241-259. DOI:10.1111/jcc4.12107
  50. Zago L, Pesenti M, Mellet E, Crivello F, Mazoyer B, Tzourio-Mazoyer N. Neural correlates of simple and complex mental calculation. NeuroImage. 2001;13(2):314-327. DOI:10.1006/nimg.2000.0697
  51. Zimmons P, Panter A. The influence of rendering quality on presence and task performance in a virtual environment in virtual reality. Proceedings of the IEEE Virtual Reality Conference (VR 03). 2003;293-294.
  52. Seth AK. Interoceptive inference, emotion, and the embodied self. Trends in cognitive sciences. 2013;17(11):565-573. doi: 10.1016/j. tics.2013.09.007"
Listing : ICMJE   

Creative Commons License Open Access by Symbiosis is licensed under a Creative Commons Attribution 4.0 Unported License