The immersive engagement of a user with the program in the real world/digital world cross-over of mixed reality is referred to as presence. UMass Amherst researchers are the first to identify reaction time as a potential presence measurement tool. Their findings, which were published in IEEE Transactions on Visualization and Computer Graphics, have implications for calibrating mixed reality to the user in real-time.
“In virtual reality, the user is in the virtual world; they have no connection with the physical world around them,” says Fatima Anwar, an assistant professor of electrical and computer engineering and one of the paper’s authors. “Mixed reality is a combination of both: You can see your physical world, but then on top of that, you have that spatially related information that is virtual.” As an example, she attaches a virtual keyboard to a physical table. This is similar to augmented reality, but it goes a step further by making the digital elements more interactive with the user and their surroundings.
The most obvious applications for mixed reality are in gaming, but Anwar claims that it is rapidly expanding into other fields such as academia, industry, construction, and healthcare. However, the quality of mixed reality experiences varies: “Does the user perceive themselves to be present in that environment?” How engrossing do they feel? “How does that affect their interactions with the environment?” Anwar wonders. This is referred to as “presence.”
If we just show the organ in front of them, and we don’t adjust for the height of the surgeon, for instance, that could delay the surgeon and could have inaccuracies for them.
Yasra Chandio
Previously, after a user exited a mixed-reality program, presence was measured using subjective questionnaires. Unfortunately, when measuring presence after the fact, it’s difficult to capture a user’s feelings about the entire experience, especially in long exposure scenes. (People are also not very articulate in describing their emotions, making them an untrustworthy data source.) The ultimate goal is to have an instantaneous measure of presence so that the mixed reality simulation can be adjusted for optimal presence in the moment. “Oh, their presence is dwindling. “Let’s stage an intervention,” Anwar suggests.
Yasra Chandio, doctoral candidate in computer engineering and lead study author, gives medical procedures as an example of the importance of this real-time presence calibration: If a surgeon needs millimeter-level precision, they may use mixed reality as a guide to tell them exactly where they need to operate.
“If we just show the organ in front of them, and we don’t adjust for the height of the surgeon, for instance, that could be delaying the surgeon and could have inaccuracies for them,” she said. Low presence can also contribute to cybersickness, a dizzy or nauseating sensation that occurs in the body when a user’s bodily perception does not match what they’re seeing. If the mixed reality system is internally monitoring presence, it can make real-time adjustments, such as moving the virtual organ rendering closer to eye level.
Reaction time, or how quickly a user interacts with virtual elements, is one marker within the mixed reality that can be measured continuously and passively. The researchers discovered that reaction time is associated with presence in such a way that slow reaction time indicates low presence and fast reaction time indicates high presence with 80% predictive accuracy even with a small dataset.
To put this to the test, the researchers placed participants in modified “Fruit Ninja” mixed reality scenarios (without the scoring), varying the authenticity of the digital elements to manipulate presence.
Presence is made up of two components: place illusion and plausibility illusion. “First of all, virtual objects should look real,” Anwar says. That is a case of spatial illusion. “The objects should look at how physical things appear, and the second thing is: are they acting realistically?” Do they obey physical laws when they behave in the real world?” This is the illusion of plausibility.
In one experiment, they altered the illusion of place, and the fruit appeared as either lifelike fruit or abstract cartoons. In another experiment, they altered the plausibility illusion by showing mugs filled with coffee either upright or sideways.
What they found: People were quicker in reacting to the lifelike fruit than they would to the cartoonish-looking food. And the same thing happened in the plausibility and implausible behavior of the coffee mug.
Reaction time is an excellent indicator of presence because it indicates whether the virtual elements are a tool or a distraction. “If a person is not feeling present, they would be looking into that environment and figuring out things,” Chandio said. “Their cognition in perception is focused on something other than the task at hand because they are figuring out what is going on.”
“Some people are going to argue, ‘Why would you not create the best scene in the first place?’ but that’s because humans are very complex,” Chandio said. “What works for me may not work for you, and what works for you may not work for Fatima, because we have different bodies, our hands move differently, and we perceive the world in different ways. We see things differently.”
