A quick intro:
David Petrescu is a final year PhD student, working on perceptually-driven computer graphics published at high-profile conferences (Eurographics). David has an interdisciplinary background in computer science, psychology and neuroscience. David has worked on the design on a low-cost Phantom Limb Pain mirror therapy using accessible VR technologies (Oculus Go). He is also an advisor on research projects about how XR can be potentially harmful to children.
Contact details:
LinkedIn Account: https://www.linkedin.com/in/dg-petrescu/
Virtual rehabilitation for physical and psychological conditions has been proven to be a fun, engaging and effective method for every age group. With the commercialisation of the VR headsets, acquiring a VR headset for therapy has become an affordable option. It would often be the case that treatment services who deliver rehabilitation therapies (such as occupational or physical therapy) would also offer virtual rehabilitation, either at the clinical setting or at the patient’s home setting. In these scenarios, the patients would be immersed in virtual, daily environments and would be expected to complete certain physical exercises or cognitive tasks, depending on their physical condition. However, little is known about how the graphics’ quality of the VR application could affect the success of the rehabilitation therapy and whether this would depend on the task undertaken virtually. Ultimately, considering this topic would inform the design and development of relevant VR applications and ensure their effectiveness clinically.
So, we asked David about his views on how the quality of graphics in VR can affect the rehabilitation effectiveness of such applications.
- In your view, would you say that when loss in graphics’ quality occurs (degradations), this could affect the rehabilitation effectiveness? Why/Why not?
When we think of graphics and rendering, we need to account for all the technologies that are part of a Virtual Experience. These include virtual physics, multi-user interaction in which keeping data latency low is crucial, and most importantly maintaining a photorealistic appearance. Most of VR graphics are relatively cartoonish because of the increased computational needs typical to VR, and whilst hardware solutions enable us to unlock photorealism, there are other limiting factors. If the framerate is not constantly high (traditionally above 90 frames per second), cybersickness becomes a major concern which can drive people away from VR usage. This is especially important for people undergoing various forms of therapy, as we want to keep the experience as enjoyable as possible. The perceptual graphics community aims to keep the perceived experience intact whilst basically hiding degradations in plain sight (without the user noticing). Therefore, in rehabilitation applications, resources could be redirected to critical aspects of the application.
- Are there any tasks, potentially relevant to physical/cognitive rehabilitation, that would be heavily dependent on the graphics quality?
Definitely! Research has clearly shown that avatar realism, for example, influences the experienced stress of people suffering from anxiety regarding job interviews, or other forms of social anxieties. Mel Slater’s group in Barcelona was very active in this field. They showed that realism affects virtual “presence” which in turns affects the effectiveness of interventions. However, other studies show that in exposure therapies of acrophobia, realism is only causing a significant effect on people that score lower on anxiety scales. What is certain is that there is compelling evidence out there that assures us that rendering quality does matter.
- What would virtual rehabilitation researchers and VR developers in such teams would need to consider during the development of such systems for physical/cognitive rehabilitation purposes regarding the tech and graphics?
I think what the industry currently lacks are platforms where clinicians can develop tailored intervention for their patients without the need of a developer team. Previously, interventions were developed using game engines for VR applications (e.g. Half-life: Alyx). Whilst this resulted in satisfactory rendering quality, those applications are made for games and the developers did not consider the potential use in medical VR. However, there is a push towards developing frameworks that allow clinicians and other medical professionals to easily develop their own environments without too much Computer Science experience. ORamaVR is one of the companies that achieved this by developing an SDK for Unity which enables easy creation of VR medical training scenarios. However, I think it is important to mention that there is currently a push in the graphics community towards achieving this more widely – when I attended SIGGRAPH this year, the biggest Computer Graphics conference with more than 14000 people from both industry and Academia, extensive workshops were run about medical XR, potential applications and limitations.
- What would virtual rehabilitation researchers and VR developers in such teams would need to consider during the development of such systems for physical/cognitive rehabilitation purposes regarding the content (VR environments) and tasks (pace, walking, etc.)?
I believe that VR is definitely moving away from the at-home, stationary paradigm. We see motion platforms gaining popularity and research about redirected walking becoming more prevalent (in which perceptual mechanisms and illusion are used to effectively make the user perceive that they are walking in a straight line, when in reality they are going around in circles). There is strong evidence that the neurological processes that are involved in active movement (participant moving under their own steam) is different than observing the equivalent motion in VR. When I think of a VR experience I don’t want to experience the same thing I would do with a normal display and moving is characteristic of immersive virtual environments. Therefore, I believe that developers should account for 1) for dynamic environments, therefore accounting for participant movement if possible 2) varying levels of realism based on the platform the intervention is delivered on (a lot of early investments in VR means that medical facilities generally have outdated HMDs which cannot support photorealistic graphics); 3) adopting the usage of perceptual models which provide selective rendering based on tasks.
- Would you say that the hardware itself (considering available, modern VR headsets only) would determine the effectiveness of virtual rehabilitation based on the graphics quality it can offer? Is there a golden option?
Some studies show that the physiological response participants experience (heartbeat or galvanic skin response) is the same regardless of graphics quality if the task participants are undertaking is designed to induce anxiety for example. But as I mentioned previously, evidence shows that presence and therapy effectiveness can be influenced by rendering quality. New headsets are now equipped with eye-tracking devices. Recording gaze patterns in an exposure therapy task might help the therapist better understand what components of the scene generate anxiety or worsen PTSD symptoms. I imagine that analysing the relation between physiological response and visual behaviour in VR in such an intervention could provide insightful information. Moreover, Meta demoed HMDs with almost retinal resolution which are also varifocal. This fixes the vergence-accomodotation conflict and discomfort that comes with it, improves graphics and could definitely help improve presence. We see NVIDIA and Meta, Unreal and Google releasing AI technologies that allow for extremely fast simulations of virtual humans and acquisition of geometry from real environments. All of these could lead to a world in which generating individualised scenarios becomes almost trivial. The golden option, in my opinion, is for developers to provide the tools necessary for clinicians to develop required scenarios for themselves (e.g. mirror therapy, exposure therapy for war veterans etc.). This should be then scaled to various technologies with different computational abilities, but also adapt generative AI techniques. I believe that if all of this is achieved, future clinicians could potentially just describe what they want to an application that generates the medical scenario required – but this cannot happen just yet.
- Are there ways to measure the effect of graphics’ quality in the clinical effectiveness of rehabilitation systems at the early, design phases and determine how to make them better?
The reason why this is difficult to achieve is because of how many things can affect graphics quality. It can be geometrical complexity, sampling, resolution etc. All of these will then be modulated by the environments, textures of objects, real-time rendering needs, multi-user interaction etc. If we want to study how certain quality changes affect a certain type of intervention, it will create a combinatorial explosion of conditions. Therefore, I believe some knowledge is required about what rendering aspect is critical in a certain type of intervention (this could be realistic avatars, realistic spiders, the fidelity of limb movements in mirror therapy etc.). Psychological recordings, presence questionnaires and longitudinal studies are needed for such a task. Moreover, I believe psychophysical testing should be done in order to obtain rigorous answers to these questions. The issue is this is extremely expensive, and some community effort is needed for a comprehensive classification. I am not aware of any taxonomy that aims to classify what degradations are tolerable for specific interventions and what objects need to be realistic. A lot of effort in the perceptual graphics community revolves around creating perceptual metrics which are inspired by biological models. Basically these metrics are abstract values that show when degradation become visible to the human eye. These metrics can also be used to automatically direct rendering within computational budgets. I think these solutions are ideal before we have sufficient data to be able to answer this question.
After this “A quick look on…” VR and graphics during virtual rehabilitation, we thank David for his time and expertise and we hope to see great advances in the development of photorealistic environments for clinical purposes.
