Mixed reality as 3D medical imaging to improve neurosurgical care

Abstract

Mixed reality (MR) is a visualization technique where virtual 3D images are projected into the real world using MR head-mounted displays (MR-HMDs). These images are anchored in space and can be used as if they were physical objects. This eliminates the need for neurosurgeons to mentally transform 2D MRI or CT images into the actual 3D anatomy of the patient. In this thesis, we investigated the added value of mixed reality in various aspects of neurosurgical care. Chapters 2 and 3 presented a workflow in a cloud environment (Lumi Augmedit, Naarden, Netherlands) with an integrated automatic segmentation algorithm that automatically creates 3D models from a CT or MRI, which can then be viewed through an MR-HMD. The segmentation algorithm was validated and in Chapter 4 this workflow showed that MR improves the spatial understanding compared to MRI and 3D models on a screen for brain tumors among medical students, residents, and neurosurgeons. We then explored the possibilities of using an MR-HMD as a neuronavigation system, where navigation accuracy is crucial. In Chapter 5, the accuracy of augmented- and mixed reality neuronavigation systems was compared to infrared neuronavigation systems, showing no significant difference. However, augmented- and mixed reality neuronavigation systems displayed more heterogeneity, with the accuracy of mixed reality neuronavigation systems being lower. To improve this accuracy, chapter 6 compared two tracking techniques (Vuforia and ArUco) using an MR-HMD for tracking multiple markers under different conditions. Vuforia showed not only better optimal accuracy but was also the most consistent under all tested parameters. In Chapter 7, the two basic principles of a neuronavigation system were built and tested with an MR-HMD: patient registration and position correction. Although the navigation accuracy still needed optimization, this chapter demonstrated a first working principle of an MR-HMD as a complete neuronavigation system with patients. Chapter 8 provided a technical description and proof-of-concept for navigated placement of external ventricular drains using an MR-HMD. The workflow was integrated into the Lumi cloud environment and designed to be fully compatible for surgery. The initial results on phantoms were acceptable, with the next step being to compare this technique with the currently used free-hand technique in terms of both accuracy and operation time on phantoms. The more medical imaging resembles the actual anatomy, the less mental translation is needed by the surgeon during surgery. Studying anatomy in 3D is therefore a logical need in surgical specialties. Mixed reality offers this possibility, and in this thesis, we have taken steps to use this technique as a neuronavigation system, simplified its use and demonstrated improved spatial understanding. There are still multiple aspects of care to be further explored, as in outpatient clinics, intra-operatively and in education. This, combined with a dynamic market where MR-HMDs will further improve, provides an exciting prospect as we move towards a future where 3D visualization becomes an integrated part of surgical care.