Intraoperative Registration by Cross-Modal Inverse Neural Rendering

Maximilian Fehrentz^1,2, Mohammad Farid Azampour², Reuben Dorent¹, Hassan Rasheed^1,2, Colin Galvin¹, Alexandra Golby¹, William M. Wells^1,3, Sarah Frisken¹, Nassir Navab², Nazim Haouchine¹

¹Harvard Medical School, Brigham and Women's Hospital, Boston, USA
²Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany
³Massachusetts Institute of Technology, Cambridge, USA

Paper Code coming soon

Overview of our method. For a given intraoperative target image, we estimate the camera pose that aligns our stylized NeRF with the intraoperative image, thereby registering the preoperative scan to the intraoperative image.

Abstract

We present in this paper a novel approach for 3D/2D intraoperative registration during neurosurgery via cross-modal inverse neural rendering. Our approach separates implicit neural representation into two components, handling anatomical structure preoperatively and appearance intraoperatively. This disentanglement is achieved by controlling a Neural Radiance Field’s appearance with a multi-style hypernetwork. Once trained, the implicit neural representation serves as a differentiable rendering engine, which can be used to estimate the surgical camera pose by minimizing the dissimilarity between its rendered images and the target intraoperative image. We tested our method on retrospective patients’ data from clinical cases, showing that our method outperforms state-of-the-art while meeting current clinical standards for registration.

Pipeline Overview

Left (preoperative): We use a NeRF to first learn density/anatomy (orange) from a mesh M extracted from an MR scan, then learn single-shot style adaptation (blue) through a hypernetwork h while freezing the rest of the NeRF, keeping the previously learned density fixed. Right (intraoperative): Iterative pose estimation on target I. The trained NeRF and hypernet (green highlights) are used as style-conditioned neural rendering engine using ray casting, with f adapted to the appearance of the intraoperative registration target I through the hypernetwork h.

Synthesis

Our method renders novel views with varying appearances while keeping a fixed anatomical structure. Given that our NeRF is created from and registered to a preoperative scan, we can render novel views with intraoperative appearance and relate it back to the preoperative volume.