Ole Jakob Elle

• Kupcikevicius, Ignas; Boretto, Luca; Grünbeck, Inger Annett; Kumar, Rahul Prasanna; Nainamalai, Varatharajan & Akhavi, Seyed Mohammadmehdi Sadat [Show all 8 contributors for this article] (2026). AI-Enabled Vessels Segmentation Model for Real-Time Laparoscopic Ultrasound Imaging. Proceedings of Machine Learning Research (PMLR). 307. Full text in Research Archive
• Sajadi, Seyedmohammadreza; Tariverdi, Abbas; Brun, Henrik; Elle, Ole Jakob & Mathiassen, Kim (2026). Robotic transesophageal echocardiography: system design and deep learning-based kinematic modeling. Frontiers in Robotics and AI. 12. doi: 10.3389/frobt.2025.1705142. Full text in Research Archive
• Frostelid, Vetle Christoffer; Wajdan, Ali; Villegas-Martinez, Manuel; Hammersbøen, Lars Egil Reine; Espinoza, Andreas & Grymyr, Ole-Johannes Holm Nielsen [Show all 9 contributors for this article] (2025). Continuous and Autonomous Monitoring of Changes in Left Ventricular dP/dt<inf>max</inf>Using an Epicardial Accelerometer. Annals of Biomedical Engineering. ISSN 0090-6964. 53(11), p. 2783–2794. doi: 10.1007/s10439-025-03828-6. Full text in Research Archive
• Homlong, Eirik Gromholt; Qadir, Hemin Ali; Kumar, Rahul Prasanna; Elle, Ole Jakob & Wiig, Ola (2025). Addressing Occlusions and Pose Challenges in Clinical Gait Analysis: A Robust 3D-to-2D Motion Pipeline. IEEE Engineering in Medicine and Biology Society. Conference Proceedings. ISSN 1557-170X. 47. doi: 10.1109/EMBC58623.2025.11254493. Full text in Research Archive Show summary

  Cerebral palsy (CP), one of the most common causes of physical disability in childhood, affects movement, balance and posture due to atypical brain development or damage to motor control areas. Clinical gait analysis (CGA) is essential for the diagnosis, treatment planning and management of CP, and motion capture systems are an important tool for assessing movement patterns. Markerless motion capture systems using computer vision and machine learning algorithms offer several advantages over traditional marker-based methods, including improved patient comfort, mobility and reduced set-up time. However, these systems face challenges, such as pose ambiguities in multi-person images and occlusions, which can reduce the accuracy of kinematic data in clinical applications.In this paper, we present a workflow that addresses these challenges by leveraging machine learning and computer vision techniques to improve the reliability and accuracy of markerless motion capture in clinical gait analysis (CGA). Specifically, our approach utilizes the projection of three-dimensional (3D) onto two-dimensional (2D) points to map the original 3D motion capture points onto the video plane and track the subject using distance-based metrics. Additionally, we integrate multiple data modalities to improve robustness. The proposed workflow enables detailed comparisons between markerless and marker-based systems. By improving the performance and applicability of markerless motion capture, this study contributes to a more accessible and effective clinical assessment of children with CP.

• Grünbeck, Inger Annett; Kumar, Rahul Prasanna; Teatini, Andrea; Elle, Ole Jakob & Wiig, Ola (2025). 4D mixed reality tool for orthopaedic surgery - a feasibility study. Computer Methods in Biomechanics and Biomedical Engineering: Imaging and Visualization. ISSN 2168-1163. 13(1). doi: 10.1080/21681163.2025.2589161. Full text in Research Archive Show summary

  Accurately diagnosing joint conditions and bone deformations relying on 2D imaging remains challenging. This paper examined the feasibility of a new workflow for real-time motion visualisation of a patient’s hip joint, using our novel 4D mixed reality tool providing a non-invasive ‘X-ray vision’ experience. A feasibility study was conducted to evaluate the workflow in a clinical context, asking surgeons to locate pain-causing bone deformations during patient examinations, using X-ray, CT, and our tool. Questionnaires with a Likert scale ranging from ‘Strongly Disagree’ to ‘Strongly Agree’ confirmed the clinical feasibility of our workflow. Surgeons highly rated our tool’s visualisation capabilities and assigned the highest marks when asked about their confidence in pinpointing the deformities’ locations (‘Strongly Agree’): X-ray (0%), CT (25%), presented tool (100%). We believe our visualisation tool will become an asset in advancing orthopaedic diagnostics in the future, contributing to improved patient care and surgical outcomes.

• Nainamalai, Varatharajan; Jenssen, Håvard; Boretto, Luca; Luthra, Nikhil André Kumar; Espinoza, Andreas & Pelanis, Egidijus [Show all 11 contributors for this article] (2025). Morphological changes on the human liver during minimally invasive surgery: Implications for image-guided interventions and surgical navigation. Surgical Endoscopy. ISSN 0930-2794. doi: 10.1007/s00464-025-12392-y. Full text in Research Archive
• d’Albenzio, Gabriella; Meng, Ruoyan; Aghayan, Davit; Pelanis, Egidijus; Sakinis, Tomas & Solberg, Ole Vegard [Show all 11 contributors for this article] (2025). Patient-specific functional liver segments based on centerline classification of the hepatic and portal veins. Computer Assisted Surgery. 30(1). doi: 10.1080/24699322.2025.2580307. Full text in Research Archive Show summary

  Purpose: Couinaud’s liver segment classification has been widely adopted for liver surgery planning, yet its rigid anatomical boundaries often fail to align precisely with individual patient anatomy. This study proposes a novel patient-specific liver segmentation method based on detailed classification of hepatic and portal veins to improve anatomical adherence and clinical relevance. Methods: Our proposed method involves two key stages: (1) surgeons annotate vascular endpoints on 3D models of hepatic and portal veins, from which vessel centerlines are computed; and (2) liver segments are calculated by assigning voxel labels based on proximity to these vascular centerlines. The accuracy and clinical applicability of our Hepatic and Portal Vein-based Classification (HPVC) were compared with conventional Plane-Based Classification (PBC), Portal Vein-Based Classification (PVC), and an automated deep learning method (nnU-Net) using volumetric measurements, Dice similarity scores, and expert evaluations. Results: HPVC demonstrated superior anatomical conformity compared to traditional methods, especially in complex segments like 5 and 8, providing segmentations more reflective of actual vascular territories. Volumetric analysis revealed significant discrepancies among the methods, particularly with nnU-Net generally producing larger segment volumes. HPVC consistently achieved higher surgeon-rated scores in patient-specific anatomical adherence, perfusion region assessment, and accuracy in surgical planning compared to PBC, PVC, and nnU-Net. Conclusion: The presented HPVC method offers substantial improvements in liver segmentation precision, especially relevant for surgical planning in anatomically complex cases. Its integration into clinical workflows via the open-source platform 3D Slicer significantly enhances its accessibility and usability.

• Jenssen, Håvard; Nainamalai, Varatharajan; Pelanis, Egidijus; Kumar, Rahul Prasanna; Abildgaard, Andreas & Kolrud, Finn Kristian [Show all 11 contributors for this article] (2025). Challenges and artificial intelligence solutions for clinically optimal hepatic venous vessel segmentation. Biomedical Signal Processing and Control. ISSN 1746-8094. 106. doi: 10.1016/j.bspc.2025.107822. Full text in Research Archive Show summary

  Background : Liver vessel identification is crucial for clinical disease assessment and treatment planning, especially concerning local treatment of liver tumors. As artificial intelligence (AI) develops in radiology, opportunities arise to craft models adept at hepatic venous vessel segmentation, opening possibilities for creating patient-specific models of the liver anatomy quickly, despite the diverse features of CT images encountered in clinical settings. Objective: This research evaluates the performance of AI models combined with various pre-processing filters for liver vessel segmentation, emphasizing clinically relevant results. A novel evaluation method was introduced to offer more anatomically accurate assessments, moving beyond traditional metrics like the Dice score. Methods: Using open-source and proprietary datasets, we implemented residual UNet and Dense UNet in combination with smoothness and vesselness filters. We used a clinical evaluation approach focused on major and minor liver vessels, thereby underscoring the precision of AI outcomes. Results: The Dense UNet model with a specific pre-processing filter produced an average Dice score of 0.8144 in our internal dataset. For the public test dataset, the score was 0.7859. Both scores were higher than those not using pre-processing filters, 0.8052 and 0.7765. Clinical assessments showed 85% of AI predictions accurately identified all wanted vessel structures, though segmentation beyond the vessel borders did occur in half the predictions. Conclusion: This study highlights the effectiveness of AI in liver vessel segmentation, with the Dense UNet model combined with pre-processing filters showing high Dice scores and clinical accuracy.

• Wajdan, Ali; Frostelid, Vetle Christoffer; Villegas-Martinez, Manuel; Halvorsen, Per Steinar; Krogh, Magnus Reinsfelt & Elle, Ole Jakob [Show all 7 contributors for this article] (2025). Deep neural network valve detection for accelerometer based cardiac monitoring. Scientific Reports. 15(1). doi: 10.1038/s41598-025-00845-2. Full text in Research Archive Show summary

  Miniaturized accelerometers incorporated in pacing leads attached directly onto the heart provide a means for continuous monitoring of cardiac function. Several functional accelerometer indices first require detection of valve events. We previously developed a deep neural network to detect timing of aortic valve closure and opening. In this study we trained and tested the performance of the network to detect timing of mitral valve closure (MVC) and opening (MVO). Furthermore, we extracted four different functional indices based on the detected valve events and investigated how these indices reflected changes in cardiac function. The neural network was tested on approximately 5900 heartbeats from 289 recordings in a total of 46 animals with a cardiac accelerometer attached to the heart during various interventions that altered function. The neural network correctly detected MVO and MVC in 89.6% and 87.5% of the beats, respectively, with a mean absolute error of 13 ms between the detected values and the annotated targets for both. The functional indices correlated well with measured left ventricular stroke work (0.67 < r < 0.84) and showed expected changes for the different interventions. Hence, automatic detection of valve events is feasible and facilitates improved cardiac monitoring when using implanted cardiac accelerometers.

• Lippert, Matthias; d’Albenzio, Gabriella; Suther, Lene Kathrine R; Dumont, Karl-Andreas; Palomar, Rafael & Odland, Hans Henrik [Show all 9 contributors for this article] (2024). HoloPatch: improving intracardiac patch fit through holographically modelled templates. European Heart Journal – Imaging Methods and Practice (EHJ-IMP). 2(3). doi: 10.1093/ehjimp/qyae103. Full text in Research Archive Show summary

  Aims Structural heart defects, including congenital ventricular septal defect closure or intracardiac rerouting, frequently require surgical reconstruction using hand-cut patch materials. Digitally modelled patch templates may improve patch fit and reduce outflow tract obstruction, residual defect risk, and conduction system damage. In this study, we benchmarked mixed-reality and a desktop application against a digitalized model of a real implanted patch. Methods and results Ten patients scheduled for the repair of various defects consented to prospective inclusion in the study. After surgery, a digital model of the implanted patch was created from the residual material. Five clinical experts created 10 digital patches, 1 per patient, both in mixed-reality and desktop application, for comparison with the reference measurements, including the digitalized model of the real patch used during the surgery. Subjective residual shunt risk prediction was performed using both modalities. Digital patches created in mixed-reality closely matched the surgical material, whereas those created using desktop applications were significantly smaller. Different evaluators showed varying preferences for the application of the residual shunt risk and area. Conclusion Digitally created patches can assist surgeons in preoperatively sizing of patch implants, potentially reducing post-operative complications.

• Boretto, Luca; Pelanis, Egidijus; Regensburger, Alois; Petkov, Kaloian; Avalos, Rafael Palomar & Fretland, Åsmund Avdem [Show all 8 contributors for this article] (2024). Intraoperative patient-specific volumetric reconstruction and 3D visualization for laparoscopic liver surgery. Healthcare technology letters. 10(6), p. 374–383. doi: 10.1049/htl2.12106. Full text in Research Archive Show summary

  Despite the benefits of minimally invasive surgery, interventions such as laparoscopic liver surgery present unique challenges, like the significant anatomical differences between preoperative images and intraoperative scenes due to pneumoperitoneum, patient pose, and organ manipulation by surgical instruments. To address these challenges, a method for intraoperative three-dimensional reconstruction of the surgical scene, including vessels and tumors, without altering the surgical workflow, is proposed. The technique combines neural radiance field reconstructions from tracked laparoscopic videos with ultrasound three-dimensional compounding. The accuracy of our reconstructions on a clinical laparoscopic liver ablation dataset, consisting of laparoscope and patient reference posed from optical tracking, laparoscopic and ultrasound videos, as well as preoperative and intraoperative computed tomographies, is evaluated. The authors propose a solution to compensate for liver deformations due to pressure applied during ultrasound acquisitions, improving the overall accuracy of the three-dimensional reconstructions compared to the ground truth intraoperative computed tomography with pneumoperitoneum. A unified neural radiance field from the ultrasound and laparoscope data, which allows real-time view synthesis providing surgeons with comprehensive intraoperative visual information for laparoscopic liver surgery, is trained.

• Lippert, Matthias; Dumont, Karl-Andreas; Birkeland, Sigurd; Nainamalai, Varatharajan; Solvin, Håvard Bjerkeseth & Suther, Lene Kathrine R [Show all 9 contributors for this article] (2024). Cardiac anatomic digital twins: findings from a single national centre. The European Heart Journal – Digital Health (EHJ-DH). 5(6), p. 725–734. doi: 10.1093/ehjdh/ztae070. Full text in Research Archive
• Boretto, Luca; Pelanis, Egidijus; Regensburger, Alois; Fretland, Åsmund Avdem; Edwin, Bjørn von Gohren & Elle, Ole Jakob (2024). Hybrid optical-vision tracking in laparoscopy: accuracy of navigation and ultrasound reconstruction. MITAT. Minimally invasive therapy & allied technologies. ISSN 1364-5706. 33(3), p. 176–183. doi: 10.1080/13645706.2024.2313032. Full text in Research Archive Show summary

  Introduction The use of laparoscopic and robotic liver surgery is increasing. However, it presents challenges such as limited field of view and organ deformations. Surgeons rely on laparoscopic ultrasound (LUS) for guidance, but mentally correlating ultrasound images with pre-operative volumes can be difficult. In this direction, surgical navigation systems are being developed to assist with intra-operative understanding. One approach is performing intra-operative ultrasound 3D reconstructions. The accuracy of these reconstructions depends on tracking the LUS probe. Material and methods This study evaluates the accuracy of LUS probe tracking and ultrasound 3D reconstruction using a hybrid tracking approach. The LUS probe is tracked from laparoscope images, while an optical tracker tracks the laparoscope. The accuracy of hybrid tracking is compared to full optical tracking using a dual-modality tool. Ultrasound 3D reconstruction accuracy is assessed on an abdominal phantom with CT transformed into the optical tracker’s coordinate system. Results Hybrid tracking achieves a tracking error < 2 mm within 10 cm between the laparoscope and the LUS probe. The ultrasound reconstruction accuracy is approximately 2 mm. Conclusion Hybrid tracking shows promising results that can meet the required navigation accuracy for laparoscopic liver surgery.

• Brun, Henrik; Lippert, Matthias; Langø, Thomas; Sanchez-Margallo, Juan; Sanchez-Margallo, Francisco & Elle, Ole Jakob (2024). Comparing assisting technologies for proficiency in cardiac morphology: 3D printing and mixed reality versus CT slice images for morphological understanding of congenital heart defects by medical students. Anatomical Sciences Education. ISSN 1935-9772. 18(1), p. 68–76. doi: 10.1002/ase.2530. Full text in Research Archive Show summary

  Learning cardiac morphology largely involves spatial abilities and studies indicate benefits from innovative 3D visualization technologies that speed up and increase the learning output. Studies comparing these teaching tools and their educational output are rare and few studies include complex congenital heart defects. This study compared the effects of 3D prints, mixed reality (MR) viewing of 3D meshes and standard cardiac CT slice images on medical students′ understanding of complex congenital heart defect morphology, measuring both objective level of understanding and subjective educational experience. The objective of this study was to compare morphological understanding and user experiences of 3D printed models, MR 3D visualization and axial 2D CT slices, in medical students examining morphological details in complex congenital heart defects. Medical students in the median 4th year of study (range 2nd to 6th) examined three of five different complex congenital heart defects by three different modalities: 3D printed model, MR viewed 3D mesh, and cardiac CT slices, answering a questionnaire on morphology and user experience. Time to complete task, diagnostic accuracy, and user experience data were collected and compared on group level. Task times were similar for all modalities. The percentage of correct answers was higher with MR visualization, which was also the preferred modality overall. Medical students both prefer and better understand the morphology of complex congenital heart disease with 3D models viewed using MR, without spending more time than with 3D prints or 2D CT images.

• Maddali, Dharani Dhar; Solvin, Håvard Bjerkeseth; Lippert, Matthias; Karabiyik, Yücel; Dahle, Gry & Hjelmervik, Jon Mikkelsen [Show all 9 contributors for this article] (2024). Monte Carlo-based rendering of 3D echocardiography for mixed reality-guided atrial septal puncture positioning. Computer Assisted Surgery. 29(1). doi: 10.1080/24699322.2024.2403444. Full text in Research Archive Show summary

  Catheter-based intervention procedures contain complex maneuvers, and they are often performed using fluoroscopic guidance assisted by 2D and 3D echocardiography viewed on a flat screen that inherently limits depth perception. Emerging mixed reality (MR) technologies, combined with advanced rendering techniques, offer potential enhancement in depth perception and navigational support. The study aims to evaluate a MR-based guidance system for the atrial septal puncture (ASP) procedure utilizing a phantom anatomical model. A novel MR-based guidance system using a modified Monte Carlo-based rendering approach for 3D echocardiographic visualization was introduced and evaluated against standard clinical 3D echocardiographic display on a flat screen. The objective was to guide the ASP procedure by facilitating catheter placement and puncture across four specific atrial septum quadrants. To assess the system’s feasibility and performance, a user study involving four experienced interventional cardiologists was conducted using a phantom model. Results show that participants accurately punctured the designated quadrant in 14 out of 16 punctures using MR and 15 out of 16 punctures using the flat screen of the ultrasound machine. The geometric mean puncture time for MR was 31 s and 26 s for flat screen guidance. User experience ratings indicated MR-based guidance to be easier to navigate and locate tents of the atrial septum. The study demonstrates the feasibility of MR-guided atrial septal puncture. User experience data, particularly with respect to navigation, imply potential benefits for more complex procedures and educational purposes. The observed performance difference suggests an associated learning curve for optimal MR utilization.

• Nainamalai, Varatharajan; Qadir, Hemin Ali; Pelanis, Egidijus; Jenssen, Håvard Bjørke; Fretland, Åsmund Avdem & Edwin, Bjørn von Gohren [Show all 8 contributors for this article] (2024). Automated algorithm for medical data structuring, and segmentation using artificial intelligence within secured environment for dataset creation. European Journal of Radiology Open (EJR Open). 13. doi: 10.1016/j.ejro.2024.100582. Full text in Research Archive Show summary

  Objective: Routinely collected electronic health records using artificial intelligence (AI)-based systems bring out enormous benefits for patients, healthcare centers, and its industries. Artificial intelligence models can be used to structure a wide variety of unstructured data. Methods: We present a semi-automatic workflow for medical dataset management, including data structuring, research extraction, AI-ground truth creation, and updates. The algorithm creates directories based on keywords in new file names. Results: Our work focuses on organizing computed tomography (CT), magnetic resonance (MR) images, patient clinical data, and segmented annotations. In addition, an AI model is used to generate different initial labels that can be edited manually to create ground truth labels. The manually verified ground truth labels are later included in the structured dataset using an automated algorithm for future research. Conclusion: This is a workflow with an AI model trained on local hospital medical data with output based/adapted to the users and their preferences. The automated algorithms and AI model could be implemented inside a secondary secure environment in the hospital to produce inferences.

• Solvin, Håvard Bjerkeseth; Diab, Simone; Elle, Ole Jakob; Holmstrøm, Henrik & Brun, Henrik (2023). Real-Time Remote-Mentored Echocardiography in Management of Newborns with Critical Congenital Heart Defects. Congenital Heart Disease. ISSN 1747-079X. 18(5), p. 551–559. doi: 10.32604/chd.2023.031537. Full text in Research Archive Show summary

  Background: The management of suspected critical congenital heart defects (CCHD) relies on timely echocardiographic diagnosis. The availability of experienced echocardiographers is limited or even non-existent in many hospitals with obstetric units. This study evaluates remote-mentored echocardiography performed by physicians without experience in imaging of congenital heart defects (CHD). Methods: The setup included a pediatric cardiologist in a separate room, guiding a physician without experience in echocardiographic imaging of CHD in the examination of a symptomatic newborn. This remote-mentoring pair was blinded to the diagnosis of the newborn and presented with a simplified patient history. The echocardiographic images were streamed to the laptop of the mentor, along with a webcam feed showing the probe position. The task was to identify CCHD in need of immediate transfer to a pediatric cardiac surgical center. The result was compared to the previously completed echocardiographic report and the clinical decision of the patient-responsible pediatric cardiologist. Results: During 17 months, 15 newborns were recruited. All six newborns with CCHD were correctly labeled by the remotementoring pair. One newborn with Tetralogy of Fallot was erroneously labeled as needing immediate transfer. Eight newborns without CCHD were correctly labeled. Conclusions: Remote-mentored echocardiography performed by examiners without experience in imaging CHD identified all newborns with CCHD in need of immediate transfer for specialist care. The setup shows promising results for improving the management of CCHD in hospitals without continuous pediatric cardiology service.

• Tariverdi, Abbas; Côté-Allard, Ulysse Teller Masao; Mathiassen, Kim; Elle, Ole Jakob; Kalvøy, Håvard & Martinsen, Ørjan Grøttem [Show all 7 contributors for this article] (2023). Reinforcement Learning-Based Switching Controller for a Milliscale Robot in a Constrained Environment. IEEE Transactions on Automation Science and Engineering. ISSN 1545-5955. 21(2), p. 2000–2016. doi: 10.1109/TASE.2023.3259905. Full text in Research Archive
• Boretto, Luca; Pelanis, Egidijus; Regensburger, Alois & Elle, Ole Jakob (2023). Laparoscopic Feature-Less 3D Reconstruction Using Neural Radiance Fields and Optical Tracking. International Federation for Medical and Biological Engineering Proceedings. ISSN 1680-0737. 110, p. 601–609. doi: 10.1007/978-3-031-62520-6_67. Full text in Research Archive
• Solvin, Håvard; Lippert, Matthias; Holmstrøm, Henrik; Elle, Ole Jakob & Brun, Henrik (2023). Real-time Remote Expert-guided Echocardiography by Medical Students. Ultrasound Journal. 15(1). doi: 10.1186/s13089-023-00328-3. Full text in Research Archive Show summary

  Background Echocardiography is a highly specialised examination performed by experienced healthcare professionals. These experienced healthcare professionals may not be available to patients during all hours in rural healthcare facilities. Remote-guided echocardiography could improve the availability of specialised care for patients living in rural areas. This study examined the feasibility of real-time remote guidance for medical students to perform an echocardiographic assessment of the left side of the heart. Thirteen healthy volunteers were recruited for remote-guided echocardiography, which was performed by 13 medical students. Student examinations/images were compared to reference echocardiography. Measurements of left ventricular fractional shortening and mitral valve blood flow velocity were also compared. Furthermore, guidance through a smartphone videoconference was compared to designated remote guidance software. Results Two-thirds of the images acquired by students were rated as medium or good quality and usable to evaluate two thirds of the cardiac structures. No significant bias was found for left ventricular fractional shortening. The measurements from the students’ exams had a variation coefficient of 14.8% compared to the reference. The calculated deviation of the insonation angle was above 25° for both E and A-wave mitral valve blood flow velocity measurements. Images acquired by guidance through smartphone videoconference were of lower quality than those obtained using the designated remote guidance software. Conclusion Real-time remote-guided echocardiography performed by medical students has limited value for clinical screening but could be useful for educational purposes. Graphical Abstract

• Gholinejad, Maryam; Edwin, Bjørn von Gohren; Elle, Ole Jakob; Dankelman, Jenny & Loeve, Arjo J. (2023). Process model analysis of parenchyma sparing laparoscopic liver surgery to recognize surgical steps and predict impact of new technologies. Surgical Endoscopy. ISSN 0930-2794. 37, p. 7083–7099. doi: 10.1007/s00464-023-10166-y. Full text in Research Archive Show summary

  Abstract Background Surgical process model (SPM) analysis is a great means to predict the surgical steps in a procedure as well as to predict the potential impact of new technologies. Especially in complicated and high-volume treatments, such as parenchyma sparing laparoscopic liver resection (LLR), profound process knowledge is essential for enabling improving surgical quality and efficiency. Methods Videos of thirteen parenchyma sparing LLR were analyzed to extract the duration and sequence of surgical steps according to the process model. The videos were categorized into three groups, based on the tumor locations. Next, a detailed discrete events simulation model (DESM) of LLR was built, based on the process model and the process data obtained from the endoscopic videos. Furthermore, the impact of using a navigation platform on the total duration of the LLR was studied with the simulation model by assessing three different scenarios: (i) no navigation platform, (ii) conservative positive effect, and (iii) optimistic positive effect. Results The possible variations of sequences of surgical steps in performing parenchyma sparing depending on the tumor locations were established. The statistically most probable chain of surgical steps was predicted, which could be used to improve parenchyma sparing surgeries. In all three categories (i–iii) the treatment phase covered the major part (~ 40%) of the total procedure duration (bottleneck). The simulation results predict that a navigation platform could decrease the total surgery duration by up to 30%. Conclusion This study showed a DESM based on the analysis of steps during surgical procedures can be used to predict the impact of new technology. SPMs can be used to detect, e.g., the most probable workflow paths which enables predicting next surgical steps, improving surgical training systems, and analyzing surgical performance. Moreover, it provides insight into the points for improvement and bottlenecks in the surgical process.

• Lilja, Didrik; Schalit, Itai; Espinoza, Andreas; Fiane, Arnt Eltvedt; Dahle, Gry & Sandbu, Helen Littorin [Show all 11 contributors for this article] (2023). Detection of inflow obstruction in left ventricular assist devices by accelerometer: A porcine model study. The Journal of Heart and Lung Transplantation. ISSN 1053-2498. 42(8), p. 1005–1014. doi: 10.1016/j.healun.2023.03.017. Full text in Research Archive Show summary

  Background Left ventricular assist devices (LVAD) provide circulatory blood pump support for severe heart failure patients. Pump inflow obstructions may lead to stroke and pump malfunction. We aimed to verify in vivo that gradual inflow obstructions, representing prepump thrombosis, are detectable by a pump-attached accelerometer, where the routine use of pump power (PLVAD) is deficient. Method In a porcine model (n = 8), balloon-tipped catheters obstructed HVAD inflow conduits by 34% to 94% in 5 levels. Afterload increases and speed alterations were conducted as controls. We computed nonharmonic amplitudes (NHA) of pump vibrations captured by the accelerometer for the analysis. Changes in NHA and PLVAD were tested by a pairwise nonparametric statistical test. Detection sensitivities and specificities were investigated by receiver operating characteristics with areas under the curves (AUC). Results NHA remained marginally affected during control interventions, unlike PLVAD. NHA elevated during obstructions within 52-83%, while mass pendulation was most pronounced. Meanwhile, PLVAD changed far less. Increased pump speeds tended to amplify the NHA elevations. The corresponding AUC was 0.85-1.00 for NHA and 0.35-0.73 for PLVAD. Conclusion Elevated NHA provides a reliable indication of subclinical gradual inflow obstructions. The accelerometer can potentially supplement PLVAD for earlier warnings and localization of pump.

• Villegas-Martinez, Manuel; Odland, Hans Henrik; Hammersbøen, Lars Egil; Sletten, Ole Jakob; Stugaard, Marie & Witsø, Marit Helene Langset [Show all 10 contributors for this article] (2023). Pulse arrival time variation as a non-invasive marker of acute response to cardiac resynchronization therapy. Europace. ISSN 1099-5129. 25(3), p. 1183–1192. doi: 10.1093/europace/euad013. Full text in Research Archive Show summary

  Successful cardiac resynchronization therapy (CRT) shortens the pre-ejection period (PEP) which is prolonged in the left bundle branch block (LBBB). In a combined animal and patient study, we investigated if changes in the pulse arrival time (PAT) could be used to measure acute changes in PEP during CRT implantation and hence be used to evaluate acute CRT response non-invasively and in real time.

• Engelsen, Peder Christian Grendstad; Heyerdahl, Fridtjof; Maddali, Dharani Dhar; Akhavi, Medhi Sadat; Undseth, Ragnhild Marie & Elle, Ole Jakob [Show all 7 contributors for this article] (2023). Ascending with ultrasound: telementored eFAST in flight—a feasibility study. Emergency Radiology. ISSN 1070-3004. 31, p. 25–31. doi: 10.1007/s10140-023-02186-x. Full text in Research Archive Show summary

  Abstract Purpose Teleultrasound uses telecommunication technologies to transmit ultrasound images from a remote location to an expert who guides the acquisition of images and interprets them in real time. Multiple studies have demonstrated the feasibility of teleultrasound. However, its application during helicopter flight using long-term evolution (LTE) for streaming has not been studied. Therefore, we conducted a study to examine the feasibility of teleultrasound in an Airbus H145 helicopter. Methods Four anesthesiologists and one military physician were recruited to perform telementored extended Focused Assessment with Sonography in Trauma (eFAST) during nine helicopter flights, each with a unique healthy volunteer. A radiologist was recruited as a remote expert, guiding the physicians in their examinations. The examining physicians reported the user experience of telementored eFAST on a questionnaire, while the remote expert rated the diagnostic quality of the images on a 1–5 Likert scale. In addition, we measured the duration of the examinations and key LTE network parameters including signal strength, quality, and continuity. Results The images were rated to an average of 4.9 by the remote expert, corresponding to good diagnostic quality. The average duration of telementored eFAST was 05:54 min. LTE coverage was negatively affected by proximity to urban areas and ceased above 2000 ft altitude. Occasional audio problems were addressed by using the Voice over LTE network for communication. The examining physicians unanimously reported on the questionnaire that they would use telementored eFAST on patients. Conclusion Telementored eFAST is feasible in ambulance helicopters and can produce images of good diagnostic quality. However, it relies on stable LTE coverage, which is influenced by many factors, including the helicopter’s altitude and flight path. Furthermore, its benefit on patient outcomes remains to be proven. Graphical abstract

• Homlong, Eirik Gromholt; Kumar, Rahul Prasanna; Elle, Ole Jakob & Wiig, Ola (2023). Automated structuring of gait data for analysis purposes - A deep learning pilot example. Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). doi: 10.1109/EMBC40787.2023.10340938. Full text in Research Archive
• Frutos, Javier Perez de; Pedersen, Andre; Pelanis, Egidijus; Bouget, David Nicolas Jean-Mar; Survarachakan, Shanmugapriya & Langø, Thomas [Show all 8 contributors for this article] (2023). Learning deep abdominal CT registration through adaptive loss weighting and synthetic data generation. PLOS ONE. 18(2). doi: 10.1371/journal.pone.0282110. Full text in Research Archive Show summary

  Purpose This study aims to explore training strategies to improve convolutional neural network-based image-to-image deformable registration for abdominal imaging. Methods Different training strategies, loss functions, and transfer learning schemes were considered. Furthermore, an augmentation layer which generates artificial training image pairs on-the-fly was proposed, in addition to a loss layer that enables dynamic loss weighting. Results Guiding registration using segmentations in the training step proved beneficial for deep-learning-based image registration. Finetuning the pretrained model from the brain MRI dataset to the abdominal CT dataset further improved performance on the latter application, removing the need for a large dataset to yield satisfactory performance. Dynamic loss weighting also marginally improved performance, all without impacting inference runtime. Conclusion Using simple concepts, we improved the performance of a commonly used deep image registration architecture, VoxelMorph. In future work, our framework, DDMR, should be validated on different datasets to further assess its value.

• Frutos, Javier Perez de; Pedersen, Andre; Pelanis, Egidijus; Bouget, David Nicolas Jean-Mar; Survarachakan, Shanmugapriya & Langø, Thomas [Show all 8 contributors for this article] (2022). Train smarter, not harder: learning deep abdominal CT registration on scarce data. arXiv. doi: 10.48550/arXiv.2211.15717. Full text in Research Archive Show summary

  Purpose: This study aims to explore training strategies to improve convolutional neural network-based image-to-image registration for abdominal imaging. Methods: Different training strategies, loss functions, and transfer learning schemes were considered. Furthermore, an augmentation layer which generates artificial training image pairs on-the-fly was proposed, in addition to a loss layer that enables dynamic loss weighting. Results: Guiding registration using segmentations in the training step proved beneficial for deep-learning-based image registration. Finetuning the pretrained model from the brain MRI dataset to the abdominal CT dataset further improved performance on the latter application, removing the need for a large dataset to yield satisfactory performance. Dynamic loss weighting also marginally improved performance, all without impacting inference runtime. Conclusion: Using simple concepts, we improved the performance of a commonly used deep image registration architecture, VoxelMorph. In future work, our framework, DDMR, should be validated on different datasets to further assess its value.

• Sajadi, Seyed Mohammad Reza; Mathiassen, Kim; Brun, Henrik & Elle, Ole Jakob (2022). Design, Kinematic Modeling, and Validation of a Robotic-Assisted Transesophageal Echocardiography System, 2022 IEEE/SICE International Symposium on System Integration (SII 2022) : January 9-12, 2022 (Virtual Conference) Narvik, Norway. IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781665445405. doi: 10.1109/SII52469.2022.9708810. Full text in Research Archive
• Survarachakan, Shanmugapriya; Prasad, Pravda Jith Ray; Naseem, Rabia; Frutos, Javier Perez de; Kumar, Rahul Prasanna & Langø, Thomas [Show all 9 contributors for this article] (2022). Deep learning for image-based liver analysis — A comprehensive review focusing on malignant lesions. Artificial Intelligence in Medicine. ISSN 0933-3657. 130, p. 1–23. doi: 10.1016/j.artmed.2022.102331. Full text in Research Archive Show summary

  Deep learning-based methods, in particular, convolutional neural networks and fully convolutional networks are now widely used in the medical image analysis domain. The scope of this review focuses on the analysis using deep learning of focal liver lesions, with a special interest in hepatocellular carcinoma and metastatic cancer; and structures like the parenchyma or the vascular system. Here, we address several neural network architectures used for analyzing the anatomical structures and lesions in the liver from various imaging modalities such as computed tomography, magnetic resonance imaging and ultrasound. Image analysis tasks like segmentation, object detection and classification for the liver, liver vessels and liver lesions are discussed. Based on the qualitative search, 91 papers were filtered out for the survey, including journal publications and conference proceedings. The papers reviewed in this work are grouped into eight categories based on the methodologies used. By comparing the evaluation metrics, hybrid models performed better for both the liver and the lesion segmentation tasks, ensemble classifiers performed better for the vessel segmentation tasks and combined approach performed better for both the lesion classification and detection tasks. The performance was measured based on the Dice score for the segmentation, and accuracy for the classification and detection tasks, which are the most commonly used metrics.

• Tariverdi, Abbas; Mathiassen, Kim; Søyseth, Vegard; Kalvøy, Håvard; Elle, Ole Jakob & Tørresen, Jim [Show all 8 contributors for this article] (2022). Physics-Based Simulation and Control Framework for Steering a Magnetically-Actuated Guidewire. In Dilger, Lale Canan (Eds.), 2022 8th International Conference on Control, Decision and Information Technologies (CoDIT). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781665496070. doi: 10.1109/CoDIT55151.2022.9803963. Full text in Research Archive
• Grünbeck, Inger Annett; Teatini, Andrea; Kumar, Rahul Prasanna; Elle, Ole Jakob & Wiig, Ola (2022). Evaluation and Comparison of Target Registration Error in Active and Passive Optical Tracking Systems. In Barbieri, Riccardo (Eds.), Proceedings of the 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC 2022). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781728127835. p. 3476–3480. doi: 10.1109/EMBC48229.2022.9871591. Full text in Research Archive
• Gholinejad, Maryam; Pelanis, Egidijus; Aghayan, Davit; Fretland, Åsmund Avdem; Edwin, Bjørn von Gohren & Terkivatan, Turkan [Show all 9 contributors for this article] (2022). Generic surgical process model for minimally invasive liver treatment methods. Scientific Reports. 12(1). doi: 10.1038/s41598-022-19891-1. Full text in Research Archive Show summary

  Abstract Surgical process modelling is an innovative approach that aims to simplify the challenges involved in improving surgeries through quantitative analysis of a well-established model of surgical activities. In this paper, surgical process model strategies are applied for the analysis of different Minimally Invasive Liver Treatments (MILTs), including ablation and surgical resection of the liver lesions. Moreover, a generic surgical process model for these differences in MILTs is introduced. The generic surgical process model was established at three different granularity levels. The generic process model, encompassing thirteen phases, was verified against videos of MILT procedures and interviews with surgeons. The established model covers all the surgical and interventional activities and the connections between them and provides a foundation for extensive quantitative analysis and simulations of MILT procedures for improving computer-assisted surgery systems, surgeon training and evaluation, surgeon guidance and planning systems and evaluation of new technologies.

• Maddali, Dharani Dhar; Brun, Henrik; Kiss, Gabriel; Hjelmervik, Jon M. & Elle, Ole Jakob (2022). Spatial Orientation in Cardiac Ultrasound Images Using Mixed Reality: Design and Evaluation. Frontiers in Virtual Reality. 3. doi: 10.3389/frvir.2022.881338. Full text in Research Archive Show summary

  Spatial orientation is an important skill in structural cardiac imaging. Until recently, 3D cardiac ultrasound has been visualized on a flat screen by using volume rendering. Mixed reality devices enhance depth perception, spatial awareness, interaction, and integration in the physical world, which can prove advantageous with 3D cardiac ultrasound images. In this work, we describe the design of a system for rendering 4D (3D + time) cardiac ultrasound data as virtual objects and evaluate it for ease of spatial orientation by comparing it with a standard clinical viewing platform in a user study. The user study required eight participants to do timed tasks and rate their experience. The results showed that virtual objects in mixed reality provided easier spatial orientation and morphological understanding despite lower perceived image quality. Participants familiar with mixed reality were quicker to orient in the tasks. This suggests that familiarity with the environment plays an important role, and with improved image quality and increased use, mixed reality applications may perform better than conventional 3D echocardiography viewing systems.

• Villegas-Martinez, Manuel; Krogh, Magnus Reinsfelt; Andersen, Øyvind Senstad; Sletten, Ole Jakob; Wajdan, Ali & Odland, Hans Henrik [Show all 8 contributors for this article] (2022). Tracking Early Systolic Motion for Assessing Acute Response to Cardiac Resynchronization Therapy in Real Time. Frontiers in Physiology. 13. doi: 10.3389/fphys.2022.903784. Full text in Research Archive
• Sajadi, Seyed Mohammad Reza; Karbasi, Seyed Mojtaba; Brun, Henrik; Tørresen, Jim; Elle, Ole Jakob & Mathiassen, Kim (2022). Towards Autonomous Robotic Biopsy—Design, Modeling and Control of a Robot for Needle Insertion of a Commercial Full Core Biopsy Instrument. Frontiers in Robotics and AI. 9. doi: 10.3389/frobt.2022.896267. Full text in Research Archive Show summary

  This paper presents the design, control, and experimental evaluation of a novel fully automated robotic-assisted system for the positioning and insertion of a commercial full core biopsy instrument under guidance by ultrasound imaging. The robotic system consisted of a novel 4° of freedom (DOF) add-on robot for the positioning and insertion of the biopsy instrument that is attached to a UR5-based teleoperation system with 6 DOF. The robotic system incorporates the advantages of both freehand and probe-guided biopsy techniques. The proposed robotic system can be used as a slave robot in a teleoperation configuration or as an autonomous or semi-autonomous robot in the future. While the UR5 manipulator was controlled using a teleoperation scheme with force controller, a reinforcement learning based controller using the Deep Deterministic Policy Gradient (DDPG) algorithm was developed for the add-on robotic system. The dexterous workspace analysis of the add-on robotic system demonstrated that the system has a suitable workspace within the US image. Two sets of comprehensive experiments including four experiments were performed to evaluate the robotic system’s performance in terms of the biopsy instrument positioning, and the insertion of the needle inside the ultrasound plane. The experimental results showed the ability of the robotic system for in-plane needle insertion. The overall mean error of all four experiments in the tracking of the needle angle was 0.446°, and the resolution of the needle insertion was 0.002 mm.

• Nainamalai, Varatharajan; Lippert, Matthias; Brun, Henrik; Elle, Ole Jakob & Kumar, Rahul Prasanna (2022). Local integration of deep learning for advanced visualization in congenital heart disease surgical planning. Intelligence-Based Medicine. 6. doi: 10.1016/j.ibmed.2022.100055. Full text in Research Archive
• Nainamalai, Varatharajan; Prasad, Pravda Jith Ray; Pelanis, Egidijus; Edwin, Bjørn von Gohren; Albregtsen, Fritz & Elle, Ole Jakob [Show all 7 contributors for this article] (2022). Evaluation of clinical applicability of automated liver parenchyma segmentation of multi-center magnetic resonance images. European Journal of Radiology Open (EJR Open). 9. doi: 10.1016/j.ejro.2022.100448. Full text in Research Archive
• Lilja, Didrik; Schalit, Itai; Espinoza, Andreas Westenvik; Pettersen, Fred-Johan; Elle, Ole Jakob & Halvorsen, Per Steinar (2022). Detection of inflow obstruction in left ventricular assist devices by accelerometer: An in vitro study. Medical Engineering and Physics. ISSN 1350-4533. 110. doi: 10.1016/j.medengphy.2022.103917. Full text in Research Archive Show summary

  Inflow obstruction in left ventricular assist devices (LVAD) may lead to embolic stroke and pump malfunction. We investigated if an accelerometer detected graded LVAD inflow obstructions. Detection performances were compared to the current continuous surveillance routine based on the pump power consumption (PLVAD). In ten mock circuit experiments, four different-sized pendulating balloons obstructed HVAD™ inflow conduits cross-section areas by 14%-75%. Nonharmonic amplitudes (NHA) of continuous signals from a triaxial accelerometer attached to the LVAD were compared against single-point PLVAD values, using load and speed alterations as control interventions. We analyzed the NHA band power with a pairwise nonparametric statistical test. The detection performances were analyzed by receiver operating characteristics with areas under the curves (AUC). The NHA remained unaffected during load alterations. In contrast, NHA increased significantly from the 27% obstruction level (AUC≥0.82), an effect amplified by increased pump speed. PLVAD did not change significantly below the maximal 75% obstruction level (AUC≤0.36). In conclusion, NHA detected the inflow obstructions much better than PLVAD. The technique may provide a future monitoring modality of any pendulating obstructive inflow pathology.

• Kamkova, Yuliia; Pelanis, Egidijus; Bjørnerud, Atle; Edwin, Bjørn von Gohren; Elle, Ole Jakob & Kumar, Rahul Prasanna (2022). A Fast Method for Whole Liver-and Colorectal Liver Metastasis Segmentations from MRI Using 3D FCNN Networks. Applied Sciences. 12(10). doi: 10.3390/app12105145. Full text in Research Archive
• Wajdan, Ali; Jahren, Tollef Struksnes; Villegas-Martinez, Manuel; Khan, Faraz Hameed; Halvorsen, Per Steinar & Odland, Hans Henrik [Show all 9 contributors for this article] (2022). Automatic Detection of Aortic Valve Events Using Deep Neural Networks on Cardiac Signals From Epicardially Placed Accelerometer. IEEE Journal of Biomedical and Health Informatics. ISSN 2168-2194. 26(9), p. 4450–4461. doi: 10.1109/JBHI.2022.3181148. Full text in Research Archive Show summary

  Background: Miniaturized accelerometers incorporated in pacing leads attached to the myocardium, are used to monitor cardiac function. For this purpose functional indices must be extracted from the acceleration signal. A method that automatically detects the time of aortic valve opening (AVO) and aortic valve closure (AVC) will be helpful for such extraction. We tested if deep learning can be used to detect these valve events from epicardially attached accelerometers, using high fidelity pressure measurements to establish ground truth for these valve events. Method: A deep neural network consisting of a CNN, an RNN, and a multi-head attention module was trained and tested on 130 recordings from 19 canines and 159 recordings from 27 porcines covering different interventions. Due to limited data, nested cross-validation was used to assess the accuracy of the method. Result: The correct detection rates were 98.9% and 97.1% for AVO and AVC in canines and 98.2% and 96.7% in porcines when defining a correct detection as a prediction closer than 40 ms to the ground truth. The incorrect detection rates were 0.7% and 2.3% for AVO and AVC in canines and 1.1% and 2.3% in porcines. The mean absolute error between correct detections and their ground truth was 8.4 ms and 7.2 ms for AVO and AVC in canines, and 8.9 ms and 10.1 ms in porcines. Conclusion: Deep neural networks can be used on signals from epicardially attached accelerometers for robust and accurate detection of the opening and closing of the aortic valve.

• Teatini, Andrea; Kumar, Rahul Prasanna; Elle, Ole Jacob & Wiig, Ola (2021). Mixed reality as a novel tool for diagnostic and surgical navigation in orthopaedics. International Journal of Computer Assisted Radiology and Surgery. ISSN 1861-6410. 16(3), p. 407–414. doi: 10.1007/s11548-020-02302-z. Full text in Research Archive
• Krogh, Magnus Reinsfelt; Halvorsen, Per Steinar; Grymyr, Ole-Johannes Holm Nielsen; Bergsland, Jacob; Elle, Ole Jakob & Fosse, Erik [Show all 7 contributors for this article] (2021). Continuous Estimation of Acute Changes in Preload Using Epicardially Attached Accelerometers. IEEE Transactions on Biomedical Engineering. ISSN 0018-9294. 68(7), p. 2067–2075. doi: 10.1109/TBME.2020.3020358. Full text in Research Archive
• Prasad, Pravda Jith Ray; Survarachakan, Shanmugapriya; Khan, Zohaib Amjad; Lindseth, Frank; Elle, Ole Jacob & Albregtsen, Fritz [Show all 7 contributors for this article] (2021). Numerical evaluation on parametric choices influencing segmentation results in radiology images—a multi-dataset study. Electronics (Basel). 10:431(4), p. 1–25. doi: 10.3390/electronics10040431. Full text in Research Archive Show summary

  Medical image segmentation has gained greater attention over the past decade, especially in the field of image-guided surgery. Here, robust, accurate and fast segmentation tools are important for planning and navigation. In this work, we explore the Convolutional Neural Network (CNN) based approaches for multi-dataset segmentation from CT examinations. We hypothesize that selection of certain parameters in the network architecture design critically influence the segmentation results. We have employed two different CNN architectures, 3D-UNet and VGG-16, given that both networks are well accepted in the medical domain for segmentation tasks. In order to understand the efficiency of different parameter choices, we have adopted two different approaches. The first one combines different weight initialization schemes with different activation functions, whereas the second approach combines different weight initialization methods with a set of loss functions and optimizers. For evaluation, the 3D-UNet was trained with the Medical Segmentation Decathlon dataset and VGG-16 using LiTS data. The quality assessment done using eight quantitative metrics enhances the probability of using our proposed strategies for enhancing the segmentation results. Following a systematic approach in the evaluation of the results, we propose a few strategies that can be adopted for obtaining good segmentation results. Both of the architectures used in this work were selected on the basis of general acceptance in segmentation tasks for medical images based on their promising results compared to other state-of-the art networks. The highest Dice score obtained in 3D-UNet for the liver, pancreas and cardiac data was 0.897, 0.691 and 0.892. In the case of VGG-16, it was solely developed to work with liver data and delivered a Dice score of 0.921. From all the experiments conducted, we observed that two of the combinations with Xavier weight initialization (also known as Glorot), Adam optimiser, Cross Entropy loss (GloAdamCE) and LeCun weight initialization, cross entropy loss and Adam optimiser LecAdamCE worked best for most of the metrics in a 3D-UNet setting, while Xavier together with cross entropy loss and Tanh activation function (GlotanhCE) worked best for the VGG-16 network. Here, the parameter combinations are proposed on the basis of their contributions in obtaining optimal outcomes in segmentation evaluations. Moreover, we discuss that the preliminary evaluation results show that these parameters could later on be used for gaining more insights into model convergence and optimal solutions.The results from the quality assessment metrics and the statistical analysis validate our conclusions and we propose that the presented work can be used as a guide in choosing parameters for the best possible segmentation results for future works.

• Tariverdi, Abbas; Venkiteswaran, Venkatasubramanian Kalpathy; Richter, Michiel; Elle, Ole Jakob; Tørresen, Jim & Mathiassen, Kim [Show all 8 contributors for this article] (2021). A Recurrent Neural-Network-Based Real-Time Dynamic Model for Soft Continuum Manipulators. Frontiers in Robotics and AI. doi: 10.3389/frobt.2021.631303. Full text in Research Archive Show summary

  This paper introduces and validates a real-time dynamic predictive model based on a neural network approach for soft continuum manipulators. The presented model provides a real-time prediction framework using neural-network-based strategies and continuum mechanics principles. A time-space integration scheme is employed to discretize the continuous dynamics and decouple the dynamic equations for translation and rotation for each node of a soft continuum manipulator. Then the resulting architecture is used to develop distributed prediction algorithms using recurrent neural networks. The proposed RNN-based parallel predictive scheme does not rely on computationally intensive algorithms; therefore, it is useful in real-time applications. Furthermore, simulations are shown to illustrate the approach performance on soft continuum elastica, and the approach is also validated through an experiment on a magnetically-actuated soft continuum manipulator. The results demonstrate that the presented model can outperform classical modeling approaches such as the Cosserat rod model while also shows possibilities for being used in practice.

• Wang, Congcong; Cheikh, Faouzi Alaya; Beghdadi, Azeddine & Elle, Ole Jacob (2020). Adaptive context encoding module for semantic segmentation. IS&T International Symposium on Electronic Imaging Science and Technology. 2020(10), p. 1–7. doi: 10.2352/ISSN.2470-1173.2020.10.IPAS-027. Full text in Research Archive Show summary

  The object sizes in images are diverse, therefore, capturing multiple scale context information is essential for semantic segmentation. Existing context aggregation methods such as pyramid pooling module (PPM) and atrous spatial pyramid pooling (ASPP) employ different pooling size or atrous rate, such that multiple scale information is captured. However, the pooling sizes and atrous rates are chosen empirically. Rethinking of ASPP leads to our observation that learnable sampling locations of the convolution operation can endow the network learnable fieldof-view, thus the ability of capturing object context information adaptively. Following this observation, in this paper, we propose an adaptive context encoding (ACE) module based on deformable convolution operation where sampling locations of the convolution operation are learnable. Our ACE module can be embedded into other Convolutional Neural Networks (CNNs) easily for context aggregation. The effectiveness of the proposed module is demonstrated on Pascal-Context and ADE20K datasets. Although our proposed ACE only consists of three deformable convolution blocks, it outperforms PPM and ASPP in terms of mean Intersection of Union (mIoU) on both datasets. All the experimental studies confirm that our proposed module is effective compared to the state-of-the-art methods.

• Wajdan, Ali; Krogh, Magnus Reinsfelt; Villegas-Martinez, Manuel; Halvorsen, Per Steinar; Elle, Ole Jacob & Remme, Espen W. (2020). Automatic detection of valve events by epicardial accelerometer allows estimation of the left ventricular pressure trace and pressure–displacement loop area. Scientific Reports. 10. doi: 10.1038/s41598-020-76637-7. Full text in Research Archive Show summary

  Measurements of the left ventricular (LV) pressure trace are rarely performed despite high clinical interest. We estimated the LV pressure trace for an individual heart by scaling the isovolumic, ejection and filling phases of a normalized, averaged LV pressure trace to the time-points of opening and closing of the aortic and mitral valves detected in the individual heart. We developed a signal processing algorithm that automatically detected the time-points of these valve events from the motion signal of a miniaturized accelerometer attached to the heart surface. Furthermore, the pressure trace was used in combination with measured displacement from the accelerometer to calculate the pressure–displacement loop area. The method was tested on data from 34 animals during different interventions. The accuracy of the accelerometer-detected valve events was very good with a median difference of 2 ms compared to valve events defined from hemodynamic reference recordings acquired simultaneously with the accelerometer. The average correlation coefficient between the estimated and measured LV pressure traces was r = 0.98. Finally, the LV pressure–displacement loop areas calculated using the estimated and measured pressure traces showed very good correlation (r = 0.98). Hence, the pressure–displacement loop area can be assessed solely from accelerometer recordings with very good accuracy.

• Pelanis, Egidijus; Teatini, Andrea; Eigl, Benjamin; Regensburger, Alois; Alzaga, Amilcar & Kumar, Rahul Prasanna [Show all 12 contributors for this article] (2020). Evaluation of a novel navigation platform for laparoscopic liver surgery with organ deformation compensation using injected fiducials. Medical Image Analysis. ISSN 1361-8415. 69. doi: 10.1016/j.media.2020.101946. Full text in Research Archive Show summary

  In laparoscopic liver resection, surgeons conventionally rely on anatomical landmarks detected through a laparoscope, preoperative volumetric images and laparoscopic ultrasound to compensate for the challenges of minimally invasive access. Image guidance using optical tracking and registration procedures is a promising tool, although often undermined by its inaccuracy. This study evaluates a novel surgical navigation solution that can compensate for liver deformations using an accurate and effective registration method. The proposed solution relies on a robotic C-arm to perform registration to preoperative CT/MRI image data and allows for intraoperative updates during resection using fluoroscopic images. Navigation is offered both as a 3D liver model with real-time instrument visualization, as well as an augmented reality overlay on the laparoscope camera view. Testing was conducted through a pre-clinical trial which included four porcine models. Accuracy of the navigation system was measured through two evaluation methods: liver surface fiducials reprojection and a comparison between planned and navigated resection margins. Target Registration Error with the fiducials evaluation shows that the accuracy in the vicinity of the lesion was 3.78 ± 1.89 mm. Resection margin evaluations resulted in an overall median accuracy of 4.44 mm with a maximum error of 9.75 mm over the four subjects. The presented solution is accurate enough to be potentially clinically beneficial for surgical guidance in laparoscopic liver surgery.

• Teatini, Andrea; Frutos, Javier Perez de; Eigl, Benjamin; Pelanis, Egidijus; Aghayan, Davit & Lai, Marco [Show all 10 contributors for this article] (2020). Influence of sampling accuracy on augmented reality for laparoscopic image-guided surgery. MITAT. Minimally invasive therapy & allied technologies. ISSN 1364-5706. 30(4), p. 1–10. doi: 10.1080/13645706.2020.1727524. Full text in Research Archive Show summary

  Purpose This study aims to evaluate the accuracy of point-based registration (PBR) when used for augmented reality (AR) in laparoscopic liver resection surgery. Material and methods The study was conducted in three different scenarios in which the accuracy of sampling targets for PBR decreases: using an assessment phantom with machined divot holes, a patient-specific liver phantom with markers visible in computed tomography (CT) scans and in vivo, relying on the surgeon’s anatomical understanding to perform annotations. Target registration error (TRE) and fiducial registration error (FRE) were computed using five randomly selected positions for image-to-patient registration. Results AR with intra-operative CT scanning showed a mean TRE of 6.9 mm for the machined phantom, 7.9 mm for the patient-specific phantom and 13.4 mm in the in vivo study. Conclusions AR showed an increase in both TRE and FRE throughout the experimental studies, proving that AR is not robust to the sampling accuracy of the targets used to compute image-to-patient registration. Moreover, an influence of the size of the volume to be register was observed. Hence, it is advisable to reduce both errors due to annotations and the size of registration volumes, which can cause large errors in AR systems.

• Satpute, Nitin; Naseem, Rabia; Pelanis, Egidijus; Gomez-Luna, Juan; Cheikh, Faouzi Alaya & Elle, Ole Jacob [Show all 7 contributors for this article] (2020). GPU acceleration of liver enhancement for tumor segmentation. Computer Methods and Programs in Biomedicine. ISSN 0169-2607. 184. doi: 10.1016/j.cmpb.2019.105285. Full text in Research Archive Show summary

  Background and objective: Medical image segmentation plays a vital role in medical image analysis. There are many algorithms developed for medical image segmentation which are based on edge or region characteristics. These are dependent on the quality of the image. The contrast of a CT or MRI image plays an important role in identifying region of interest i.e. lesion(s). In order to enhance the contrast of image, clinicians generally use manual histogram adjustment technique which is based on 1D histogram specification. This is time consuming and results in poor distribution of pixels over the image. Cross modality based contrast enhancement is 2D histogram specification technique. This is robust and provides a more uniform distribution of pixels over CT image by exploiting the inner structure information from MRI image. This helps in increasing the sensitivity and accuracy of lesion segmentation from enhanced CT image. The sequential implementation of cross modality based contrast enhancement is slow. Hence we propose GPU acceleration of cross modality based contrast enhancement for tumor segmentation. Methods: The aim of this study is fast parallel cross modality based contrast enhancement for CT liver images. This includes pairwise 2D histogram, histogram equalization and histogram matching. The sequential implementation of the cross modality based contrast enhancement is computationally expensive and hence time consuming. We propose persistence and grid-stride loop based fast parallel contrast enhancement for CT liver images. We use enhanced CT liver image for the lesion or tumor segmentation. We implement the fast parallel gradient based dynamic seeded region growing for lesion segmentation. Results: The proposed parallel approach is 104.416 ( ± 5.166) times faster compared to the sequential implementation and increases the sensitivity and specificity of tumor segmentation. Conclusion: The cross modality approach is inspired by 2D histogram specification which incorporates spatial information existing in both guidance and input images for remapping the input image intensity values. The cross modality based liver contrast enhancement improves the quality of tumor segmentation.

• Zachariadis, Orestis; Teatini, Andrea; satpute, Nitin; Gómez-Luna, Juan; Mutlu, Onur & Elle, Ole Jacob [Show all 7 contributors for this article] (2020). Accelerating B-spline interpolation on GPUs: Application to medical image registration. Computer Methods and Programs in Biomedicine. ISSN 0169-2607. 193, p. 1–12. doi: 10.1016/j.cmpb.2020.105431. Full text in Research Archive
• Kumar, Rahul Prasanna; Pelanis, Egidijus; Bugge, Robin Antony Birkeland; Brun, Henrik; Palomar, Rafael & Aghayan, Davit [Show all 9 contributors for this article] (2020). Use of mixed reality for surgery planning: Assessment and development workflow. Journal of Biomedical Informatics: X. 8. doi: 10.1016/j.yjbinx.2020.100077. Full text in Research Archive Show summary

  Meticulous preoperative planning is an important part of any surgery to achieve high levels of precision and avoid complications. Conventional medical 2D images and their corresponding three-dimensional (3D) reconstructions are the main components of an efficient planning system. However, these systems still use flat screens for visualisation of 3D information, thus losing depth information which is crucial for 3D spatial understanding. Currently, cutting-edge mixed reality systems have shown to be a worthy alternative to provide 3D information to clinicians. In this work, we describe development details of the different steps in the workflow for the clinical use of mixed reality, including results from a qualitative user evaluation and clinical use-cases in laparoscopic liver surgery and heart surgery. Our findings indicate a very high general acceptance of mixed reality devices with our applications and they were consistently rated high for device, visualisation and interaction areas in our questionnaire. Furthermore, our clinical use-cases demonstrate that the surgeons perceived the HoloLens to be useful, recommendable to other surgeons and also provided a definitive answer at a multi-disciplinary team meeting.

• Tariverdi, Abbas; Venkiteswaran, Venkatasubramanian Kalpathy; Martinsen, Ørjan Grøttem; Elle, Ole Jacob; Tørresen, Jim & Misra, Sarthak (2020). Dynamic modeling of soft continuum manipulators using lie group variational integration. PLOS ONE. 15:e0236121(7), p. 1–29. doi: 10.1371/journal.pone.0236121. Full text in Research Archive Show summary

  This paper presents the derivation and experimental validation of algorithms for modeling and estimation of soft continuum manipulators using Lie group variational integration. Existing approaches are generally limited to static and quasi-static analyses, and are not sufficiently validated for dynamic motion. However, in several applications, models need to consider the dynamical behavior of the continuum manipulators. The proposed modeling and estimation formulation is obtained from a discrete variational principle, and therefore grants outstanding conservation properties to the continuum mechanical model. The main contribution of this article is the experimental validation of the dynamic model of soft continuum manipulators, including external torques and forces (e.g., generated by magnetic fields, friction, and the gravity), by carrying out different experiments with metal rods and polymer-based soft rods. To consider dissipative forces in the validation process, distributed estimation filters are proposed. The experimental and numerical tests also illustrate the algorithm’s performance on a magnetically-actuated soft continuum manipulator. The model demonstrates good agreement with dynamic experiments in estimating the tip position of a Polydimethylsiloxane (PDMS) rod. The experimental results show an average absolute error and maximum error in tip position estimation of 0.13 mm and 0.58 mm, respectively, for a manipulator length of 60.55 mm.

• Khan, Zohaib Amjad; Beghdadi, Azeddine; Cheikh, Faouzi Alaya; Kaaniche, Mounir; Pelanis, Egidijus & Palomar, Rafael [Show all 9 contributors for this article] (2020). Towards a video quality assessment based framework for enhancement of laparoscopic videos. Progress in Biomedical Optics and Imaging. ISSN 1605-7422. 11316. doi: 10.1117/12.2549266. Full text in Research Archive Show summary

  Laparoscopic videos can be affected by different distortions which may impact the performance of surgery and introduce surgical errors. In this work, we propose a framework for automatically detecting and identifying such distortions and their severity using video quality assessment. There are three major contributions presented in this work (i) a proposal for a novel video enhancement framework for laparoscopic surgery; (ii) a publicly available database for quality assessment of laparoscopic videos evaluated by expert as well as non-expert observers and (iii) objective video quality assessment of laparoscopic videos including their correlations with expert and non-expert scores.

• Wang, Congcong; Mohammed, Ahmed Kedir; Cheikh, Faouzi Alaya; Beghdadi, Azeddine & Elle, Ole Jacob (2019). Multiscale deep desmoking for laparoscopic surgery. Progress in Biomedical Optics and Imaging. ISSN 1605-7422. 10949:109491Y, p. 1–9. doi: 10.1117/12.2507822. Full text in Research Archive Show summary

  In minimally invasive surgery, smoke generated by such as electrocautery and laser ablation deteriorates image quality severely. This creates discomfortable view for the surgeon which may increase surgical risk and degrade the performance of computer assisted surgery algorithms such as segmentation, reconstruction, tracking, etc. Therefore, real-time smoke removal is required to keep a clear field of view. In this paper, we propose a real-time smoke removal approach based on Convolutional Neural Network (CNN). An encoder-decoder architecture with Laplacian image pyramid decomposition input strategy is proposed. This is an end-to-end network which takes the smoke image and its Laplacian image pyramid decomposition as inputs, and outputs a smoke free image directly without relying on any physical models or estimation of intermediate parameters. This design can be further embedded to deep learning based follow-up image guided surgery processes such as segmentation and tracking tasks easily. A dataset with synthetic smoke images generated from Blender and Adobe Photoshop is employed for training the network. The result is evaluated quantitatively on synthetic images and qualitatively on a laparoscopic dataset degraded with real smoke. Our proposed method can eliminate smoke effectively while preserving the original colors and reaches 26 fps for a video of size 512 × 512 on our training machine. The obtained results not only demonstrate the efficiency and effectiveness of the proposed CNN structure, but also prove the potency of training the network on synthetic dataset.

• Wajdan, Ali; Krogh, Magnus Reinsfelt; Villegas-Martinez, Manuel; Halvorsen, Per Steinar; Grymyr, Ole-Johannes Holm Nielsen & Elle, Ole Jacob [Show all 7 contributors for this article] (2019). Monitoring cardiac function by accelerometer -Detecting start systole from the acceleration signal makes additional ECG recordings for R-peak detection redundant. IEEE Engineering in Medicine and Biology Society. Conference Proceedings. ISSN 1557-170X. p. 4922–4925. doi: 10.1109/EMBC.2019.8856417. Full text in Research Archive Show summary

  A miniaturized accelerometer attached to the heart has been used for monitoring functional parameters such as early systolic velocity and displacement. Currently, processing of the accelerometer signal for derival of these functional parameters depends on determining start systole by detecting the ECG R-peaks. This study proposes an alternative method using only the accelerometer signal to detect start systole, making additional ECG recordings for this purpose redundant. A signal processing method for automatic detection of start systole by accelerometer alone was developed and compared with detected R-peaks in 15 pigs during 5 different interventions showing a difference of 30 ± 17 ms. Furthermore, the derived early systolic velocity and displacement using only accelerometer measurements correlated well (r 2 =0.91 and 0.82, respectively) with minor differences compared to the current method using ECG R-peaks as time reference. The results show that an accelerometer can be used to monitor cardiac function without the need to measure ECG which can simplify the monitoring system.

• Krogh, Magnus Reinsfelt; Halvorsen, Per Steinar; Elle, Ole Jacob; Bergsland, Jacob & Remme, Espen W. (2019). Dynamic gravity compensation does not increase detection of myocardial ischemia in combined accelerometer and gyro sensor measurements. Scientific Reports. 9(1). doi: 10.1038/s41598-018-35630-x. Full text in Research Archive Show summary

  Previous studies have shown that miniaturised accelerometers can be used to monitor cardiac function and automatically detect ischemic events. However, accelerometers cannot differentiate between acceleration due to motion and acceleration due to gravity. Gravity filtering is essential for accurate integration of acceleration to yield velocity and displacement. Heart motion is cyclic and mean acceleration over time is zero. Thus, static gravity filtering is performed by subtracting mean acceleration. However, the heart rotates during the cycle, the gravity component is therefore not constant, resulting in overestimation of motion by static filtering. Accurate motion can be calculated using dynamic gravity filtering by a combined gyro and accelerometer. In an animal model, we investigated whether increased accuracy using dynamic filtering, compared to using static filtering, would enhance the ability to detect ischemia. Additionally, we investigated how well the gyro alone could detect ischemia based on the heart’s rotation. Dynamic filtering tended towards lower sensitivity and specificity, using receiver operating characteristics analysis, for ischemia-detection compared to static filtering (area under the curve (AUC): 0.83 vs 0.93, p = 0.125). The time-varying gravity component indirectly reflects the heart’s rotation. Hence, static filtering has the advantage of indirectly including rotation, which alone demonstrated excellent sensitivity to ischemia (AUC = 0.98).

• Pelanis, Egidijuis; Kumar, Rahul; Aghayan, David; Palomar, Rafael; Fretland, Åsmund A. & Brun, Henrik [Show all 8 contributors for this article] (2019). Use of mixed reality for imåproved spatial understanding of liver anatomy. MITAT. Minimally invasive therapy & allied technologies. ISSN 1364-5706. doi: 10.1080/13645706.2019.1616558. Full text in Research Archive
• Miseikis, Justinas; Brijacak, Inka; Yahyanejad, Saeed; Glette, Kyrre; Elle, Ole Jacob & Tørresen, Jim (2019). Two-Stage Transfer Learning for Heterogeneous Robot Detection and 3D Joint Position Estimation in a 2D Camera Image Using CNN. IEEE International Conference on Robotics and Automation (ICRA). ISSN 1050-4729. 2019-May, p. 8883–8889. doi: 10.1109/ICRA.2019.8794077. Full text in Research Archive Show summary

  Collaborative robots are becoming more common on factory floors as well as regular environments, however, their safety still is not a fully solved issue. Collision detection does not always perform as expected and collision avoidance is still an active research area. Collision avoidance works well for fixed robot-camera setups, however, if they are shifted around, Eye-to-Hand calibration becomes invalid making it difficult to accurately run many of the existing collision avoidance algorithms. We approach the problem by presenting a stand-alone system capable of detecting the robot and estimating its position, including individual joints, by using a simple 2D colour image as an input, where no Eye-to-Hand calibration is needed. As an extension of previous work, a two-stage transfer learning approach is used to re-train a multi-objective convolutional neural network (CNN) to allow it to be used with heterogeneous robot arms. Our method is capable of detecting the robot in real-time and new robot types can be added by having significantly smaller training datasets compared to the requirements of a fully trained network. We present data collection approach, the structure of the multi-objective CNN, the two-stage transfer learning training and test results by using real robots from Universal Robots, Kuka, and Franka Emika. Eventually, we analyse possible application areas of our method together with the possible improvements.

• Teatini, Andrea; Pelanis, Egidijus; Aghayan, Davit; Kumar, Rahul Prasanna; Palomar, Rafael & Fretland, Åsmund Avdem [Show all 8 contributors for this article] (2019). The effect of intraoperative imaging on surgical navigation for laparoscopic liver resection surgery. Scientific Reports. 9:18687, p. 1–11. doi: 10.1038/s41598-019-54915-3. Full text in Research Archive Show summary

  Conventional surgical navigation systems rely on preoperative imaging to provide guidance. In laparoscopic liver surgery, insufflation of the abdomen (pneumoperitoneum) can cause deformations on the liver, introducing inaccuracies in the correspondence between the preoperative images and the intraoperative reality. This study evaluates the improvements provided by intraoperative imaging for laparoscopic liver surgical navigation, when displayed as augmented reality (AR). Significant differences were found in terms of accuracy of the AR, in favor of intraoperative imaging. In addition, results showed an effect of user-induced error: image-to-patient registration based on annotations performed by clinicians caused 33% more inaccuracy as compared to image-to-patient registration algorithms that do not depend on user annotations. Hence, to achieve accurate surgical navigation for laparoscopic liver surgery, intraoperative imaging is recommendable to compensate for deformation. Moreover, user annotation errors may lead to inaccuracies in registration processes.

• Pelanis, Egidijus; Kumar, Rahul Prasanna; Aghayan, Davit; Palomar, Rafael; Fretland, Åsmund Avdem & Brun, Henrik [Show all 8 contributors for this article] (2019). Use of mixed reality for improved spatial understanding of liver anatomy. MITAT. Minimally invasive therapy & allied technologies. ISSN 1364-5706. p. 1–7. doi: 10.1080/13645706.2019.1616558. Full text in Research Archive Show summary

  Introduction: In liver surgery, medical images from pre-operative computed tomography and magnetic resonance imaging are the basis for the decision-making process. These images are used in surgery planning and guidance, especially for parenchyma-sparing hepatectomies. Though medical images are commonly visualized in two dimensions (2D), surgeons need to mentally reconstruct this information in three dimensions (3D) for a spatial understanding of the anatomy. The aim of this work is to investigate whether the use of a 3D model visualized in mixed reality with Microsoft HoloLens increases the spatial understanding of the liver, compared to the conventional way of using 2D images. Material and methods: In this study, clinicians had to identify liver segments associated to lesions. Results: Twenty-eight clinicians with varying medical experience were recruited for the study. From a total of 150 lesions, 89 were correctly assigned without significant difference between the modalities. The median time for correct identification was 23.5 [4–138] s using the magnetic resonance imaging images and 6.00 [1–35] s using HoloLens (p < 0.001). Conclusions: The use of 3D liver models in mixed reality significantly decreases the time for tasks requiring a spatial understanding of the organ. This may significantly decrease operating time and improve use of resources.

• Naseem, Rabia; Cheikh, Faouzi Alaya; Beghdadi, Azeddine; Elle, Ole Jacob & Lindseth, Frank (2019). Cross modality guided liver image enhancement of CT using MRI. European Workshop on Visual Information Processing. ISSN 2471-8963. 2019-October:8946196, p. 46–51. doi: 10.1109/EUVIP47703.2019.8946196. Full text in Research Archive Show summary

  Low contrast Computed Tomographic (CT) images often hamper the diagnosis of critical tumors found in various human organs. Contrast enhancement schemes play significant role in improving the visualization of these structures. To achieve this objective, Crossmodality Guided Enhancement (CMGE) method is proposed in this paper. The idea is to exploit the diversity of the information extracted from one modality to enhance the important structures including vessels and tumors in another modality. Our method employs information from liver Magnetic Resonance Image (MRI) to generate an enhanced CT image. It entails applying two dimensional histogram specification to map 2D histogram of CT to that of MRI followed by application of top and bottom hat transformations. These morphological operations highlight areas brighter than their surroundings and suppress darker areas. The final image is obtained by combining the results of these operations. Our method is compared with other state of the art contrast enhancement methods both visually and in terms of quality assessment metrics IEM and EME. The results show that our method performs better than these methods. CMGE technique yields improved contrast in low contrast CT images of the human liver and highlights tumors and vessels.

• Luzon, Javier; Kumar, Rahul Prasanna; Stimec, Bojan V.; Elle, Ole Jacob; Bakka, Arne Olav & Edwin, Bjørn [Show all 7 contributors for this article] (2019). Semi-automated vs. manual 3D reconstruction of central mesenteric vascular models: the surgeon’s verdict. Surgical Endoscopy. ISSN 0930-2794. p. 1–11. doi: 10.1007/s00464-019-07275-y. Full text in Research Archive
• Sdiri, Bilel; Kaaniche, Mounir; Cheikh, Faouzi Alaya; Beghdadi, Azeddine & Elle, Ole Jacob (2019). Efficient enhancement of stereo endoscopic images based on joint wavelet decomposition and binocular combination. IEEE Transactions on Medical Imaging. ISSN 0278-0062. 38(1), p. 33–45. doi: 10.1109/TMI.2018.2853808. Full text in Research Archive Show summary

  The success of minimally invasive interventions and the remarkable technological and medical progress have made endoscopic image enhancement a very active research field. Due to the intrinsic endoscopic domain characteristics and the surgical exercise, stereo endoscopic images may suffer from different degradations which affect its quality. Therefore, in order to provide the surgeons with a better visual feedback and improve the outcomes of possible subsequent processing steps, namely, a 3-D organ reconstruction/registration, it would be interesting to improve the stereo endoscopic image quality. To this end, we propose, in this paper, two joint enhancement methods which operate in the wavelet transform domain. More precisely, by resorting to a joint wavelet decomposition, the wavelet subbands of the right and left views are simultaneously processed to exploit the binocular vision properties. While the first proposed technique combines only the approximation subbands of both views, the second method combines all the wavelet subbands yielding an inter-view processing fully adapted to the local features of the stereo endoscopic images. Experimental results, carried out on various stereo endoscopic datasets, have demonstrated the efficiency of the proposed enhancement methods in terms of perceived visual image quality.

• Miseikis, Justinas; Brijacak, Inka; Yahyanejad, Saeed; Glette, Kyrre; Elle, Ole Jacob & Tørresen, Jim (2018). Transfer Learning for Unseen Robot Detection and Joint Estimation on a Multi-Objective Convolutional Neural Network, 2018 IEEE International Conference on Intelligence and Safety for Robotics (ISR 2018). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781538655481. p. 337–342. doi: 10.1109/IISR.2018.8535937. Full text in Research Archive Show summary

  A significant problem of using deep learning techniques is the limited amount of data available for training. There are some datasets available for the popular problems like item recognition and classification or self-driving cars, however, it is very limited for the industrial robotics field. In previous work, we have trained a multi-objective Convolutional Neural Network (CNN) to identify the robot body in the image and estimate 3D positions of the joints by using just a 2D image, but it was limited to a range of robots produced by Universal Robots (UR). In this work, we extend our method to work with a new robot arm - Kuka LBR iiwa, which has a significantly different appearance and an additional joint. However, instead of collecting large datasets once again, we collect a number of smaller datasets containing a few hundred frames each and use transfer learning techniques on the CNN trained on UR robots to adapt it to a new robot having different shapes and visual features. We have proven that transfer learning is not only applicable in this field, but it requires smaller well-prepared training datasets, trains significantly faster and reaches similar accuracy compared to the original method, even improving it on some aspects.

• Wang, Congcong; Cheikh, Faouzi Alaya; Kaaniche, Mounir & Elle, Ole Jacob (2018). Liver surface reconstruction for image guided surgery. Progress in Biomedical Optics and Imaging. ISSN 1605-7422. 1057:105762H6, p. 1–8. doi: 10.1117/12.2297398. Full text in Research Archive Show summary

  In image guided surgery, stereo laparoscopes have been introduced to provide a 3D view of the organs during the laparoscopic intervention. This stereo video could possibly be used for other purposes other than simple viewing: such as depth estimation, 3D rendering of the scene and 3D organ modeling. This paper aims at reconstructing 3D liver surface based on stereo vision technique. The estimated surface of the liver can later be used for registration to preoperative 3D model constructed from MRI/CT scans. For this purpose, we resort to a variational disparity estimation technique by minimizing a global energy function over the entire image. More precisely, based on the gray level and gradient constancy assumptions, a data term and a local as well as a nonlocal smoothness terms are defined to build the cost function. The latter is minimized, by using an appropriate optimization technique, to estimate the disparity map. In order to reduce the disparity search range and the influence of noise, the global variational approach is performed on the coarsest level of the multi-resolution pyramidal representation of the stereo images. Then the obtained low-resolution disparity map is up-sampled with a modified joint bilateral filtering method to the original scale. In vivo liver datasets with ground truth is difficult to obtain, so the proposed method is evaluated quantitatively on two cardiac phantom datasets from Hamlyn Center achieving an accuracy of about 2.2 mm for heart1 and 2.1 mm for heart2. Reconstructed points up to 97% for heart1 and 100% for heart2 are obtained. Qualitative validation on in vivo porcine procedure's liver datasets has shown that our proposed method can estimate the untextured surfaces geometry well.

• Teatini, Andrea; Wang, Congcong; Palomar, Rafael; Cheikh, Faouzi Alaya; Beghdadi, Azeddine & Edwin, Bjørn [Show all 7 contributors for this article] (2018). Validation of stereo vision based liver surface reconstruction for image guided surgery. In Nussbaum, Peter & George, Sony (Ed.), 2018 Colour and Visual Computing Symposium (CVCS); 19th and 20th September, 2018, Gjøvik, Norway. IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781538656457. doi: 10.1109/CVCS.2018.8496589. Full text in Research Archive Show summary

  Image guided surgery systems aim to provide navigation to surgeons in order to improve accuracy and safety of the procedures. Through stereo reconstruction algorithms, it is possible to generate 3D surfaces intra-operatively by means of a laparoscopic stereo-camera. This study aims to setup a simulation system and quantitatively validate a recent proposed reconstruction algorithm with application to laparoscopic liver resection surgery. The intra-operative surface will be used to guide model to patient registration. This will also improve accuracy of the navigation by correcting intra-operative deformations of the liver, such as those due to pneumoperitoneum. The accuracy results of the reconstruction method was found to be 3.7±0.8 mm in terms of Hausdorff distance. The validation therefore indicates the feasibility and accuracy of the surface reconstruction method.

• Wang, Congcong; Cheikh, Faouzi Alaya; Kaaniche, Mounir; Beghdadi, Azeddine & Elle, Ole Jacob (2018). Variational based smoke removal in laparoscopic images. Biomedical engineering online. 17:139(139), p. 1–10. doi: 10.1186/s12938-018-0590-5. Full text in Research Archive Show summary

  Background In laparoscopic surgery, image quality can be severely degraded by surgical smoke, which not only introduces errors for the image processing algorithms (used in image guided surgery), but also reduces the visibility of the observed organs and tissues. To overcome these drawbacks, this work aims to remove smoke in laparoscopic images using an image preprocessing method based on a variational approach. Methods In this paper, we present the physical smoke model where the degraded image is separated into two parts: direct attenuation and smoke veil and propose an efficient variational-based desmoking method for laparoscopic images. To estimate the smoke veil, the proposed method relies on the observation that smoke veil has low contrast and low inter-channel differences. A cost function is defined based on this prior knowledge and is solved using an augmented Lagrangian method. The obtained smoke veil is then subtracted from the original degraded image, resulting in the direct attenuation part. Finally, the smoke free image is computed using a linear intensity transformation of the direct attenuation part. Results The performance of the proposed method is evaluated quantitatively and qualitatively using three datasets: two public real smoked laparoscopic datasets and one generated synthetic dataset. No-reference and reduced-reference image quality assessment metrics are used with the two real datasets, and show that the proposed method outperforms the state-of-the-art ones. Besides, standard full-reference ones are employed with the synthetic dataset, and indicate also the good performance of the proposed method. Furthermore, the qualitative visual inspection of the results shows that our method removes smoke effectively from the laparoscopic images. Conclusion All the obtained results show that the proposed approach reduces the smoke effectively while preserving the important perceptual information of the image. This allows to provide a better visualization of the operation field for surgeons and improve the image guided laparoscopic surgery procedure.

• Brun, Henrik; Bugge, Robin Antony Birkeland; Suther, Lene Kathrine R; Birkeland, Sigurd; Kumar, Raj & Pelanis, Egidijus [Show all 7 contributors for this article] (2018). Mixed reality holograms for heart surgery planning: first user experience in congenital heart disease. European Heart Journal-Cardiovascular Imaging. ISSN 2047-2404. 0, p. 1–6. doi: 10.1093/ehjci/jey184. Full text in Research Archive Show summary

  Aims: Proof of concept and feasibility study for preoperative diagnostic use of mixed reality (MR) holograms of individual 3D heart models from standard cardiac computed tomography angiograms (CTA) images. Optimal repair for complex congenital heart disease poses high demands on 3D anatomical imagination. Three-dimensional printed heart models are increasingly used for improved morphological understanding during surgical and interventional planning. Holograms are a dynamic and interactive alternative, probably with wider applications. Methods and results: A 3D heart model was segmented from CTA images in a patient with double outlet right ventricle and transposition of the great arteries (DORV-TGA). The hologram was visualized in the wearable MR platform HoloLens® for 36 paediatric heart team members who filled out a diagnostic and quality rating questionnaire. Morphological and diagnostic output from the hologram was assessed and the 3D experience was evaluated. Locally developed app tools such as hologram rotation, scaling, and cutting were rated. Anatomy identification and diagnostic output was high as well as rating of 3D experience. Younger and female users rated the app tools higher. Conclusion: This preliminary study demonstrates that MR holograms as surgical planning tool for congenital heart disease may have a high diagnostic value and contribute to understanding complex morphology. The first users experience of the hologram presentation was found to be very positive, with a preference among the female and the younger users. There is potential for improvement of the hologram manipulation tools.

• Wang, Congcong; Sharma, Vivek; Fan, Yu; Cheikh, Faouzi Alaya; Beghdadi, Azeddine & Elle, Ole Jacob [Show all 7 contributors for this article] (2018). Can image enhancement be beneficial to find smoke images in laparoscopic surgery? Color and Imaging Conference (CIC). ISSN 2166-9635. 2018-November, p. 163–168. doi: 10.2352/ISSN.2169-2629.2018.26.163. Full text in Research Archive Show summary

  Laparoscopic surgery has a limited field of view. Laser ablation in a laproscopic surgery causes smoke, which inevitably influences the surgeon's visibility. Therefore, it is of vital importance to remove the smoke, such that a clear visualization is possible. In order to employ a desmoking technique, one needs to know beforehand if the image contains smoke or not, to this date, there exists no accurate method that could classify the smoke/non-smoke images completely. In this work, we propose a new enhancement method which enhances the informative details in the RGB images for discrimination of smoke/non-smoke images. Our proposed method utilizes weighted least squares optimization framework (WLS). For feature extraction, we use statistical features based on bivariate histogram distribution of gradient magnitude (GM) and Laplacian of Gaussian (LoG). We then train a SVM classifier with binary smoke/non-smoke classification task. We demonstrate the effectiveness of our method on Cholec80 dataset. Experiments using our proposed enhancement method show promising results with improvements of 4% in accuracy and 4% in FI-Score over the baseline performance of RGB images. In addition, our approach improves over the saturation histogram based classification methodologies Saturation Analysis (SAN) and Saturation Peak Analysis (SPA) by 1/5% and 1/6% in accuracy/F1-Score metrics. We can employ our enhancement method in replacement of RGB images for classifier training e.g., CNN architectures, which in turn can lead to more accurate classification. Code will be released for public use.

• Miseikis, Justinas; Knobelreiter, Patrick; Brijacak, Inka; Yahyanejad, Saeed; Glette, Kyrre & Elle, Ole Jacob [Show all 7 contributors for this article] (2018). Robot localisation and 3D position estimation using a free-moving camera and cascaded convolutional neural networks, 2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM 2018). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781538618554. p. 181–187. doi: 10.1109/AIM.2018.8452236. Full text in Research Archive Show summary

  Many works in collaborative robotics and human-robot interaction focuses on identifying and predicting human behaviour while considering the information about the robot itself as given. This can be the case when sensors and the robot are calibrated in relation to each other and often the reconfiguration of the system is not possible, or extra manual work is required. We present a deep learning based approach to remove the constraint of having the need for the robot and the vision sensor to be fixed and calibrated in relation to each other. The system learns the visual cues of the robot body and is able to localise it, as well as estimate the position of robot joints in 3D space by just using a 2D color image. The method uses a cascaded convolutional neural network, and we present the structure of the network, describe our own collected dataset, explain the network training and achieved results. A fully trained system shows promising results in providing an accurate mask of where the robot is located and a good estimate of its joints positions in 3D. The accuracy is not good enough for visual servoing applications yet, however, it can be sufficient for general safety and some collaborative tasks not requiring very high precision. The main benefit of our method is the possibility of the vision sensor to move freely. This allows it to be mounted on moving objects, for example, a body of the person or a mobile robot working in the same environment as the robots are operating in.

• Miseikis, Justinas; Brijacak, Inka; Yahyanejad, Saeed; Glette, Kyrre; Elle, Ole Jacob & Tørresen, Jim (2018). Multi-Objective Convolutional Neural Networks for Robot Localisation and 3D Position Estimation in 2D Camera Images, 2018 15th International Conference on Ubiquitous Robots (UR 2018). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781538663356. p. 597–603. doi: 10.1109/URAI.2018.8441813. Full text in Research Archive Show summary

  The field of collaborative robotics and humanrobot interaction often focuses on the prediction of human behaviour, while assuming the information about the robot setup and configuration being known. This is often the case with fixed setups, which have all the sensors fixed and calibrated in relation to the rest of the system. However, it becomes a limiting factor when the system needs to be reconfigured or moved. We present a deep learning approach, which aims to solve this issue. Our method learns to identify and precisely localise the robot in 2D camera images, so having a fixed setup is no longer a requirement and a camera can be moved. In addition, our approach identifies the robot type and estimates the 3D position of the robot base in the camera image as well as 3D positions of each of the robot joints. Learning is done by using a multiobjective convolutional neural network with four previously mentioned objectives simultaneously using a combined loss function. The multi-objective approach makes the system more flexible and efficient by reusing some of the same features and diversifying for each objective in lower layers. A fully trained system shows promising results in providing an accurate mask of where the robot is located and an estimate of its base and joint positions in 3D. We compare the results to our previous approach of using cascaded convolutional neural networks.

• Teatini, Andrea; Frutos, Javier Perez de; Langø, Thomas; Edwin, Bjørn & Elle, Ole Jacob (2018). Assessment and comparison of target registration accuracy in surgical instrument tracking technologies. IEEE Engineering in Medicine and Biology Society. Conference Proceedings. ISSN 1557-170X. 40th, p. 1845–1848. doi: 10.1109/EMBC.2018.8512671. Full text in Research Archive Show summary

  Image guided surgery systems aim to support surgeons by providing reliable pre-operative and intra-operative imaging of the patient combined with the corresponding tracked instrument location. The image guidance is based on a combination of medical images, such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT) and Ultrasonography (US), and surgical instrument tracking. For this reason, tracking systems are of great importance as they determine location and orientation of the equipment used by surgeons. Assessment of the accuracy of these tracking systems is hence of the utmost importance to determine how much error is introduced in image guided surgery only due to tracking inaccuracy. Thus, this study aimed to compare in a surgical Operating Room (OR) accuracy of the two most used tracking systems, Optical Tracking (OT) and Electromagnetic Tracking (EMT), in terms of Target Registration Error (TRE) assessment at multiple distances from the target position. Results of the experiments show that optical tracking performs more accurately in tracking the instrument tip than electromagnetic tracking in the experimental conditions. This was tested using a phantom designed for accuracy measurement in a wide variety of positions and orientations. Nevertheless, EMT remains a viable option for flexible instruments, due to its reliability in tracking without the need for line of sight.

• Frutos, Javier Perez de; Hofstad, Erlend Fagertun; Solberg, Ole Vegard; Tangen, Geir Arne; Lindseth, Frank & Langø, Thomas [Show all 8 contributors for this article] (2018). Laboratory test of Single Landmark registration method for ultrasound-based navigation in laparoscopy using an open-source platform. International Journal of Computer Assisted Radiology and Surgery. ISSN 1861-6410. 13(12), p. 1927–1936. doi: 10.1007/s11548-018-1830-7. Full text in Research Archive Show summary

  Purpose Test the feasibility of the novel Single Landmark image-to-patient registration method for use in the operating room for future clinical trials. The algorithm is implemented in the open-source platform CustusX, a computer-aided intervention research platform dedicated to intraoperative navigation and ultrasound, with an interface for laparoscopic ultrasound probes. Methods The Single Landmark method is compared to fiducial landmark on an IOUSFAN (Kyoto Kagaku Co., Ltd., Japan) soft tissue abdominal phantom and T2 magnetic resonance scans of it. Results The experiments show that the accuracy of the Single Landmark registration is good close to the registered point, increasing with the distance from this point (12.4 mm error at 60 mm away from the registered point). In this point, the registration accuracy is mainly dominated by the accuracy of the user when clicking on the ultrasound image. In the presented set-up, the time required to perform the Single Landmark registration is 40% less than for the FLRM. Conclusion The Single Landmark registration is suitable for being integrated in a laparoscopic workflow. The statistical analysis shows robustness against translational displacements of the patient and improvements in terms of time. The proposed method allows the clinician to accurately register lesions intraoperatively by clicking on these in the ultrasound image provided by the ultrasound transducer. The Single Landmark registration method can be further combined with other more accurate registration approaches improving the registration at relevant points defined by the clinicians.

• Aghayan, Davit; Pelanis, Egidijus; Fretland, Åsmund Avdem; Kazaryan, Airazat M.; Sahakyan, Mushegh & Røsok, Bård Ingvald [Show all 10 contributors for this article] (2017). Laparoscopic parenchyma-sparing liver resection for colorectal metastases. Radiology and Oncology. ISSN 1318-2099. 52(1), p. 36–41. doi: 10.1515/raon-2017-0046. Full text in Research Archive Show summary

  Background. Laparoscopic liver resection (LLR) of colorectal liver metastases (CLM) is increasingly performed in specialized centers. While there is a trend towards a parenchyma-sparing strategy in multimodal treatment for CLM, its role is yet unclear. In this study we present short- and long-term outcomes of laparoscopic parenchyma-sparing liver resection (LPSLR) at a single center. Patients and methods. LLR were performed in 951 procedures between August 1998 and March 2017 at Oslo University Hospital, Oslo, Norway. Patients who primarily underwent LPSLR for CLM were included in the study. LPSLR was defined as non-anatomic hence the patients who underwent hemihepatectomy and sectionectomy were excluded. Perioperative and oncologic outcomes were analyzed. The Accordion classification was used to grade postoperative complications. The median follow-up was 40 months. Results. 296 patients underwent primary LPSLR for CLM. A single specimen was resected in 204 cases, multiple resections were performed in 92 cases. 5 laparoscopic operations were converted to open. The median operative time was 134 minutes, blood loss was 200 ml and hospital stay was 3 days. There was no 90-day mortality in this study. The postoperative complication rate was 14.5%. 189 patients developed disease recurrence. Recurrence in the liver occurred in 146 patients (49%), of whom 85 patients underwent repeated surgical treatment (liver resection [n = 69], ablation [n = 14] and liver transplantation [n = 2]). Five-year overall survival was 48%, median overall survival was 56 months. Conclusions. LPSLR of CLM can be performed safely with the good surgical and oncological results. The technique facilitates repeated surgical treatment, which may improve survival for patients with CLM.

• Kumar, Rahul Prasanna; Barkhatov, Leonid; Edwin, Bjørn; Albregtsen, Fritz & Elle, Ole Jakob (2017). Portal and hepatic vein segmentation with leak restriction: A pilot study. International Federation for Medical and Biological Engineering Proceedings. ISSN 1680-0737. 65, p. 823–826. doi: 10.1007/978-981-10-5122-7_206. Full text in Research Archive
• Palomar, Rafael; Gómez-Luna, Juan; Cheikh, Faouzi Alaya; Olivares-Bueno, Joaquín & Elle, Ole Jakob (2017). High-Performance Computation of Bézier Surfaces on Parallel and Heterogeneous Platforms. International journal of parallel programming. ISSN 0885-7458. p. 1–28. doi: 10.1007/s10766-017-0506-1. Full text in Research Archive Show summary

  Bézier surfaces are mathematical tools employed in a wide variety of applications. Some works in the literature propose parallelization strategies to improve performance for the computation of Bézier surfaces. These approaches, however, are mainly focused on graphics applications and often are not directly applicable to other domains. In this work, we propose a new method for the computation of Bézier surfaces, together with approaches to efficiently map the method onto different platforms (CPUs, discrete and integrated GPUs). Additionally, we explore CPU–GPU cooperation mechanisms for computing Bézier surfaces using two integrated heterogeneous systems with different characteristics. An exhaustive performance evaluation—including different data-types, rendering and several hardware platforms—is performed. The results show that our method achieves speedups as high as 3.12x (double-precision) and 2.47x (single-precision) on CPU, and 3.69x (double-precision) and 13.14x (single-precision) on GPU compared to other methods in the literature. In heterogeneous platforms, the CPU–GPU cooperation increases the performance up to 2.09x with respect to the GPU-only version. Our method and the associated parallelization approaches can be easily employed in domains other than computer-graphics (e.g., image registration, bio-mechanical modeling and flow simulation), and extended to other Bézier formulations and Bézier constructions of higher order than surfaces.

• Krogh, Magnus Reinsfelt; Nghiem, Giang Minh; Halvorsen, Per Steinar; Elle, Ole Jakob; Grymyr, Ole-Johannes & Hoff, Lars [Show all 7 contributors for this article] (2017). Gravity compensation method for combined accelerometer and gyro sensors used in cardiac motion measurements. Annals of Biomedical Engineering. ISSN 0090-6964. 45(5), p. 1292–1304. doi: 10.1007/s10439-017-1798-4. Full text in Research Archive Show summary

  A miniaturized accelerometer fixed to the heart can be used for monitoring of cardiac function. However, an accelerometer cannot differentiate between acceleration caused by motion and acceleration due to gravity. The accuracy of motion measurements is therefore dependent on how well the gravity component can be estimated and filtered from the measured signal. In this study we propose a new method for estimating the gravity, based on strapdown inertial navigation, using a combined accelerometer and gyro. The gyro was used to estimate the orientation of the gravity field and thereby remove it. We compared this method with two previously proposed gravity filtering methods in three experimental models using: (1) in silico computer simulated heart motion; (2) robot mimicked heart motion; and (3) in vivo measured motion on the heart in an animal model. The new method correlated excellently with the reference (r2 > 0.93) and had a deviation from reference peak systolic displacement (6.3 ± 3.9 mm) below 0.2 ± 0.5 mm for the robot experiment model. The new method performed significantly better than the two previously proposed methods (p < 0.001). The results show that the proposed method using gyro can measure cardiac motion with high accuracy and performs better than existing methods for filtering the gravity component from the accelerometer signal.

• Palomar, Rafael; Cheikh, Faouzi Alaya; Edwin, Bjørn; Fretland, Åsmund Avdem; Beghdadi, Azeddine & Elle, Ole Jakob (2017). A novel method for planning liver resections using deformable Bezier surfaces and distance maps. Computer Methods and Programs in Biomedicine. ISSN 0169-2607. 144, p. 135–145. doi: 10.1016/j.cmpb.2017.03.019. Full text in Research Archive Show summary

  Background and Objective: For more than a decade, computer-assisted surgical systems have been helping surgeons to plan liver resections. The most widespread strategies to plan liver resections are: drawing traces in individual 2D slices, and using a 3D deformable plane. In this work, we propose a novel method which requires low level of user interaction while keeping high flexibility to specify resections. Methods: Our method is based on the use of Bézier surfaces, which can be deformed using a grid of control points, and distance maps as a base to compute and visualize resection margins (indicators of safety) in realtime. Projection of resections in 2D slices, as well as computation of resection volume statistics are also detailed. Results: The method was evaluated and compared with state-of-the-art methods by a group of surgeons (n = 5, 5–31 years of experience). Our results show that theproposed method presents planning times as low as state-of-the-art methods (174 s median time) with high reproducibility of results in terms of resected volume. In addition, our method not only leads to smooth virtual resections easier to perform surgically compared to other state-of-the-art methods, but also shows superior preservation of resection margins. Conclusions: Our method provides clinicians with a robust and easy-to-use method for planning liver resections with high reproducibility, smoothness of resection and preservation of resection margin. Our results indicate the ability of the method to represent any type of resection and being integrated in real clinical work-flows.

• Sdiri, Bilel; Beghdadi, Azeddine; Cheikh, Faouzi Alaya; Pedersen, Marius & Elle, Ole Jakob (2016). An adaptive contrast enhancement method for stereo endoscopic images combining binocular just noticeable difference model and depth information. IS&T International Symposium on Electronic Imaging Science and Technology. doi: 10.2352/ISSN.2470-1173.2016.13.IQSP-212. Full text in Research Archive
• Ševo, Igor; Avramovic, Aleksej; Balasingham, Ilangko; Elle, Ole Jakob; Bergsland, Jacob & Aabakken, Lars (2016). Edge density based automatic detection of inflammation in colonoscopy videos. Computers in Biology and Medicine. ISSN 0010-4825. 72, p. 138–150. doi: 10.1016/j.compbiomed.2016.03.017. Full text in Research Archive
• Mathiassen, Kim; Fjellin, Jørgen Enger; Glette, Kyrre Harald; Hol, Per Kristian & Elle, Ole Jakob (2016). An Ultrasound Robotic System Using the Commercial Robot UR5. Frontiers in Robotics and AI. 3. doi: 10.3389/frobt.2016.00001. Full text in Research Archive Show summary

  The use of robots in health care has increased dramatically over the last decade. One area of research has been to use robots to conduct ultrasound examinations, either controlled by a physician or autonomously. This paper examines the possibility of using the commercial robot UR5 from Universal Robots to make a tele-operated robotic ultrasound system. Physicians diagnosing patients using ultrasound probes are prone to repetitive strain injuries, as they are required to hold the probe in uncomfortable positions and exert significant static force. The main application for the system is to relieve the physician of this strain by letting the them control a robot that holds the probe. A set of requirements for the system is derived from the state-of-the-art systems found in the research literature. The system is developed through a low-level interface for the robot, effectively building a new software framework for controlling it. Compliance force control and forward flow haptic control of the robot was implemented. Experiments are conducted to quantify the performance of the two control schemes. The force control is estimated to have a bandwidth of 16.6 Hz, while the haptic control is estimated to have a bandwidth of 65.4 Hz for the position control of the slave and 13.4 Hz for the force control of the master. Overall, the system meets the derived requirements and the main conclusion is that it is feasible to use the UR5 robot for robotic ultrasound applications.

• Mathiassen, Kim; Glette, Kyrre & Elle, Ole Jakob (2016). Visual servoing of a medical ultrasound probe for needle insertion. IEEE International Conference on Robotics and Automation (ICRA). ISSN 1050-4729. 2016-June, p. 3426–3433. doi: 10.1109/ICRA.2016.7487520. Full text in Research Archive
• Miseikis, Justinas; Glette, Kyrre; Elle, Ole Jakob & Tørresen, Jim (2016). Automatic Calibration of a Robot Manipulator and Multi 3D Camera System. In Hirata, Yasuhisa (Eds.), Proc. of 2016 IEEE/SICE International Symposium on System Integration. IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781509033294. p. 735–741. doi: 10.1109/SII.2016.7844087. Full text in Research Archive
• Palomar, Rafael; Cheikh, Faouzi Alaya; Beghdadi, Azeddine & Elle, Ole Jakob (2016). Intra-operative modeling of the left Atrium: A simulation approach using poisson surface reconstruction. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 9805, p. 354–365. doi: 10.1007/978-3-319-43775-0_32. Full text in Research Archive
• Mathiassen, Kim; Dall'Alba, Diego; Muradore, Riccardo; Fiorini, Paolo & Elle, Ole Jakob (2016). Robust real-time needle tracking in 2-D ultrasound images using statistical filtering. IEEE Transactions on Control Systems Technology. ISSN 1063-6536. PP(99), p. 1–13. doi: 10.1109/TCST.2016.2587733. Full text in Research Archive Show summary

  Percutaneous image-guided tumor ablation is a minimally invasive surgical procedure for the treatment of malignant tumors using a needle-shaped ablation probe. Automating the insertion of a needle by using a robot could increase the accuracy and decrease the execution time of the procedure. Extracting the needle tip position from the ultrasound (US) images is of paramount importance for verifying that the needle is not approaching any forbidden regions (e.g., major vessels and ribs), and could also be used as a direct feedback signal to the robot inserting the needle. A method for estimating the needle tip has previously been developed combining a modified Hough transform, image filters, and machine learning. This paper improves that method by introducing a dynamic selection of the region of interest in the US images and filtering the tracking results using either a Kalman filter or a particle filter. Experiments where a biopsy needle has been inserted into a phantom by a robot have been conducted, guided by an infrared tracking system. The proposed method has been accurately evaluated by comparing its estimations with the needle tip's positions manually detected by a physician in the US images. The results show a significant improvement in precision and more than 85% reduction of 95th percentile of the error compared with the previous automatic approaches. The method runs in real time with a frame rate of 35.4 frames/s. The increased robustness and accuracy can make our algorithm usable in autonomous surgical systems for needle insertion.

• Palomar, Rafael; Cheikh, Faouzi Alaya; Edwin, Bjørn; Beghdadhi, Azeddine & Elle, Ole Jakob (2016). Surface reconstruction for planning and navigation of liver resections. Computerized Medical Imaging and Graphics. ISSN 0895-6111. 53, p. 30–42. doi: 10.1016/j.compmedimag.2016.07.003. Full text in Research Archive Show summary

  Computer-assisted systems for planning and navigation of liver resection procedures rely on the use of patient-specific 3D geometric models obtained from computed tomography. In this work, we propose the application of Poisson surface reconstruction (PSR) to obtain 3D models of the liver surface with applications to planning and navigation of liver surgery. In order to apply PSR, the introduction of an efficient transformation of the segmentation data, based on computation of gradient fields, is proposed. One of the advantages of PSR is that it requires only one control parameter, allowing the process to be fully automatic once the optimal value is estimated. Validation of our results is performed via comparison with 3D models obtained by state-of-art Marching Cubes incorporating Laplacian smoothing and decimation (MCSD). Our results show that PSR provides smooth liver models with better accuracy/complexity trade-off than those obtained by MCSD. After estimating the optimal parameter, automatic reconstruction of liver surfaces using PSR is achieved keeping similar processing time as MCSD. Models from this automatic approach show an average reduction of 79.59% of the polygons compared to the MCSD models presenting similar smoothness properties. Concerning visual quality, on one hand, and despite this reduction in polygons, clinicians perceive the quality of automatic PSR models to be the same as complex MCSD models. On the other hand, clinicians perceive a significant improvement on visual quality for automatic PSR models compared to optimal (obtained in terms of accuracy/complexity) MCSD models. The median reconstruction error using automatic PSR was as low as 1.03 ± 0.23 mm, which makes the method suitable for clinical applications. Automatic PSR is currently employed at Oslo University Hospital to obtain patient-specific liver models in selected patients undergoing laparoscopic liver resection.

• Miseikis, Justinas; Glette, Kyrre; Elle, Ole Jakob & Tørresen, Jim (2016). Multi 3D Camera Mapping for Predictive and Reflexive Robot Manipulator Trajectory Estimation. In Jin, Yaochu & Kollias, Stefanos (Ed.), Proc. of 2016 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781509042401. doi: 10.1109/SSCI.2016.7850237. Full text in Research Archive
• Kaiser, Mandy; Detert, Markus; Rube, Martin A.; El-Dirdiri, Abubakr El-Tahir Mohamed; Elle, Ole Jakob & Melzer, Andreas [Show all 8 contributors for this article] (2015). Resonant marker design and fabrication techniques for device visualization during interventional magnetic resonance imaging. Biomedizinische Technik. ISSN 1862-278X. 60(2), p. 89–103. doi: 10.1515/bmt-2013-0097. Full text in Research Archive
• Grymyr, Ole-Johannes; Remme, Espen W.; Espinoza, Andreas Westenvik; Skulstad, Helge; Elle, Ole Jakob & Fosse, Erik [Show all 7 contributors for this article] (2015). Assessment of 3D motion increases the applicability of accelerometers for monitoring left ventricular function. Interactive Cardiovascular and Thoracic Surgery. ISSN 1569-9293. 20(3), p. 329–337. doi: 10.1093/icvts/ivu404. Full text in Research Archive
• Kumar, Rahul Prasanna; Albregtsen, Fritz; Reimers, Martin; Edwin, Bjørn; Langø, Thomas & Elle, Ole Jakob (2015). Blood vessel segmentation and centerline tracking using local structure analysis. International Federation for Medical and Biological Engineering Proceedings. ISSN 1680-0737. 45, p. 122–125. doi: 10.1007/978-3-319-11128-5_31. Full text in Research Archive Show summary

  Blood vessel visualization is important for improving planning and navigation of several interventional procedures. In this paper, we present a novel method for simultaneous blood vessel segmentation, centerline tracking and radius estimation. Our method is based on local structure analysis within the connected regions of the blood vessels. The proposed method is mainly divided into trunk analysis and bifurcation analysis. In the trunk analysis, the vessel is segmented using 2D cross-section analysis while in the bifurcation analysis, the vessel is segmented using modified vesselness. Our method was evaluated on angiogram images. When comparing the processing time for finding the blood vessel centerline, our proposed method was found to be on average more than 20 times faster than multiscale vesselness with thinning and more than 7 times faster than our own earlier method of blood vessel centerline extraction. Also, the centerline extraction was found to be accurate with a mean error less than 1 voxel in comparison to the corresponding geometric centers.

• Muradore, Riccardo; Fiorini, Paolo; Akgun, Gokhan; Barkana, Duygun Erol; Bonfe, Marcello & Boriero, Fabrizio [Show all 28 contributors for this article] (2015). Development of a cognitive robotic system for simple surgical tasks. International Journal of Advanced Robotic Systems (IJARS). ISSN 1729-8806. 12:37. doi: 10.5772/60137. Full text in Research Archive
• Greisiger, Ralf; Shallop, Jon K.; Hol, Per Kristian; Elle, Ole Jakob & Jablonski, Greg (2015). Cochlear implantees: Analysis of behavioral and objective measures for a clinical population of various age groups. Cochlear Implants International (CII). ISSN 1467-0100. 16(4), p. 1–19. doi: 10.1080/14670100.2015.1110372. Full text in Research Archive

View all works in NVA

• Engelsen, Peder Christian Grendstad; Heyerdahl, Fridtjof; Maddali, Dharani Dhar; Akhavi, Mehdi Sadat; Undseth, Ragnhild Marie & Elle, Ole Jakob [Show all 7 contributors for this article] (2024). Correction to: Ascending with ultrasound: telementored eFAST in flight—a feasibility study (Emergency Radiology, (2023), 31, 1, (25-31), 10.1007/s10140-023-02186-x). Emergency Radiology. ISSN 1070-3004. 31(4), p. 629–629. doi: 10.1007/s10140-024-02265-7. Full text in Research Archive
• Brattgjerd, Jan Egil; Wiig, Ola & Elle, Ole Jakob (2023). De første erfaringene med hologrammer i anatomiundervisningen. Full text in Research Archive
• Sajadi, Seyed Mohammad Reza; Mathiassen, Kim; Brun, Henrik & Elle, Ole Jakob (2022). Design, Kinematic Modeling, and Validation of a Robotic-Assisted Transesophageal Echocardiography System. Full text in Research Archive Show summary

  Transesophageal Echocardiography is a safe and minimally invasive tool to provide high-quality anatomic and hemodynamic information of the heart and its structures. However, the risk of exposure to a significant level of radiation during combined fluoroscopy and TEE-guided examinations and musculoskeletal disorders due to holding the probe for prolonged periods of time are two main shortcomings of using TEE manually. This paper presents the design, kinematic modeling, and validation of a novel 6 DOF robotic-assisted transesophageal echocardiography system to overcome the above mentioned shortcomings. The robotic system has been designed based on a commercially available TEE probe with 5 actuated Degrees of freedom on the TEE handle and an extra DOF to control the imaging plane. The constant curvature model is employed for the forward and inverse kinematic modeling of the robotic system. The kinematic model and the performance of the robotic system are validated with an experimental setup using a 6DOF OptiTrack real-time tracking system.

• Miseikis, Justinas; Brijacak, Inka; Yahyanejad, Saeed; Glette, Kyrre; Elle, Ole Jacob & Tørresen, Jim (2019). Two-Stage Transfer Learning for Heterogeneous Robot Detection and 3D Joint Position Estimation in a 2D Camera Image Using CNN. Full text in Research Archive
• Mathiassen, Kim; Glette, Kyrre & Elle, Ole Jakob (2016). Visual servoing of a medical ultrasound probe for needle insertion. Full text in Research Archive
• Kjølleberg, Even; Kårstad, Gry; Reuterdahl, Anne-Cathrine; Elle, Ole Jakob; Tørresen, Jim & Mathiassen, Kim (2015). Robotlegene kommer! [Internet]. NRK. Full text in Research Archive
• Frutos, Javier Perez de; Langø, Thomas; Lindseth, Frank; Elle, Ole Jakob & Reinertsen, Ingerid Reime (2023). Registration techniques for navigation in laparascopic surgery. Norges teknisk-naturvitenskapelige universitet. ISSN 9788232675210. Full text in Research Archive Show summary

  In minimally invasive surgery, where a direct sight of the treated organ or structures is not feasible but with a laparoscope, image guided systems have been developed to provide the surgeon with real time information of the surgical area, in the form of images of the anatomy of the patient, and surgical navigation. In order to do so, image-to-patient and image-to-image registration techniques are required to bring physical and image information on to the same system of reference. Essentially, both methods rely on point-to-point correspondence, using a collection of selected features on both the fixed and target domains. In the case of image-to-patient, position landmarks are used. Whereas in image-to-image, features related to the intensity information of the image, or the topology of segmentations, are the usual approach. Nevertheless, these procedures require time and are susceptible to human error. In image-to-patient in particular, when sampling landmarks on the patient anatomy. Image-to-image methods on the other hand, operate between different images and virtual models, being complex to implement and to achieve good results, especially in multi-modal setups. The presented research is framed in the field of hepatocellular carcinoma surgical treatment, the most common type of liver cancer. Wedge resection is the usual choice to remove the cancerous cells, aiming to spare as much healthy tissue as possible while ensuring a complete extraction of the tumor. However, during the intervention, the organ undergoes major deformation due to mobilisation and detachment of the abdominal wall. This deformation is not reflected in the pre-operative images, showing an inaccurate location of relevant anatomical structures as well as the target lesion. Through a laparoscope and laparoscopic ultrasound, the surgeon can build a mental image of the anatomy of the patient during the intervention, adding extra strain on the practitioner. Registration techniques can leverage the procedure outcome and ease the surgery by aligning the virtual models of the patient to the situation on the surgical table. Furthermore, by updating the pre-operative model with the real-time information acquired when inspecting the organ. In particular, the present study focuses on the use of registration for navigation in minimally invasive interventions, including image-to-patient and image-to-image registration. Main contributions include the evaluation of tracking technologies for surgical navigation; a novel image-to-patient registration method (single landmark registration method) which can take advantage of the laparoscopic ultrasound to improve the registration of the pre-operative images; the use of deep learning for image-to-image registration in medical applications, and development of new training methods; and the research on the influence of human accuracy in landmark sampling during image-to-patient registration, in augmented reality applications for surgical navigation.

• Elle, Ole Jakob; Glette, Kyrre; Høvin, Mats Erling; Hol, Per Kristian & Mathiassen, Kim (2017). A semi-autonomous robotic system for needle tracking and visual servoing using 2D medical ultrasound. Universitetet i Oslo. Full text in Research Archive Show summary

  The main aim of the thesis is to contribute to a semi-autonomous active sensing system for needle tracking during percutaneous needle insertion. The objective is twofold: first to develop a system for acquiring ultrasound images, then to analyze these images in order to track the needle. The thesis is thus divided into two main lines of research: (1) investigate whether it is feasible to make a robotic system for acquiring ultrasound images using commercially available components; (2) investigate how to track a needle using 2D ultrasound images and possibly other sensor information in real-time. In this thesis we have developed an ultrasound robotic system using the UR5 robot from Universal Robots, based on a set of requirements derived from the state of the art in the literature. The system has both force and haptic control. We developed and tested several novel methods for tracking a needle in an ultrasound image. In addition, a novel visual servoing method has been developed to ensure that the needle lies in the imaging plane of the ultrasound probe during insertions. The thesis concludes that it is feasible to make an ultrasound robotic system using commercially available components and shows that such a system meets the derived requirements. It also concludes that the novel needle tracking methods presented are both accurate and precise compared to previous methods presented in the literature. It is also shown that using velocity measurements from a needle insertion robot significantly improves the both the accuracy and precision, compared to only using ultrasound images. The visual servoing method shows promise for solving the alignment problem of keeping the needle in the imaging plane of the ultrasound probe.

View all works in NVA