Life science today involves computational analysis of a large amount and variety of data, such as volumetric data acquired by state-of-the-art microscopes, or mesh data resulting from analysis of such data or simulations. Visualisation is often the first step in making sense of the data, and a crucial part of building and debugging analysis pipelines. It is therefore important that visualisations can be quickly prototyped, as well as developed or embedded into full applications. In order to better judge spatiotemporal relationships, immersive hardware, such as Virtual or Augmented Reality (VR AR) headsets and associated controllers are becoming invaluable tools. In this work we introduce scenery, a flexible VR AR visualisation framework for the Java VM that can handle mesh and arbitrarily large volumetric data, containing multiple views, timepoints, and color channels. scenery is free and open-source software, works on all major platforms and uses the Vulkan or OpenGL rendering APIs. We introduce scenery's main features and detail example applications, such as its use in the biomedical image analysis software Fiji, or for visualising agent-based simulations.
Background Current imaging methods such as Magnetic Resonance Imaging (MRI), Confocal microscopy, Electron Microscopy (EM) or Selective Plane Illumination Microscopy (SPIM) yield three-dimensional (3D) data sets in need of appropriate computational methods for their analysis. The reconstruction, segmentation and registration are best approached from the 3D representation of the data set. The Visualization Toolkit (VTK) is an open source, permissively licensed, cross-platform toolkit for scientific data processing, visualization, and data analysis. It is over two decades old, originally developed for a very different graphics card architecture. Modern graphics cards feature fully programmable, highly parallelized architectures with large core counts. VTK’s rendering code was rewritten to take advantage of modern graphics cards, maintaining most of the toolkit’s programming interfaces. This offers the opportunity to compare the performance of old and new rendering code on the same systems cards. Significant improvements in rendering speeds and memory footprints mean that scientific data can be visualized in greater detail than ever before. The widespread use of VTK means that these improvements will reap significant benefits. To the editor: Current state-of-the-art light sheet microscopes rely on sophisticated control software to perform the acquisition of gigabytes of image data per second over the course of hours or even days. Typically the microscopes acquire data in a first step, and only in a second step this data is processed and visualized offline. The delay between data acquisition and data assessment wastes time and storage space. Technology that makes it possible to view and assess the data during imaging would offer significant advantages. However, even the most advanced microscopes only display the latest image plane acquired or projection while the raw volumetric data is saved to disk [1–4]. Visualization applications nowadays not only face increasingly larger datasets, but have to solve increasingly complex research questions. They often require more than a single algorithm and consequently a software solution will exceed the possibilities of simple research prototypes. Well-established systems intended for such complex visual analysis purposes have usually been designed for classical, mesh-based graphics approaches. For particle-based data, however, existing visualization frameworks are too generic—e.g. lacking possibilities for consistent low-level GPU optimization for high-performance graphics—and at the same time are too limited—e.g. by enforcing the use of structures suboptimal for some computations. Thus, we developed the system software MegaMol for visualization research on particle-based data. On the one hand, flexible data structures and functional module design allow for easy adaption to changing research questions, e.g. studying vapors in thermodynamics, solid material in physics, or complex functional macromolecules like proteins in biochemistry. Therefore, MegaMol is designed as a development framework. On the other hand, common functionality for data handling and advanced rendering implementations are available and beneficial for all applications. We present several case studies of work implemented using our system as well as a comparison to other freely available or open source systems.