Hyperspectral imaging is a form of spectral imaging wherein information from across the electromagnetic spectrum is collected in many narrow spectral bands and processed. From the different spectral images that are collected, information of the objects that are imaged can be derived. For example, as certain objects leave unique spectral signatures in images which may even depend on the status of the object, information obtained by multi-spectral imaging can provide information regarding the presence and/or status of objects in a region that is imaged. After selection of a spectral range that will be imaged, as spectral images in this complete spectral range can be acquired, one does not need to have detailed prior knowledge of the objects, and post-processing may allow to obtain all available information. Whereas originally hyperspectral remote sensing was mainly used for mining and geology, other applications such as ecology, agriculture and surveillance also make use of the imaging technique.
It is known to use photogrammetric techniques to infer three-dimensional information, in particular elevation information, from the acquired two-dimensional images. An example of such a technique is disclosed in Alsadik, B. S., Gerke, M., & Vosselman, G. (2012), “Optimal Camera Network Design For 3D Modeling Of Cultural Heritage”, ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, I-3, 7-12.
Some agricultural and ecological applications are known wherein hyperspectral remote sensing is used, e.g. for monitoring the development and health of crops, grape variety detection, monitoring individual forest canopies, detection of the chemical composition of plants as well as early detection of disease outbreaks, monitoring of impact of pollution and other environmental factors, etc. are some of the agricultural applications of interest. Hyperspectral imaging also is used for studies of inland and coastal waters for detecting biophysical properties. In mineralogy, detection of valuable minerals such as gold or diamonds can be performed using hyperspectral sensing, but also detection of oil and gas leakage from pipelines and natural wells are envisaged. Detection of soil composition on earth or even at other planets, asteroids or comets also are possible applications of hyperspectral imaging. In surveillance, hyperspectral imaging can for example be performed for detection of living creatures.
International patent application publication WO 2011/073430 A1, in the name of the present applicant, discloses a sensing device for obtaining geometric referenced multi-spectral image data of a region of interest in relative movement with respect to the sensing device. The sensing device comprises a first two dimensional sensor element and a spectral filter. The spectral filter and the first sensor element are arranged for obtaining spectral information at a first wavelength or wavelength range using a part of the first sensor element and for obtaining spectral information at a second wavelength or wavelength range using another part of the first sensor element. As a result of this arrangement, different parts of a single image acquired with the first sensor will represent the imaged scenery as seen is radiation of a different respective wavelength.
To date, there is no satisfactory way to apply the aforementioned photogrammetric techniques directly to multi-spectral or hyperspectral images such as those acquired by means of the first sensor of WO 2011/073430 A1. Nevertheless, full photogrammetric information about the acquired multi-spectral or hyperspectral images is essential to correctly stitch together an image of the scanned scenery in any particular wavelength band.
In WO 2011/073430 A1, this problem is solved by providing a second sensor on the same substrate, which simultaneously acquires panchromatic images of the area imaged by the first sensor. The panchromatic images are used to perform 3D modeling, and the 3D model information is subsequently transposed to the composite images of the scenery in the different wavelength bands that are covered by the multi-spectral or hyperspectral first sensor.
Unpublished international patent application no. PCT/EP2015/065523, in the name of the present applicant, describes a sensing device for obtaining geometric referenced multi-spectral image data of a region of interest in relative movement with respect to the sensing device, the sensing device comprising: at least a first two-dimensional sensor element, the sensing device being adapted for obtaining subsequent multi-spectral images during said relative motion of the region of interest with respect to the sensing device thus providing distinct spectral information for different parts of a region of interest using the first sensor element; a second two-dimensional sensor element, the sensing device being adapted for providing, using the second sensor element, an image of the region of interest for generating geometric referencing information to be coupled to the distinct spectral information; the first two-dimensional sensor element being operable to capture a first sequence of frames at a first frame rate and the second two-dimensional sensor element being operable to capture a second sequence of frames at a second frame rate; wherein the first frame rate is higher than the second frame rate; and wherein the sensing device further comprises a processor configured to generate intermediate geometric referencing information to be coupled to frames of said first sequence of frames for which no synchronous frame from said second sequence of frames is available, said intermediate geometric referencing information being derived from one or more temporally similar frames from said second sequence of frames. In this way, the system of PCT/EP2015/065523 is able to reduce the number of panchromatic images required to perform geometric referencing, relative to the system of WO 2011/073430 A1, by means of interpolation or extrapolation. However, the system of PCT/EP2015/065523 does not change the assumption that panchromatic images remain necessary.
The article by inventor of the present invention A. Sima et al., “Semi-Automated Registration of Close-Range Hyperspectral Scans Using Oriented Digital Camera Imagery And A 3D Model”, The Photogrammetric Record, vo. 29 (2104), no. 145, discloses a semi-automated method for registering terrestrial panoramic hyperspectral imagery with LIDAR models and conventional digital photography. The method relies on finding corresponding points between images acquired in significantly different parts of the electromagnetic spectrum, from different viewpoints, and with different spatial resolution and geometric projections. The article recognizes that the number of homologous points that can be matched between a hyperspectral band and a covering (panchromatic) digital camera image varies with the wavelength, and proposes a technique that relies on a judicial selection of spectral bands in which a maximum number of points can be matched.
Unpublished international patent application no. PCT/EP2015/065524, in the name of the present applicant, describes a method for performing photogrammetric 3D reconstruction of objects imaged in a sequence of images, the images containing distinct regions representing imaged radiation in respective distinct wavelengths, the method comprising: selecting a plurality of subsets from the sequence of images, each one of the plurality of subset containing a plurality of images, each image of which represents a field of view that overlaps with a field of view of at least one other image in the same subset; generating a set of intermediate 3D models by performing photogrammetric 3D reconstruction on the images in respective ones of the subsets; and recombining the intermediate 3D models from the set of 3D models into a combined 3D model. However, this method still requires a significant amount of processing.
International patent application publication no. WO 2014/151746 A2 in the name of URC Ventures Inc. discloses techniques for analyzing images acquired via mobile devices in various ways, including to estimate measurements for one or more attributes of one or more objects in the images. For example, the described techniques are stated to be usable to measure the volume of a stockpile of material or other large object, based on images acquired via a mobile device that is carried by a human user as he or she passes around some or all of the object. During the acquisition of a series of digital images of an object of interest, various types of user feedback may be provided to a human user operator of the mobile device, and particular images may be selected for further analysis in various manners. Furthermore, the calculation of object volume and/or other determined object information may include generating and manipulating a computer model or other representation of the object from selected images. The techniques disclosed in this article are not concerned with images that represent radiation in distinct wavelengths.
The article by Lichun Li et al., “A new navigation approach of terrain contour matching based on 3-D terrain reconstruction from onboard image sequence”, Science China Technological Sciences, vol. 35 (2010), no. 5, 1176-1183, discloses a passive navigation method of terrain contour matching by reconstructing the 3-D terrain from the image sequence acquired by an onboard camera. To achieve automation and simultaneity of the image sequence processing for navigation, a correspondence registration method based on control points tracking is proposed which tracks the sparse control points through the whole image sequence and uses them as correspondence in the relation geometry solution. A key frame selection method based on the images overlapping ratio and intersecting angles is explored, and then the requirement for the camera system configuration is provided. The proposed method also includes an optimal local homography estimating algorithm according to the control points, which helps correctly predict points to be matched and their corresponding speed. Consequently, the real-time 3-D terrain of the trajectory thus reconstructed is matched with the referenced terrain map, and the result provides navigating information. The techniques disclosed in this article are not concerned with images that represent radiation in distinct wavelengths.
The article by Turner Darren et al., “Direct Georeferencing of Ultrahigh-Resolution UAV Imagery”, IEEE Transactions on Geoscience and Remote Sensing, vol. 52 (2014), no. 5, 2738-2745, discloses a concept for a camera-global positioning system (GPS) module that allows the synchronization of camera exposure with an airframe's position as recorded by a GPS with 10-20-cm accuracy. Lever arm corrections were applied to the camera positions to account for the positional difference between the GPS antenna and the camera center. Image selection algorithms were implemented to eliminate blurry images and images with excessive overlap. This study compared three different software methods (Photoscan, Pix4D web service, and an in-house Bundler method), evaluating each based on processing time, ease of use, and the spatial accuracy of the final mosaic produced. The article mentions the need to remove some of these overlapping images to improve processing efficiency, which is achieved by reading the image location information from image EXIF headers. An image overlap of around 80% is stated to yield the best results for SfM-based image processing algorithms. Image subsetting is achieved by computing the distance between consecutive images. The techniques disclosed in this article are not concerned with images that represent radiation in distinct wavelengths.
International patent application publication no. WO 2014/031284 A1 in the name of Visual Intelligence LP, discloses an imaging sensor system comprising: a mount unit affixed to a vehicle or a platform and having at least three imaging sensors disposed within the mount unit, wherein a first, second and third imaging sensor each has a focal axis passing through an aperture in the mount unit, wherein the first image sensor generates a first image area of the target area comprising a first array of pixels, wherein the second image sensor generates a second image area of the target area comprising a first array of pixels, wherein the first and second imaging sensors are offset to have a first image overlap area in the target area, wherein the first sensors image data bisects the second sensors image data in the first image overlap area. Various embodiments are described. The camera array assembly is configured such that adjoining borders of the relevant image areas overlap slightly. While this document suggests ortho-rectifying the sequence of overlapping images using standard photogrammetry techniques to produce an orthomap in which each pixel has an unique latitude and longitude coordinate and a unique elevation coordinate, it does not address dealing with images acquired in different spectral ranges.
International patent application publication no. WO 2004/027348 A2, in the name of M7 Visual Intelligence LP, discloses a method to calibrate an on-board remote sensing system using a self-locking travel pattern and target remote sensing data. The self-locking travel pattern includes a number of parallel travel lines having overlapping swath widths between adjacent travel lines. The overlapping swath widths are used to determine the boresight angles and range offset of the remote sensor device. In addition, the method can be used to generate estimated horizontal and vertical displacement errors. These estimated errors can be used as correction factors for the range offset and boresight angles.
There is a need for an alternative way of geometrically referencing multi-spectral or hyperspectral images that does not rely on simultaneously acquired panchromatic images and that has limited computational complexity.