3D Reconstruction Method of Rapeseed Plants in the Whole Growth Period Using RGB-D Camera

The 3D reconstruction method using RGB-D camera has a good balance in hardware cost, point cloud quality and automation. However, due to the limitation of inherent structure and imaging principle, the acquired point cloud has problems such as a lot of noise and difficult registration. This paper proposes a three-dimensional reconstruction method using Azure Kinect to solve these inherent problems. Shoot color map, depth map and near-infrared image of the target from six perspectives by Azure Kinect sensor. Multiply the 8-bit infrared image binarization with the general RGB-D image alignment result provided by Microsoft to remove ghost images and most of the background noise. In order to filter the floating point and outlier noise of the point cloud, a neighborhood maximum filtering method is proposed to filter out the abrupt points in the depth map. The floating points in the point cloud are removed before generating the point cloud, and then using the through filter filters out outlier noise. Aiming at the shortcomings of the classic ICP algorithm, an improved method is proposed. By continuously reducing the size of the down-sampling grid and the distance threshold between the corresponding points, the point clouds of each view are continuously registered three times, until get the complete color point cloud. A large number of experimental results on rape plants show that the point cloud accuracy obtained by this method is 0.739mm, a complete scan time is 338.4 seconds, and the color reduction is high. Compared with a laser scanner, the proposed method has considerable reconstruction accuracy and a significantly ahead of the reconstruction speed, but the hardware cost is much lower and it is easy to automate the scanning system. This research shows a low-cost, high-precision 3D reconstruction technology, which has the potential to be widely used for non-destructive measurement of crop phenotype.


Introduction
Traditional measurement of agronomic traits adopts manual methods, which has disadvantages such as low efficiency, strong subjectivity, poor repeatability, damage to plants, and incomplete measurement , etc. Computer vision technology can analyze complete phenotype parameters such as plant structure, shape, color and texture t one time, making it possible to quantitatively study the growth laws of crops. Threedimensional reconstruction is an important type of computer vision technology. It performs digital modeling of crops in a computer, stores the three-dimensional geometry and color of plant shapes and organs in the computer to achieve rapid, lowcost, fast agronomic traits of crops and accurate non-destructive measurement [1] . 3D reconstruction technologies which are currently used in agriculture mainly include laser scanners, binocular vision, motion recovery structures and RGB-D cameras, etc.
(1)Three-dimensional laser scanner. It is a high-precisio n point cloud acquisition instrument. For example, Katrine et al. used a laser scanner to measure the daily growth ch anges of 10 genotype of Brassica naps, explored how differ ent stresses affect plant phenotype, and better understood t he interaction between plant responses and energetic regula tion [2] . Ana et al. used terrestrial laser scanner (TLS) and m obile scanning system (MMS) to obtain the point cloud of t he vine trunk and calculated the volume of the vine trunk through a grape-shaped artificial object (VSAO) calibration method, and proposed a skeleton based on internal measur ement. The algorithm of modeling a cylinder with a certain height and diameter effectively solved the problem of volu me estimation [3] . Zhang et al. used the airborne lidar syste m to obtain the three-dimensional point cloud of forest tree s, and extracted the ground and building point clouds thro ugh the irregular triangulation method and plane fitting filt ering algorithm, which provided a basis for high-precision quantitative estimation of forest canopy biomass [4] . Su et al. used laser scanners to collect lidar data at six growth stage s of 20 maize varieties under drought stress, and calculated three drought-related phenotypic data such as plant height, plant area index and predicted leaf area, in order to ident ify the effects of drought stress. The effect of the phenotyp e in each growth stage provides a reference [5] . Indu et al. u sed ground lidar to track a cylinder along a single branch layer by layer to achieve three-dimensional reconstruction o f a single tree, and used super-body clustering method and multiple regression technology to achieve automatic calcula tion of tree parameters such as leaf area [6] .
(2)The structural method of exercise recovery (SFM). It is a technology that can automatically recover camera parameters and the three-dimensional structure of the scene from multiple image sequences. Wei et al. proposed a high-throughput wheat phenotype measurement method based on volume reconstruction and estimated the fresh weight of wheat. This method has greatly improved in plant reconstruction, model processing and feature extraction [7] . Zhang et al. collected multiple multi-spectral images of four-leaf-age rapeseed and obtained the three-dimensional point cloud of rapeseed through the motion restoration structure algorithm. After denosing the point cloud, the control points and control length were evaluated to study plant nutrition and pest stress. The spatial distribution is of great significance [8] . Liang  (3)Binocular vision method. It generates parallax through different viewing angles of two cameras, and then obtains the distance information of the object through depth calculation. centroid of the smallest bounding rectangle of litchi fruits, and calculated the swing angles of litchi fruits under three disturbance states using the principle of pendulum, which solved the visual positioning problem of picking robots in the natural environment [13] .
(4)RGB-D camera method. The RGB-D camera adds depth measurement to the function of the RGB ordinary camera. The mainstream solutions include structured light and TOF (time of flight). It has simple operation, low cost and high efficiency, it always has great potential in the field of 3D reconstruction. Hu et al. proposed a Kinect-based automatic measurement system for leafy vegetable crops, which reconstructed plant point clouds from multiple perspectives, and simultaneously measured key growth parameters such as relative height and absolute height, projected area and volume, and the obtained data are all expressed a good degree of fit (R 2 = 0.9457-0.9914) [14] . Xu et al. used the Kinect V2 to take color images and depth images of rape branches under four viewing angles, and performed image preprocessing, point cloud registration and point cloud filtering to obtain the color point cloud of rape branches, and then used European clustering to identify the angle the phenotypic parameters of siliques were measured and the overall accuracy was better [15] . Xu et al. proposed a three-dimensional scanning reconstruction system with multiple RGB-D Cameras (Xtion sensors). The initial pose of the camera was obtained through a pre-calibrated image acquisition platform. The reconstruction system can obtain high-precision three-dimensional models of various complex large scenes [16] . Martin et al. used Kinect V2 to collect images of field corn plants, and measured plant height after three-dimensional reconstruction. The measurement accuracy exceeded the measurement results of three-dimensional laser scanners [17] . Manuel et al. used high-resolution threedimensional images, used ICP registration algorithm for registration, and used random sampling consensus algorithm to remove the incoming soil point cloud. By comparing the true seedling position on the ground, it was shown that the method obtained the location of the corn is close to the ground distribution [18] . Feng [19] . Liu et al. proposed a threedimensional reconstruction algorithm based on depth information segmentation and clustering of strawberry canopy morphology. The color image and intensity image were registered to reconstruct the strawberry canopy structure morphology with color information [20] . Yu et al. placed Kinect V2 on the cart, took the depth image directly above the cotton crop in the field, calculated the plant height in the stitched depth image, and realized the non-destructive and rapid measurement of the cotton plant height in the field [21] . Efi et al. proposed a sweet pepper detection algorithm that combines adaptive threshold segmentation and sensors. The image was divided into rectangular sub-images with approximately uniform lighting conditions, and the RGB image was converted into a 3D natural difference index image. Threshold adaptive calculation and application, the algorithm was robust to the selected threshold and the noise from the camera itself [22] .
In summary, the point cloud obtained by the laser scanner method has the highest accuracy, but it is expensive, cumbersome to operate, and time-consuming, making it difficult to realize an automated system. The SFM method has higher accuracy and the lowest cost, but high-precision threedimensional reconstruction requires a large number of images, which leads to the most serious consumption of computing resources [23] . Binocular vision has the fastest reconstruction speed and is easy to automate, but its high-precision point cloud requires extremely high-performance image sensors and a stable imaging environment. In contrast, the RGB-D camera represented by Kinect is more balanced in three aspects: point cloud accuracy, processing speed and automatability. At the same time, this method has the highest requirements for image processing algorithms, mainly due to two factors. The first one requires a registration algorithm to unify the point clouds from different coordinate systems into the same coordinate system [24] . Other outliers and floating point noise were removed based on the normals of the viewpoint and the surface. This method can effectively remove outliers and improve the quality of point clouds [28] .
Aiming at the needs of multi-scale, non-destructive, highthroughput and high-precision measurement of rape phenotypic traits, this paper proposes a three-dimensional reconstruction method using Azure Kinect. This method collects color maps, depth maps and infrared maps from 6 viewing angles, improves the color map-depth map alignment method provided by Microsoft to remove ghost images, and proposes a neighborhood maximum filter method to filter out floating points in the depth map, and then use the improved ICP registration algorithm [29] to fuse the local point cloud into a complete point cloud, and finally realize the accurate three-dimensional color modeling of a single rapeseed during the whole growth period. The threedimensional reconstruction method has low cost, high accuracy, fast speed and easy automation. It will provide important basic data for the non-destructive measurement of rape phenotype, growth law research, selection and breeding, and design of agricultural machinery, etc., also can be popularized and applied to other crops.

Experimental Setup and Data Acquisition
In 2019 and 2020, rapeseed was planted at Huazhong Agricultural University for two consecutive years, with varieties

The complete color point cloud calculation method of rape moss
Taking rapeseed at the first flowering stage as an example,   determine, the robustness is poor, and it is difficult to find a parameter that has a good effect on the two noise point clouds.

3D point cloud registration
By registering and fusing point clouds from multiple angles into a whole, the complete 3D shape of the rape plant can be Repeatedly using this method for multiple viewing angles, a complete point cloud of a rape plant can be obtained.

Registration Optimization Based on ICP Method
(1) Point cloud registration from two perspectives the Euclidean distance, as showed in formula (2).
In order to minimize ( ), the center of gravity of the point cloud P and the point cloud X are calculated separately, as in formula (3).
Use the center of gravity to obtain the cross-covariance matrix of the two point clouds, as in formula (4).
Based on the classic ICP algorithm, this paper proposes an improved method, as shown in Figure 4

Results and analysis
The experiments were carried out on raw data obtained from 40 pots of rapeseed, in which 10 pots in each growth period.
For each pot of plant, 6 frames of data from different views, which cover 360° and these data were processed by the proposed method to show the performance and robustness of the proposed method.

Point-Cloud Noise Removal
We tested and compared two filtering methods: the traditional method of through filtering + statistical filtering, as shown in Figure 6 (a); the recommended method of neighborhood maximum filtering + through filtering, as shown in Figure 6   The neighborhood optimal filtering method needs to set an appropriate neighborhood size and threshold. The size of the neighborhood is too small to judge the sudden change of the depth information, which leads to the deletion of too many valid points. However, too many points will participate in the judgment if the setting is too large, so that the deep mutation information is inaccurate and the mutation point cannot be found.
Through the statistics of the gentle point cloud of a certain part of rape, it is found that the maximum depth difference in the local neighborhood is not obvious, and the difference is basically 5 points. This paper compares the size of different neighborhoods and the size of the threshold to eliminate the effect of floating point noise, and selects the final parameters. Figure 7 shows the filtering results of different parameters, the neighborhood sizes are 3*3, 5*5, and 7*7, and the thresholds range from 3, 5 to 7.
Through comparison, it is found that when the neighborhood size is 3*3 and the threshold is 5. The effect of removing the noise point cloud is the best.

Point cloud registration experiment
We tested and compared the performance of the classic ICP method and the method in this paper for the two-view point cloud registration, which is shown in Figure 8 and Table 3. For any rape plant, the method in this paper undergoes 3 iterations of registration, and finally merges into a point cloud precisely. and (d) are the same type of test results of rape seedlings.   The color 3D point cloud obtained is compared with the method in this paper. Figure 9 seconds, while that of the laser scanner is 991.6 seconds. The former is 1/3 of the latter. If an automated image acquisition device is used, the scanning method of Azure Kinect will be faster.
In general, the reconstruction accuracy of this method is close to that of a laser scanner, but the speed is 3 times faster. This is mainly due to the fact that the data collection speed of the laser   used the whole growing period of rapeseed plants to test.

Conclusion
The overall effect is good, which is helpful to the advance ment and development of rapeseed phenotype nondestructi ve testing technology. It has also achieved good effects on crops such as fruit seedlings, corn, soybeans, and cotton. In general, the research results of this manuscript provide a useful example for RGB-D 3D reconstruction.

Data Availability:
The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest:
The authors declare that they have no conflicts of interest.