The problem of autonomous transportation in industrial scenarios is receiving a renewed interest due to the way it can revolutionise internal logistics, especially in unstructured environments. This paper presents a novel architecture allowing a robot to detect, localise, and track multiple pallets using machine learning techniques based on an on-board 2D laser rangefinder. The architecture is composed of two main components: the first stage is a pallet detector employing a Faster Region-based Convolutional Neural Network (Faster R-CNN) detector cascaded with a CNN-based classifier; the second stage is a Kalman filter for localising and tracking detected pallets, which we also use to defer commitment to a pallet detected in the first step until sufficient confidence has been acquired via a sequential data acquisition process. For fine-tuning the CNNs, the architecture has been systematically evaluated using a real-world dataset containing 340 labeled 2D scans, which have been made freely available in an online repository. Detection performance has been assessed on the basis of the average accuracy over k-fold cross-validation, and it scored 99.58 in our tests. Concerning pallet localisation and tracking, experiments have been performed in a scenario where the robot is approaching the pallet to fork. Although data have been originally acquired by considering only one pallet, artificial data have been generated as well to mimic the presence of multiple targets in the robot workspace. Our experimental results confirm that the system is capable of identifying, localising and tracking pallets with a high success rate while being robust to false positives.
The paper presents a method for automatically detecting pallets and estimating their position and orientation. For detection we use a sliding window approach with efficient candidate generation, fast integral features and a boosted classifier. Specific information regarding the detection task such as region of interest, pallet dimensions and pallet structure can be used to speed up and validate the detection process. Stereo reconstruction is employed for depth estimation by applying Semi-Global Matching aggregation with Census descriptors. Offline test results show that successful detection is possible under 0.5 seconds. We present newly added modules for our autonomous load handling system. The stereo camera system provides information about the scene in front of the automated forklift. A new alternative module for pallet detection is described. Several processing modules for unloading operations are also presented. Our system is evaluated by means of detection rate and by performing field tests. The tests show that it is capable of providing a sufficiently accurate position of the pallets in order to perform loading and unloading operations in multiple scenarios. In this study, a novel approach for the detection of parcel loading positions on a pallet is presented. This approach was realized as a substantial change in comparison with traditional system design of contour detection in de-palletizing processes. Complex 3D-vision systems, costly laser scanners or throughput decreasing local sensor solutions integrated in grippers are substituted by a low-cost photonic mixing device (PMD) camera. By combining PMD technology and a predetermined model of loading situations, stored during assembling the pallet, this approach can compensate for the drawbacks of each respective system. An essential part of the approach are computer-graphics methods specific to the given problem to both detect the deviation between the nominal and the actual loading position and if necessary an automated correction of the packaging scheme. From an economic point of view, this approach can decrease the costs of mandatory contour checking in automated de-palletizing processes. A visual pallet recognition system based on the accurate color segmentation is proposed. The basic idea is to get the pallet color feature samples from the images in working environment before the autonomous mobile forklift works. Then the algorithms of morphological filtering, Sobel edge detection and Hough tranform are used to get the pallet forking side. At last, according to two corners of the forking side, the pallet midpoint coordinates of the forking part and the direction of the forking side are calculated, which are the pose provided for the autonomous mobile forklift engagement. Experimental results show that the visual recogition system has good real-time and robustness. Recently real-time active 3D range cameras based on time-of-flight technology (PMD) have become available. Those cameras can be considered as a competing technique for stereo-vision based surface reconstruction. Since those systems directly yield accurate 3d measurements, they can be used for benchmarking vision based approaches, especially in highly dynamic environments. Therefore, a comparative study of the two approaches is relevant. In this work the achievable accuracy of the two techniques, PMD and stereo, is compared on the basis of patch-let estimation. As patchlet we define an oriented small planar 3d patch with associated surface normal. Least-squares estimation schemes for estimating patchlets from PMD range images as well as from a pair of stereo images are derived. It is shown, how the achivable accuracy can be estimated for both systems. Experiments under optimal conditions for both systems are performed and the achievable accuracies are compared. It has been found that the PMD system outperformed the stereo system in terms of achievable accuracy for distance measurements, while the estimation of normal direction is comparable for both systems.