Convolution plays a crucial role in various applications in signal and image processing, analysis and recognition. It is also the main building block of convolution neural networks (CNNs). Designing appropriate convolution neural networks on manifold-structured point clouds can inherit and empower recent advances of CNNs to analyzing and processing point cloud data. However, one of the major challenges is to define a proper way to "sweep" filters through the point cloud as a natural generalization of the planar convolution and to reflect the point cloud's geometry at the same time. In this paper, we consider generalizing convolution by adapting parallel transport on the point cloud. Inspired by a triangulated surface based method [Stefan C. Schonsheck, Bin Dong, and Rongjie Lai, arXiv:1805.07857.], we propose the Narrow-Band Parallel Transport Convolution (NPTC) using a specifically defined connection on a voxelized narrow-band approximation of point cloud data. With that, we further propose a deep convolutional neural network based on NPTC (called NPTC-net) for point cloud classification and segmentation. Comprehensive experiments show that the proposed NPTC-net achieves similar or better results than current state-of-the-art methods on point clouds classification and segmentation.
The second type of convolution is defined by first projecting the point cloud locally on an Euclidean domain and then employ regular convolution. This type of methods can also be formulate as ). For example, Tangent convolutions @cite_28 define kernels on the tangent plane, and use 2 principal directions of a local PCA as @math . Pointconv @cite_17 constructs local kernels by interpolation in @math , i.e. letting @math which is essentially a local Euclidean convolution.
@cite_28 @cite_17
2797997528 2900731076
Unlike images which are represented in regular dense grids, 3D point clouds are irregular and unordered, hence applying convolution on them can be difficult. In this paper, we extend the dynamic filter to a new convolution operation, named PointConv. PointConv can be applied on point clouds to build deep convolutional networks. We treat convolution kernels as nonlinear functions of the local coordinates of 3D points comprised of weight and density functions. With respect to a given point, the weight functions are learned with multi-layer perceptron networks and density functions through kernel density estimation. The most important contribution of this work is a novel reformulation proposed for efficiently computing the weight functions, which allowed us to dramatically scale up the network and significantly improve its performance. The learned convolution kernel can be used to compute translation-invariant and permutation-invariant convolution on any point set in the 3D space. Besides, PointConv can also be used as deconvolution operators to propagate features from a subsampled point cloud back to its original resolution. Experiments on ModelNet40, ShapeNet, and ScanNet show that deep convolutional neural networks built on PointConv are able to achieve state-of-the-art on challenging semantic segmentation benchmarks on 3D point clouds. Besides, our experiments converting CIFAR-10 into a point cloud showed that networks built on PointConv can match the performance of convolutional networks in 2D images of a similar structure. A convenient way of analysing Riemannian manifolds is to embed them in Euclidean spaces, with the embedding typically obtained by flattening the manifold via tangent spaces. This general approach is not free of drawbacks. For example, only distances between points to the tangent pole are equal to true geodesic distances. This is restrictive and may lead to inaccurate modelling. Instead of using tangent spaces, we propose embedding into the Reproducing Kernel Hilbert Space by introducing a Riemannian pseudo kernel. We furthermore propose to recast a locality preserving projection technique from Euclidean spaces to Riemannian manifolds, in order to demonstrate the benefits of the embedding. Experiments on several visual classification tasks (gesture recognition, person re-identification and texture classification) show that in comparison to tangent-based processing and state-of-the-art methods (such as tensor canonical correlation analysis), the proposed approach obtains considerable improvements in discrimination accuracy. We present an approach to semantic scene analysis using deep convolutional networks. Our approach is based on tangent convolutions - a new construction for convolutional networks on 3D data. In contrast to volumetric approaches, our method operates directly on surface geometry. Crucially, the construction is applicable to unstructured point clouds and other noisy real-world data. We show that tangent convolutions can be evaluated efficiently on large-scale point clouds with millions of points. Using tangent convolutions, we design a deep fully-convolutional network for semantic segmentation of 3D point clouds, and apply it to challenging real-world datasets of indoor and outdoor 3D environments. Experimental results show that the presented approach outperforms other recent deep network constructions in detailed analysis of large 3D scenes. One. Introduction: On the Absence of Geometry from Morphometrics.- First Part. The Measurement of Biological Shape.- Two. Shapes and Measures of Shape.- A. Properties of the Euclidean Plane and Euclidean Space.- B. Outlines and Homologous Landmarks.- C. Definitions of Shape, Shape Change, Shape Measurement.- D. Shapes as Data.- Three. Critique of an Applied Field: Conventional Cephalometrics.- A. Landmarks, Curvature, and Growth.- B. Registration.- Four. New Statistical Methods for Shape.- A. Analysis of the Tangent Angle Function.- 1. History.- 2. Sampling from the tangent angle function.- 3. Conic replacement curves and their estimation Fit of a conic to a point - Geometric interpretation - Estimator for a circle - Estimation for the general conic - Computation of the extremum -Linear constraints.- 4. Conic splining Joint conic fitting under constraint -An example - Application - Analysis of parameters.- B. Extension to Three Dimensions: A Sketch.- C. Skeletons Definition - A multivariate statistical method - Bibliographic note.- Second Part. The Measurement of Shape Change Using Biorthogonal Grids.- Five. The Study of Shape Transformation after D'Arcy Thompson.- A. The Original Method.- 1. Thompson's own work.- 2. Later examples.- 3. Difficulties.- B. Analysis of Growth Gradients.- C. Simulations.- D. Other Morphometric Schemes Vector displacements - Multivariate morphometrics.- Six. The Method of Biorthogonal Grids.- A. Representation of Affine Transformations.- B. General Lines of Growth and Biorthogonal Grids.- C. Summarizing the Grids.- Technical Note 1. Existence and Form of Biorthogonal Grids.- Technical Note 2. Interpolation from Landmark Locations and Arcs The measure of roughness - The vector space and its associated functions - Interpolation from boundary values - Interpolation from boundary values and interior points - Note on computation.- Technical Note 3. Construction of Integral Curves.- Technical Note 4. On Homologous Points.- Seven. Examples of Biorthogonal Analysis.- A. Comparison of Square and Biorthogonal Grids: Thompson's Diodon Figure.- B. Phylogeny and Ontogeny of Primate Crania.- 1. Functional craniology and craniometrics.- 2. Data for this exercise.- 3. Two types of transformations.- 4. Comparative ontogeny of the apes and man.- Eight. Future Directions for Transformation Analysis.- A. Statistical Methods The symmetric tensor field - Concordance -Linear methods.- B. Computation Other kinds of information about homology -Three dimensions.- C. Likely Applications Computed tomography - Orthodontics - Developmental biology.- Nine. Envoi.- Literature Cited. Extracting local features from 3D shapes is an important and challenging task that usually requires carefully designed 3D shape descriptors. However, these descriptors are hand-crafted and require intensive human intervention with prior knowledge. To tackle this issue, we propose a novel deep learning model, namely circle convolutional restricted Boltzmann machine (CCRBM), for unsupervised 3D local feature learning. CCRBM is specially designed to learn from raw 3D representations. It effectively overcomes obstacles such as irregular vertex topology, orientation ambiguity on the 3D surface, and rigid or slightly non-rigid transformation invariance in the hierarchical learning of 3D data that cannot be resolved by the existing deep learning models. Specifically, by introducing the novel circle convolution, CCRBM holds a novel ring-like multi-layer structure to learn 3D local features in a structure preserving manner. Circle convolution convolves across 3D local regions via rotating a novel circular sector convolution window in a consistent circular direction. In the process of circle convolution, extra points are sampled in each 3D local region and projected onto the tangent plane of the center of the region. In this way, the projection distances in each sector window are employed to constitute a novel local raw 3D representation called projection distance distribution (PDD). In addition, to eliminate the initial location ambiguity of a sector window, the Fourier transform modulus is used to transform the PDD into the Fourier domain, which is then conveyed to CCRBM. Experiments using the learned local features are conducted on three aspects: global shape retrieval, partial shape retrieval, and shape correspondence. The experimental results show that the learned local features outperform other state-of-the-art 3D shape descriptors. Video frame interpolation typically involves two steps: motion estimation and pixel synthesis. Such a two-step approach heavily depends on the quality of motion estimation. This paper presents a robust video frame interpolation method that combines these two steps into a single process. Specifically, our method considers pixel synthesis for the interpolated frame as local convolution over two input frames. The convolution kernel captures both the local motion between the input frames and the coefficients for pixel synthesis. Our method employs a deep fully convolutional neural network to estimate a spatially-adaptive convolution kernel for each pixel. This deep neural network can be directly trained end to end using widely available video data without any difficult-to-obtain ground-truth data like optical flow. Our experiments show that the formulation of video interpolation as a single convolution process allows our method to gracefully handle challenges like occlusion, blur, and abrupt brightness change and enables high-quality video frame interpolation. This paper presents Point Convolutional Neural Networks (PCNN): a novel framework for applying convolutional neural networks to point clouds. The framework consists of two operators: extension and restriction, mapping point cloud functions to volumetric functions and vise-versa. A point cloud convolution is defined by pull-back of the Euclidean volumetric convolution via an extension-restriction mechanism. The point cloud convolution is computationally efficient, invariant to the order of points in the point cloud, robust to different samplings and varying densities, and translation invariant, that is the same convolution kernel is used at all points. PCNN generalizes image CNNs and allows readily adapting their architectures to the point cloud setting. Evaluation of PCNN on three central point cloud learning benchmarks convincingly outperform competing point cloud learning methods, and the vast majority of methods working with more informative shape representations such as surfaces and or normals. Unlike on images, semantic learning on 3D point clouds using a deep network is challenging due to the naturally unordered data structure. Among existing works, PointNet has achieved promising results by directly learning on point sets. However, it does not take full advantage of a point's local neighborhood that contains fine-grained structural information which turns out to be helpful towards better semantic learning. In this regard, we present two new operations to improve PointNet with a more efficient exploitation of local structures. The first one focuses on local 3D geometric structures. In analogy to a convolution kernel for images, we define a point-set kernel as a set of learnable 3D points that jointly respond to a set of neighboring data points according to their geometric affinities measured by kernel correlation, adapted from a similar technique for point cloud registration. The second one exploits local high-dimensional feature structures by recursive feature aggregation on a nearest-neighbor-graph computed from 3D positions. Experiments show that our network can efficiently capture local information and robustly achieve better performances on major datasets. Our code is available at this http URL A surface interpolation method for irregular meshes of curves is proposed. When a face in the mesh is interpolated, a surface patch on the face is individually generated from localized boundary information. The tangent planes of generated patches are continuous. This method employs an extended Bezier patch as a surface equation. The patch can be defined, specifying independently normal derivatives along the boundary curves. The total procedure of generating patches is simple and quick. Unlike on images, semantic learning on 3D point clouds using a deep network is challenging due to the naturally unordered data structure. Among existing works, PointNet has achieved promising results by directly learning on point sets. However, it does not take full advantage of a point's local neighborhood that contains fine-grained structural information which turns out to be helpful towards better semantic learning. In this regard, we present two new operations to improve PointNet with a more efficient exploitation of local structures. The first one focuses on local 3D geometric structures. In analogy to a convolution kernel for images, we define a point-set kernel as a set of learnable 3D points that jointly respond to a set of neighboring data points according to their geometric affinities measured by kernel correlation, adapted from a similar technique for point cloud registration. The second one exploits local high-dimensional feature structures by recursive feature aggregation on a nearest-neighbor-graph computed from 3D positions. Experiments show that our network can efficiently capture local information and robustly achieve better performances on major datasets. Our code is available at http: www.merl.com research license#KCNet Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent approaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand. To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames. This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation. This paper advocates a novel framework for segmenting a dataset in a Riemannian manifold @math into clusters lying around low-dimensional submanifolds of @math . Important examples of @math , for which the proposed clustering algorithm is computationally efficient, are the sphere, the set of positive definite matrices, and the Grassmannian. The clustering problem with these examples of @math is already useful for numerous application domains such as action identification in video sequences, dynamic texture clustering, brain fiber segmentation in medical imaging, and clustering of deformed images. The proposed clustering algorithm constructs a data-affinity matrix by thoroughly exploiting the intrinsic geometry and then applies spectral clustering. The intrinsic local geometry is encoded by local sparse coding and more importantly by directional information of local tangent spaces and geodesics. Theoretical guarantees are established for a simplified variant of the algorithm even when the clusters intersect. To avoid complication, these guarantees assume that the underlying submanifolds are geodesic. Extensive validation on synthetic and real data demonstrates the resiliency of the proposed method against deviations from the theoretical model as well as its superior performance over state-of-the-art techniques. We present a general method of studying the transport process ( X(t) ) , t≥0, in the Euclidean space ℝ m , m≥2, based on the analysis of the integral transforms of its distributions. We show that the joint characteristic functions of ( X(t) ) are connected with each other by a convolution-type recurrent relation. This enables us to prove that the characteristic function (Fourier transform) of ( X(t) ) in any dimension m≥2 satisfies a convolution-type Volterra integral equation of second kind. We give its solution and obtain the characteristic function of ( X(t) ) in terms of the multiple convolutions of the kernel of the equation with itself. An explicit form of the Laplace transform of the characteristic function in any dimension is given. The complete solution of the problem of finding the initial conditions for the governing partial differential equations, is given. Abstract We describe a method for recovering the underlying parametrization of scattered data (mi) lying on a manifold M embedded in high-dimensional Euclidean space. The method, Hessian-based locally linear embedding, derives from a conceptual framework of local isometry in which the manifold M, viewed as a Riemannian submanifold of the ambient Euclidean space ℝn, is locally isometric to an open, connected subset Θ of Euclidean space ℝd. Because Θ does not have to be convex, this framework is able to handle a significantly wider class of situations than the original ISOMAP algorithm. The theoretical framework revolves around a quadratic form ℋ(f) = ∫M ∥Hf(m)∥dm defined on functions f : M ↦ ℝ. Here Hf denotes the Hessian of f, and ℋ(f) averages the Frobenius norm of the Hessian over M. To define the Hessian, we use orthogonal coordinates on the tangent planes of M. The key observation is that, if M truly is locally isometric to an open, connected subset of ℝd, then ℋ(f) has a (d + 1)-dimensional null space consisting of the constant functions and a d-dimensional space of functions spanned by the original isometric coordinates. Hence, the isometric coordinates can be recovered up to a linear isometry. Our method may be viewed as a modification of locally linear embedding and our theoretical framework as a modification of the Laplacian eigenmaps framework, where we substitute a quadratic form based on the Hessian in place of one based on the Laplacian. Consider the problem of finding the local constrained minimum @math of the function f on the set [ F = x R^n | _i (x) 0, _ j + k (x) = 0; i = 1,2, ,k; j = 1,2, ,l . ]One method of solution is to minimize the associated penalty function [ p_0 (x) = f(x) - _ i = 1 ^k ( 0, _i (x) ) + _ j = 1 ^l | _ j + k (x) | for x R^n , 0. ]Let @math be the minimum of this penalty function. It is known that, provided @math is sufficiently small, @math .However, until recently, a serious drawback to this particular penalty function was that its first order derivatives are not everywhere defined. Thus, well-known gradient type methods usually applied to unconstrained optimization problems were necessarily excluded.This paper presents a method that enables a modified form of the gradient type approaches to be applied to a perturbation of the penalt... Establishing correspondence between shapes is a fundamental problem in geometry processing, arising in a wide variety of applications. The problem is especially difficult in the setting of non-isometric deformations, as well as in the presence of topological noise and missing parts, mainly due to the limited capability to model such deformations axiomatically. Several recent works showed that invariance to complex shape transformations can be learned from examples. In this paper, we introduce an intrinsic convolutional neural network architecture based on anisotropic diffusion kernels, which we term Anisotropic Convolutional Neural Network (ACNN). In our construction, we generalize convolutions to non-Euclidean domains by constructing a set of oriented anisotropic diffusion kernels, creating in this way a local intrinsic polar representation of the data ( patch'), which is then correlated with a filter. Several cascades of such filters, linear, and non-linear operators are stacked to form a deep neural network whose parameters are learned by minimizing a task-specific cost. We use ACNNs to effectively learn intrinsic dense correspondences between deformable shapes in very challenging settings, achieving state-of-the-art results on some of the most difficult recent correspondence benchmarks. We propose a generalization of convolutional neural networks (CNNs) to irregular domains, through the use of an inferred graph structure. In more details, we introduce a three-step methodology to create convolutional layers that are adapted to the signals to process: 1) From a training set of signals, infer a graph representing the topology on which they evolve; 2) Identify translation operators in the vertex domain; 3) Emulate a convolution operator by translating a localized kernel on the graph. Using these layers, a convolutional neural network is built, and is trained on the initial signals to perform a classification task. Contributions are twofold. First, we adapt a definition of translations on graphs to make them more robust to irregularities, and to take into account locality of the kernel. Second, we introduce a procedure to build CNNs from data. We apply our methodology on a scrambled version of the CIFAR-10 and Haxby datasets. Without using any knowledge on the signals, we significantly outperform existing methods. Moreover, our approach extends classical CNNs on images in the sense that such networks are a particular case of our approach when the inferred graph is a grid. Let M be a smooth compact oriented Riemannian manifold, and let ΔM be the Laplace–Beltrami operator on M. Say ( 0 f S ( R ^+) ) , and that f (0) = 0. For t > 0, let Kt(x, y) denote the kernel of f (t2 ΔM). We show that Kt is well-localized near the diagonal, in the sense that it satisfies estimates akin to those satisfied by the kernel of the convolution operator f (t2Δ) on ( R ^n ) . We define continuous ( S )-wavelets on M, in such a manner that Kt(x, y) satisfies this definition, because of its localization near the diagonal. Continuous ( S )-wavelets on M are analogous to continuous wavelets on ( R ^n ) in ( S ) ( ( R ^n )). In particular, we are able to characterize the Holder continuous functions on M by the size of their continuous ( S )-wavelet transforms, for Holder exponents strictly between 0 and 1. If M is the torus ( T^2 ) or the sphere S2, and f (s) = se−s (the “Mexican hat” situation), we obtain two explicit approximate formulas for Kt, one to be used when t is large, and one to be used when t is small. We present a new algorithm for manifold learning and nonlinear dimensionality reduction. Based on a set of unorganized data points sampled with noise from a parameterized manifold, the local geometry of the manifold is learned by constructing an approximation for the tangent space at each data point, and those tangent spaces are then aligned to give the global coordinates of the data points with respect to the underlying manifold. We also present an error analysis of our algorithm showing that reconstruction errors can be quite small in some cases. We illustrate our algorithm using curves and surfaces both in two-dimensional three-dimensional (2D 3D) Euclidean spaces and in higher-dimensional Euclidean spaces. We also address several theoretical and algorithmic issues for further research and improvements. Convolution is the core operation for many deep neural networks. The Winograd convolution algorithms have been shown to accelerate the widely-used small convolution sizes. Quantized neural networks can effectively reduce model sizes and improve inference speed, which leads to a wide variety of kernels and hardware accelerators that work with integer data. The state-of-the-art Winograd algorithms pose challenges for efficient implementation and execution by the integer kernels and accelerators. We introduce a new class of Winograd algorithms by extending the construction to the field of complex and propose optimizations that reduce the number of general multiplications. The new algorithm achieves an arithmetic complexity reduction of @math x over the direct method and an efficiency gain up to @math over the rational algorithms. Furthermore, we design and implement an integer-based filter scaling scheme to effectively reduce the filter bit width by @math without any significant accuracy loss.