In a communication network, point-to-point traffic volumes over time are critical for designing protocols that route information efficiently and for maintaining security, whether at the scale of an internet service provider or within a corporation. While technically feasible, the direct measurement of point-to-point traffic imposes a heavy burden on network performance and is typically not implemented. Instead, indirect aggregate traffic volumes are routinely collected. We consider the problem of estimating point-to-point traffic volumes, x_t, from aggregate traffic volumes, y_t, given information about the network routing protocol encoded in a matrix A. This estimation task can be reformulated as finding the solutions to a sequence of ill-posed linear inverse problems, y_t = A x_t, since the number of origin-destination routes of interest is higher than the number of aggregate measurements available. Here, we introduce a novel multilevel state-space model of aggregate traffic volumes with realistic features. We implement a naive strategy for estimating unobserved point-to-point traffic volumes from indirect measurements of aggregate traffic, based on particle filtering. We then develop a more efficient two-stage inference strategy that relies on model-based regularization: a simple model is used to calibrate regularization parameters that lead to efficient and scalable inference in the multilevel state-space model. We apply our methods to corporate and academic networks, where we show that the proposed inference strategy outperforms existing approaches and scales to larger networks. We also design a simulation study to explore the factors that influence the performance. Our results suggest that model-based regularization may be an efficient strategy for inference in other complex multilevel models.
Previous models for emission tomography (ET) do not distinguish the physics of ET from that of transmission tomography. We give a more accurate general mathematical model for ET where an unknown emission density ? = ?(x, y, z) generates, and is to be reconstructed from, the number of counts n*(d) in each of D detector units d. Within the model, we give an algorithm for determining an estimate ? of ? which maximizes the probability p(n*|?) of observing the actual detector count data n* over all possible densities ?. Let independent Poisson variables n(b) with unknown means ?(b), b = 1, ···, B represent the number of unobserved emissions in each of B boxes (pixels) partitioning an object containing an emitter. Suppose each emission in box b is detected in detector unit d with probability p(b, d), d = 1, ···, D with p(b, d) a one-step transition matrix, assumed known. We observe the total number n* = n*(d) of emissions in each detector unit d and want to estimate the unknown ? = ?(b), b = 1, ···, B. For each ?, the observed data n* has probability or likelihood p(n*|?). The EM algorithm of mathematical statistics starts with an initial estimate ?0 and gives the following simple iterative procedure for obtaining a new estimate ?new, from an old estimate ?old, to obtain ?k, k = 1, 2, ···, ?new(b)= ?old(b) ?Dd=1 n*(d)p(b,d) ??old(b?)p(b?,d),b=1,···B.