Differential privacy is a promising approach to privacy preserving data analysis with a well-developed theory for functions. Despite recent work on implementing systems that aim to provide differential privacy, the problem of formally verifying that these systems have differential privacy has not been adequately addressed. This paper presents the first results towards automated verification of source code for differentially private interactive systems. We develop a formal probabilistic automaton model of differential privacy for systems by adapting prior work on differential privacy for functions. The main technical result of the paper is a sound proof technique based on a form of probabilistic bisimulation relation for proving that a system modeled as a probabilistic automaton satisfies differential privacy. The novelty lies in the way we track quantitative privacy leakage bounds using a relation family instead of a single relation. We illustrate the proof technique on a representative automaton motivated by PINQ, an implemented system that is intended to provide differential privacy. To make our proof technique easier to apply to realistic systems, we prove a form of refinement theorem and apply it to show that a refinement of the abstract PINQ automaton also satisfies our differential privacy definition. Finally, we begin the process of automating our proof technique by providing an algorithm for mechanically checking a restricted class of relations from the proof technique.
This paper deals with probabilistic and nondeterministic processes represented by a variant of labeled transition systems where any outgoing transition of a state s is augmented with probabilities for the possible successor states. Our main contributions are algorithms for computing this bisimulation equivalence classes as introduced by Larsen and Skou (1996, Inform. and Comput.99, 1?28), and the simulation preorder a la Segala and Lynch (1995, Nordic J. Comput.2, 250?273). The algorithm for deciding bisimilarity is based on a variant of the traditional partitioning technique and runs in time O(mn(logm+logn)) where m is the number of transitions and n the number of states. The main idea for computing the simulation preorder is the reduction to maximum flow problems in suitable networks. Using the method of Cheriyan, Hagerup, and Mehlhorn, (1990, Lecture Notes in Computer Science, Vol. 443, pp. 235?248, Springer-Verlag, Berlin) for computing the maximum flow, the algorithm runs in time O((mn6+m2n3) logn). Moreover, we show that the network-based technique is also applicable to compute the simulation-like relation of Jonsson and Larsen (1991, “Proc. LICS'91” pp. 266?277) in fully probabilistic systems (a variant of ordinary labeled transition systems where the nondeterminism is totally resolved by probabilistic choices). In this paper, we introduce weak bisimulation in the framework of Labeled Concurrent Markov Chains, that is, probabilistic transition systems which exhibit both probabilistic and nondeterministic behavior. By resolving the nondeterminism present, these models can be decomposed into a possibly infinite number of computation trees. We show that in order to compute weak bisimulation it is sufficient to restrict attention to only a finite number of these computations. Finally, we present an algorithm for deciding weak bisimulation which has polynomial-time complexity in the number of states of the transition system. This paper considers a weak simulation preorder for Markov chains that allows for stuttering. Despite the second-order quantification in its definition, we present a polynomial-time algorithm to compute the weak simulation preorder of a finite Markov chain. 