recent advances in computing and measurement technologies have led to an explosion in the amount of data with network structures in a variety of fields including social networks , communication networks , biological networks , disease transmission networks and so on .this creates an urgent need to understand the generative mechanism of these networks and to explore various characteristics of the network structures in a principled way .statistical models are useful tools to this end , since they can capture the regularities of network processes and variability of network configurations of interests , and help to understand the uncertainty associated with observed outcomes . at the same time , data with network structures pose new challenges for statistical inference , in particular asymptotic analysis when only one realized network is observed and one is often interested in the asymptotic phenomena with the growing size of the network .the in- and out - degrees of vertices ( or degrees for undirected networks ) preliminarily summarize the information contained in a network , and their distributions provide important insights for understanding the generative mechanism of networks . in the undirected case ,the degree sequence has been extensively studied . in particular , its distributionshave been explored under the framework of the exponential family parameterized by the so - called `` potentials '' of vertices recently , for example , the `` -model '' by for binary edges or `` maximum entropy models '' by for weighted edges in which the degree sequence is the exclusively sufficient statistic .it is also worth to note that the asymptotic theory of the maximum likelihood estimates ( mles ) for these models have not been derived until very recently . in the directed case , how to construct andsample directed graphs with given in- and out - degree ( sometimes referred as `` bi - degree '' ) sequences have been studied . however , statistical inference is not available , especially for asymptotic analysis .the distributions of the bi - degrees were studied in through empirical examples for social networks , but the work lacked theoretical analysis . in this paper , we study the distribution of the bi - degree sequence when it is the sufficient statistic in a directed graph .recall the koopman pitman darmois theorem or the principle of maximum entropy , which states that the probability mass function of the bi - degree sequence must admit the form of the exponential family .we will characterize the exponential family distributions for the bi - degree sequence with three types of weighted edges ( binary , discrete and continuous ) and conduct the maximum likelihood inference .  in the model we study , one out - degree parameter and one in - degree parameterare needed for each vertex . as a result ,the total number of parameters is twice of the number of the vertices . as the size of the network increases , the number of parameters goes to infinity .this makes asymptotic inference very challenging .establishing the uniform consistency and asymptotic normality of the mle are the aims of this paper . to the best of our knowledge , it is the first time that such results are derived in directed exponential random graph models with weighted edges .we remark further that our proofs are highly nontrivial .one key feature of our proofs lies in approximating the inverse of the fisher information matrix by a simple matrix with small approximation errors .this approximation is utilized to derive a newton iterative algorithm with geometrically fast rate of convergence , which leads to the proof of uniform consistency , and it is also utilized to derive approximately explicit expressions of the estimators , which leads to the proof of asymptotic normality . furthermore , the approximate inverse makes the asymptotic variances of estimators explicit and concise .we note that have studied problems related to the present paper but the methods therein can not be applied to the model we study .this is explained in detail at the end of the next section after we state the main theorems .next , we formally describe the models considered in this paper .consider a directed graph on vertices labeled by .let be the weight of the directed edge from to , where is the set of all possible weight values , and be the adjacency matrix of .we consider three cases : , and , where the first case is the usual binary edge .we assume that there are no self - loops , that is , .let be the out - degree of vertex and be the out - degree sequence of the graph .similarly , define as the in - degree of vertex and as the in - degree sequence .the pair or are the bi - degree sequence. then the density or probability mass function on parameterized by exponential family distributions with respect to some canonical measure is where is the log - partition function , is a parameter vector tied to the out - degree sequence , and is a parameter vector tied to the in - degree sequence .this model can be viewed as a directed version of the -model .it can also be represented as the log - linear model and the algorithm developed for the log - linear model can be used to compute the mle . as explained by , quantifies the effect of an outgoing edge from vertex and quantifies the effect of an incoming edge connecting to vertex .if is large and positive , vertex will tend to have a relatively large out - degree .similarly , if is large and positive , vertex tends to have a relatively large in - degree . note that \\[-8pt]\nonumber & = & \prod _ { i , j=1;i\neq j}^n \exp\bigl((\alpha_i+ \beta_j ) a_{i , j } \bigr),\end{aligned}\ ] ] which implies that the random variables , are mutually independent and can be expressed as since an out - edge from vertex pointing to is the in - edge of coming from , it is immediate that moreover , since the sample is just one realization of the random graph , the density or probability mass function ( [ eq : density : whole ] ) is also the likelihood function . note that if one transforms to , the likelihood does not change .therefore , for identifiability , constraints on or are necessary . in this paper , we choose to set .other constraints are also possible , for example , or . in total , there are independent parameters and the natural parameter space becomes note that model ( [ eq : density : whole ] ) can serve as the null model for hypothesis testing , for example , , or be used to reconstruct directed networks and make statistical inference in a situation in which only the bi - degree sequence is available due to privacy consideration . moreover , many complex directed network models reply on the bi - degree sequences , indirectly or directly .thus , model ( [ eq : density : whole ] ) can be used for preliminary analysis of network data for choosing suitable statistics in describing network configurations , for example , .it is worth to note that the above discussions only consider independent edges .exponential random graph models ( ergms ) , sometimes referred as exponential - family random graph models , for example , , can be more general . if dependent network configurations such as -stars and trianglesare included as sufficient statistics , then edges are not independent and such models incur `` near - degeneracy '' in the sense of , in which almost all realized graphs essentially either contain no edges or are complete .it has been shown in that most realizations from many ergms look similar to the results of a simple erds rnyi model , which implies that many distinct models have essentially the same mle , and it was also proved and characterized in the degeneracy observed in the ergm with the counts of edges and triangles as the exclusively sufficient statistics .further , by assuming a finite dimension of the parameter space , it was shown in that the mle is not consistent in the ergm when the sufficient statistics involve -stars , triangles and motifs of -nodes ( ) , while it is consistent when edges are dyadic independent . in view of the model degeneracy and problematic properties of estimators in the ergm for dependent network configurations , we choose not to consider dependent edges in this paper .  for the remainder of the paper , we proceed as follows . in section [section : main ] , we first introduce notation and key technical propositions that will be used in the proofs .we establish asymptotic results in the cases of binary weights , continuous weights and discrete weights in sections [ subsection : binary ] , [ subsection : continuous ] and [ subsection : discrete ] , respectively .simulation studies are presented in section [ section : simulation ] .we further discuss the results in section [ section : discussion ] .since the technical proofs in sections [ subsection : continuous ] and [ subsection : discrete ] are similar to those in section [ subsection : binary ] , we show the proofs for the theorems in section [ subsection : binary ] in the , while the proofs for sections [ subsection : continuous ] and [ subsection : discrete ] , as well as those for proposition [ pro : inverse : appro ] , theorem [ theorem : newton : converg ] and lemmas [ lemma - binary-1 ] and [ lemma - binary-2 ] in section [ subsection : binary ] are relegated to the online supplementary material .[ section : main ]   let , , , . for a subset ,let and denote the interior and closure of , respectively .for a vector , denote by , the -norm of .for an matrix , let denote the matrix norm induced by the -norm on vectors in , that is ,   in order to characterize the fisher information matrix , we introduce a class of matrices .given two positive numbers and with , we say the matrix belongs to the class if the following holds : clearly , if , then is a diagonally dominant , symmetric nonnegative matrix and has the following structure : where ( by ) and ( by ) are diagonal matrices , is a nonnegative matrix whose nondiagonal elements are positive and diagonal elements equal to zero .define for and .then for , for and .we propose to approximate the inverse of , , by the matrix , which is defined as where when and when .note that can be rewritten as where , is an matrix whose elements are all equal to , and .  to quantify the accuracy of this approximation, we define another matrix norm for a matrix by .then we have the following proposition , whose proof is given in the online supplementary material .[ pro : inverse : appro ] if with , then for large enough  , where is a constant that does not depend on , and .note that if and are bounded constants , then the upper bound of the above approximation error is on the order of , indicating that is a high - accuracy approximation to .further , based on the above proposition , we immediately have the following lemma .[ lemma : inverse : bound ] if with , then for a vector , where .let and .henceforth , we will use to denote the fisher information matrix of the parameter vector and show . in the next three subsections , we will analyze three specific choices of the weight set : , , , respectively . for each case , we specify the distribution of the edge weights , the natural parameter space , the likelihood equations , and prove the existence , uniqueness , consistency and asymptotic normality of the mle .we defer the proofs for the results in section [ subsection : binary ] to the and all other proofs for sections [ subsection : continuous ] and [ subsection : discrete ] to the online supplementary material .   in the case of binary weights ,that is , , is the counting measure , and , are mutually independent bernoulli random variables with the log - partition function is and the likelihood equations are \\[-8pt]\nonumber b_j & = & \sum_{k=1,k\neq j}^n \frac { e^{\hat{\alpha}_k + \hat{\beta}_j } } { 1 + e^{\hat{\alpha}_k + \hat{\beta}_j } } , \qquad j=1,\ldots , n-1,\end{aligned}\ ] ] where is the mle of and .note that in this case , the likelihood equations are identical to the moment equations .we first establish the existence and consistency of by applying theorem [ theorem : newton : converg ] in the .define a system of functions : note the solution to the equation is precisely the mle .then the jacobian matrix of can be calculated as follows . for , and for , first , note that since the jacobian is diagonally dominant with nonzero diagonals , it is positive definite , implying that the likelihood function has a unique optimum .second , it is not difficult to verify that , thus proposition [ pro : inverse : appro ] and theorem [ theorem : newton : converg ] can be applied .let denote the true parameter vector .the constants , and in the upper bounds of theorem [ theorem : newton : converg ] are given in the following lemma , whose proof is given in the online supplementary material .[ lemma - binary-1 ] take and in theorem [ theorem : newton : converg ] .assume \\[-8pt]\nonumber & & \qquad \le \sqrt{(n-1)\log(n-1)}.\end{aligned}\ ] ] then we can choose the constants , and in theorem [ theorem : newton : converg ] as where and are constants .the following lemma assures that condition ( [ assumption : binary : a ] ) holds with a large probability , whose proof is again given in the online supplementary material .[ lemma - binary-2 ] with probability at least , we have   combining the above two lemmas , we have the result of consistency .  [ theorem : binary : con ] assume that with , where is a constant , and that , where denotes the probability distribution ( [ eq : density : whole ] ) on under the parameter .then as goes to infinity , with probability approaching one , the mle exists and satisfies further , if the mle exists , it is unique .next , we establish asymptotic normality of and outline the main ideas in the following .let denote the log - likelihood function of the parameter vector given the sample .note that , and is the fisher information matrix of the parameter vector .clearly , does not have an explicit expression according to the system of likelihood equations ( [ eq : likelihood - binary ] ) .however , if can be approximately represented as a function of with an explicit expression , then the central limit theorem for immediately follows by noting that under certain regularity conditions where denotes the element of .the identity between the likelihood equations and the moment equations provides such a possibility .specifically , if we apply taylor s expansion to each component of , the second - order term in the expansion is , which implies that obtaining an expression of crucially depends on the inverse of .note that according to the previous calculation . although does not have a closed form , we can use to approximate it and proposition [ pro : inverse : appro ] establishes an upper bound on the error of this approximation , which is on the order of if and are bounded constants .regarding the asymptotic normality of , we note that both and are sums of independent bernoulli random variables . by the central limit theorem for the bounded case in , page 289 , we know that and are asymptotically standard normal if diverges . since is an increasing function on when and a decreasing function when , we have in all , we have the following proposition .[ pro : binary : central ] assume that . if , then for any fixed , as , the vector consisting of the first elements of is asymptotically multivariate normal with mean zero and covariance matrix given by the upper left block of .the central limit theorem is stated in the following and proved by establishing a relationship between and ( see details in the and the online supplementary material ) .[ theorem : binary : central ] assume that . if , where is a constant , then for any fixed , as , the vector consisting of the first elements of is asymptotically multivariate normal with mean and covariance matrix given by the upper left block of .  by theorem [ theorem : binary : central ] , for any fixed , as , the convergence rate of is . since ,the rate of convergence is between and .  in this subsection , we have presented the main ideas to prove the consistency and asymptotic normality of the mle for the case of binary weights . in the next two subsections , we apply similar ideas to the cases of continuous and discrete weights , respectively .   another important case of model ( [ eq : density : whole ] ) is when the weight of the edge is continuous .for example , in communication networks , if an edge denotes the talking time between two people in a telephone network , then its weight is continuous . in the case of continuous weights ,that is , , is the borel measure and , are mutually independent exponential random variables with the density and the natural parameter space is to follow the tradition that the rate parameters are positive in exponential families , we take the transformation , and .the corresponding natural parameter space then becomes here , we denote by the mle of .the log - partition is and the likelihood equations are \\[-8pt]\nonumber b_j & = & \sum_{k=1;k\neq j}^n ( \hat{\alpha}_k + \hat{\beta}_j ) ^{-1},\qquad j=1 , \ldots , n.\end{aligned}\ ] ]  similar to section [ subsection : binary ] , we define a system of functions : the solution to the equation is the mle , and the jacobian matrix of can be calculated as follows .for , and for , it is not difficult to see that such that proposition [ pro : inverse : appro ] can be applied , and the constants in the upper bounds of theorem [ theorem : newton : converg ] are given in the following lemma .[ lemma - continuous-1 ] assume that satisfies for any and where is an absolute constant .then we have ^{-1}f\bigl(\bar{\bolds{\theta}}^*\bigr)\bigr{\vert}_\infty\le\biggl ( \frac{2c_1q_n^6}{nq_n^4}+\frac{1}{(n-1)q_n^2 } \biggr ) \sqrt { \frac{8(n-1)\log n } { \gamma q_n^2 } } .\ ] ] further , take and in theorem [ theorem : newton : converg ] , that is , an open ball . if , then we can choose and .the following lemma assures condition ( [ assumption : continuous : a ] ) holds with a large probability .[ lemma - continuous-2 ] with probability at least , we have   combining the above two lemmas , we have the result of consistency .  [ theorem : continuous : con ] assume that satisfies and .if , then as goes to infinity , with probability approaching one , the mle exists and satisfies further , if the mle exists , it is unique .again , note that both and are sums of independent exponential random variables , and is the fisher information matrix of .it is not difficult to show that the third moment of the exponential random variable with rate parameter is . under the assumption of , we have and note that if , the above expression goes to zero .this implies that the condition for the lyapunov s central limit theorem holds .therefore , is asymptotically standard normal if .similarly , is also asymptotically standard normal under the same condition . noting that = v_{i , i}^{-1 } ( g_i - { \mathbb{e}}(g_i ) ) + v_{2n,2n}^{-1 } ( b_n - { \mathbb{e}}(b_n ) ) ] , , and /(1/\hat{v}_{n+i , n+i}+1/\hat { v}_{n+j , n+j})^{1/2} ] .for any , we assume for define the newton iterates ^{-1 } f(\bolds { \theta}^{(k)}) ] .let ^{-1 } = w + s ] with .if , then with if with the constant satisfying , then as , . by lemma [ lemma - binary-1 ] and condition ( [ assumption : binary : a ] ) , for sufficiently small , \\ & & { } \times\frac{(\log n)^{1/2}}{n^{1/2 } } \bigl ( c_{11 } e^{6{\vert}\bolds{\theta}^*{\vert}_\infty } + c_{12}e^{2{\vert}\bolds{\theta}^*{\vert}_\infty } \bigr ) \\ & \le & o \biggl ( \frac{(\log n)^{1/2}e^{12{\vert}\bolds{\theta } ^*{\vert}_\infty } } { n^{1/2 } } \biggr ) + o \biggl(\frac{(\log n)^{1/2}e^{8{\vert}\bolds { \theta}^*{\vert}_\infty } } { n^{1/2 } } \biggr).\end{aligned}\ ] ] therefore , if , then as .consequently , by theorem [ theorem : newton : converg ] , exists .denote the limit as , then it satisfies by lemma [ lemma - binary-2 ] , condition ( [ assumption : binary : a ] ) holds with probability approaching one , thus the above inequality also holds with probability approaching one .the uniqueness of the mle comes from proposition [ prop : regularminimal ] .   before proving theorem [ theorem :binary : central ] , we first establish two lemmas .  [central : binary : lemma1 ] let and $ ] . then   note that where is a diagonal matrix , and by inequality ( c3 ) in , we have thus ,   [ central : binary : lemma2 ] assume that the conditions in theorem [ theorem : binary : con ] hold . if and , then for any , + o_p\bigl ( n^{-1/2 } \bigr).\ ] ]  by theorem [ theorem : binary : con ] , we have let .by taylor s expansion , for any , where and . by the likelihood equations ( [ eq : likelihood - binary ] ) , we have where and , equivalently , since , we have note that , and . by direct calculations ,we have and by proposition [ pro : inverse : appro ] , we have \le o\biggl ( \frac { e^{22{\vert}\bolds{\theta } ^*{\vert}_\infty } \log n } { n } \biggr).\ ] ] if and , then this completes the proof.  proof of theorem [ theorem : binary : central ] by ( [ eq - binary - central - theta ] ) , we have + \bigl[r \bigl\ { \mathbf{g } - { \mathbb{e}}(\mathbf{g } ) \bigr\}\bigr]_i + \bigl(v^{-1 } \mathbf{h}\bigr)_i.\ ] ] by lemmas [ central : binary : lemma1 ] and [ central : binary : lemma2 ] , if and , then +o_p \bigl(n^{-1/2}\bigr).\ ] ] therefore , theorem [ theorem : binary : central ] follows directly from proposition [ pro : binary : central ] .we thank runze li for the role he played as editor , an associate editor and two referees for their valuable comments and suggestions that have led to significant improvement of the manuscript .

Abstract: although asymptotic analyses of undirected network models based on degree sequences have started to appear in recent literature , it remains an open problem to study statistical properties of directed network models . in this paper , we provide for the first time a rigorous analysis of directed exponential random graph models using the in - degrees and out - degrees as sufficient statistics with binary as well as continuous weighted edges . we establish the uniform consistency and the asymptotic normality for the maximum likelihood estimate , when the number of parameters grows and only one realized observation of the graph is available . one key technique in the proofs is to approximate the inverse of the fisher information matrix using a simple matrix with high accuracy . numerical studies confirm our theoretical findings .   ./style / arxiv - general.cfg  ,


educational platforms such as coursera , udacity , and edx provide easy access to high level education to everyone who has a decent internet access . in september , these platforms had more than million users essentially , students attending the offered courses and this number is expected to further increase in the near future .the term `` massive open online course '' , or simply mooc , is very descriptive of the service these platforms offer .a mooc is the result of their partnership with a faculty member in a top university , whose role is to design the course and organize the course material so that it takes advantage of the most popular internet apps that the platform utilizes .courses offered include literally everything .even though the service provided is certainly useful , the viability of moocs will strongly depend on their sources of revenues .currently , investments from vcs have secured their survival for a short term , but what about their future ?a feature that could be the main source of revenue for moocs is the so - called _ verified certificate _ which the students can get at a cost of a few dozens of dollars .the verified certificate keeps information about the performance of a student in a course ( or in a chain of courses ) and can be used to justify a student s quality to potential employers .so , the verified certificate should have _ reliable information _ about the student performance in the courses she has participated in . even though the means to guarantee this in the traditional university system is well - established, achieving this in a mooc is a challenge .the big issue is in the massive student participation .of course , the internet provides tools so that organizing exams with huge numbers of students is possible .but what about assessment and grading ? as the most popular courses attract students or more and the vision of moocs enthusiasts is for millions of students per course , is grading of assignments or exams possible ?no doubt , professional graders would be extremely costly .organizing the material using multiple - choice questions and answers that could be graded automatically can not be an option when the students are asked to prepare an essay or a formal mathematical proof or express their critical thinking over some issue .grading is a typical example of a _ human computation _ task in these cases .the only solution that seems consistent to the moocs vision is known as _ peer grading _ , according to which the grading task is outsourced to the students that participated to the exam themselves .this approach has been already implemented in some moocs and standalone experimental tools such as ` crowdgrader.org `  , ` peergrading.org `  , and our own ` co - rank `  are already available .even though the approach seems straightforward , there are subtle implementation issues .for example , allowing the students to use cardinal scores is problematic , since they participate both in the exam and in grading and they may have incentives to assign low grades in order to improve their personal relative performance . even if we assume that they gradehonestly , their experience in doing so is very limited and the result will most probably be unreliable .  in this paper , we focus our attention on _ ordinal peer grading _ , which has recently received attention in the ai and machine learning community  .following the setting that we considered in our previous work  , each student gets a bundle of a small number ( our favourite number that we have extensively used recently is ) of exam papers so that each exam paper is given to the same number of students .each student has to _ rank _ the exam papers in her bundle ( in terms of quality ) and an _ aggregation rule _will then combine the ( partial ) rankings submitted by the students and come up with a final ranking of all exam papers ; this will be the grading outcome .information about the position of a student in the final ranking ( e.g. , top out of students ) can be included in her verified certificate .  in , we formally proved that a simple aggregation rule , inspired from borda s rule from social choice theory  , recovers correctly an expected fraction of of the pairwise relations in the underlying _ ground truth _ ranking , when bundles of size are used and students make no mistakes when grading .the assumption for a ground truth and the comparison of the grading outcome to it is similar in spirit to recent approaches that combine voting and learning  .the new aspect in ( as well as in the current paper ) is the relaxed requirement of recovering the ground truth only _approximately_. experimental results show that this rule has very good performance in an imperfect grading scenario inspired by a noisy model of generating random rankings that has been proposed by mallows  .note that , unlike other studies , we investigate the potential of applying ordinal peer grading exclusively , without involving any professionals in grading .in this paper , we follow a different approach . to explain our rationale, we remark that theoretical analysis requires to handle with extra care dependencies between several random variables that appear due to the distribution of exam papers to bundles .the analysis of borda was possible only due to its particular definition ; we have not managed to extend the analysis to any other aggregation rule .also , the notation in the theoretical guarantee for borda above hides large constant terms that constitute the bound of theoretical interest only .we would like to develop a `` theory '' for determining the performance of borda with the highest possible accuracy . and ,of course , why should we restrict our study only to borda ?instead , we define and study a large class of _ simple _ aggregation rules , which we call _ type - ordering aggregation rules _ ; the class includes borda .we present a theoretical framework for assessing the performance of each member of this class with respect to a series of performance objectives . a crucial step in our studyis that we have completely neglected the dependencies between the random variables that make the rigorous analysis difficult .this sacrifice of mathematical rigor is formally incorrect unless the number of students tends to infinity ; this can be justified by the massive participation in moocs .but the best justification of our approach is that the theoretical predictions of performance are experimentally shown to be _ exact _ , which means that the dependencies have no positive or negative impact on performance .furthermore , once ( statistical ) information about the grading behaviour of students and the desired performance objectives are known , our framework can serve as an _ optimization toolkit _ for selecting the optimal type - ordering aggregation rule .this requires an exact solution to an instance of the feedback arc set problem which , albeit np - hard in general , can be solved exactly for the instances that do arise .our theoretical framework allows us to obtain a series of results .for example , we establish that borda is the optimal type - ordering aggregation rule when students act as perfect graders .this is rather surprising , since borda is among the simplest aggregation rules in the class we consider .even though it was not observed to be optimal in any other scenario we considered , its performance is always extremely close to optimality .furthermore , as mentioned above , the optimization task of deciding the optimal aggregation rule strongly depends on the information about grading behaviour .we study how inaccuracies of this information affect the choice of the optimal aggregation rule and its performance for the mallows model .the results suggest a very minor impact and , essentially , a tiny sample of a student population is enough for building a fairly accurate model of grading behaviour .overall , our approach combines theory , simulations , and experimentation and is presented graphically in figure [ fig : approach ] .the lower chain of the figure describes what one would expect from a simulated exam .there is a student population and some of them participate in an exam .the preparation level of the participating students that determines their performance in the exam is a random variable following a uniform probability distribution .after the exam , each student acts as the grader of a small number of exam papers submitted by other students .the grading performance can depend on the preparation level as well .the grades are combined using the aggregation rule and the final ranking is compared to the ground truth to come up with the observed performance .the most interesting part of figure [ fig : approach ] is the upper chain .first , a _ field experiment _ can be used to extract information about the student population , translated into a noise model .we have performed such a field experiment with students in our home institution ; we describe it in detail and present the collected data later in the paper .these data are used to build a _ noise model _ which , together with the desired performance objective , are then given as input to the optimization engine which constructs the optimal aggregation rule for the particular noise model and performance objective , and reports a theoretical prediction about the performance the rule is expected to have .the optimal aggregation rule can also be applied to the grades from our simulated exams ( hence , the vertical arrow in figure [ fig : approach ] ) and a comparison of the theoretically predicted performance with the observed performance of the simulated exam will validate our theory .the rest of the paper is structured as follows .we begin with preliminary definitions and useful notation in section [ sec : prelim ] .the type - ordering aggregation rules and our theoretical framework are presented in section [ sec : rules ] .the field experiment and the validation of our framework are discussed in section [ sec : exp ] .we assume that students have participated in an exam and have submitted their exam papers .our approach to ordinal peer grading has three distinct tasks : the distribution of papers to students , the grading task by each student , and the aggregation of the grades into a final result .we describe these tasks in detail here and give definitions that will be useful later .all students that participated in the exam will have to participate in grading as well .the goal of the first task is to balance their grading load .this is done by distributing ( copies of ) each paper to the students so that each exam paper is given to exactly students and each student receives exactly ( distinct ) exam papers .the papers that a student receives form her _bundle_. these are the exam papers which the student has to grade .crucially , the bundle of a student should not contain her own exam paper .a -regular bipartite graph with nodes on each side of the bipartition ( called an -bundle graph in ) can be used to represent the distribution of exam papers to students .each node of sets and represents a student .an edge of the graph between a node and a node indicates that the exam paper of the student corresponding to node is in the bundle of the student corresponding to node .the restriction on the degree of the nodes of set means that each exam paper is given to exactly students and the restriction on the degree of the nodes of means that all bundles have size .  in our previous work , we considered bundle graphs that satisfy a particular structural property , namely they contain no cycle of length .this was a technical constraint that was required only in our theoretical analysis .experimental results in that paper indicate that uniformly random -regular bipartite graphs are almost as good as bundle graphs .these are the bundle graphs we considered in the current work .a random -regular graph can be built as follows .starting from the complete bipartite graph with node sets and , first remove the edges between nodes corresponding to the same students in and . then , draw a perfect matching uniformly at random among all perfect matchings of that do not include previously removed edges .repeat this step times ; this is possible due to hall s matching theorem since , at the beginning of every step , the edges that have not previously been removed form a regular bipartite graph .the union of the edges in the perfect matchings forms the bundle graph .   throughout the paper ,we assume that there is an underlying strict ranking of the exam papers , the _ ground truth _ , which we aim to recover . as it will shortly become apparent ,the setting we consider is so restrictive that we should not expect to recover the ground truth exactly . instead, we would like to recover the ground truth approximately .a restriction of our setting is that each student is given only exam papers to grade .another restriction is that the grading task for each student is simply to rank the exam papers in her bundle , in decreasing order of quality .we consider different scenarios for the grading behaviour of the students . in a first scenario ,assume that , after the end of the exam , the instructor announces indicative solutions and gives detailed instructions that the students can use during grading . here, we assume that students will act as _perfect graders_. admittedly , this is an unrealistic assumption but we include it as an extreme case in our study together with many others .our most realistic scenario poses another challenging restriction . in general , we assume that students make mistakes when grading . for modelling purposes , we assume that a student grades as follows . when she receives a bundle of papers , she draws a random ranking according to a probability distribution that characterizes the grading behaviour of all students participating in the exam .so , the behaviour of a student in an imperfect grading scenario is characterized by a _ noise matrix _} ] and affects her position in the ground truth and her ability to grade as well .the ground truth is the ranking of the students in decreasing order of quality .a student of quality performs the grading task as follows : she considers every pair of exam papers and in her bundle , such that appears ahead of in the ground truth , and temporarily determines with probability and with probability ; the pairwise relation will evolve into her ranking of the exam papers in her bundle .if , after considering all pairs of exam papers in the bundle , the pairwise relation is cyclic , the whole process is repeated from scratch . otherwise , the ranking of the exam papers in the bundle induced by is the grading outcome of student .the above process has been implemented and is used in our simulations with bundles of size . in our theoretical analysis , we have computed the corresponding noise matrix by sampling students with uniform qualities and simulating the grading behaviour described above .the resulted noise matrix is presented here for ease of reference .\end{aligned}\ ] ] due to its similarities with the well - known mallows model  for generating random rankings , we refer to this grading behaviour as mallows grading .the third important task is to aggregate the partial rankings provided by the graders into a final output ranking .this is done using an _aggregation rule_. a simple but very compelling aggregation rule is inspired by the borda count voting rule . in our context, borda computes a score for each exam paper by examining the positions it has in the rankings of the graders that have this exam paper in their bundles .a first position by an exam paper contributes points to its score , a second position contributes points , and so on .the outcome of borda is a ranking of the exam papers in non - increasing order of their borda scores .when we use borda , we assume that ties are broken uniformly at random but other tie - breaking schemes could be considered as well .more generally , a _ positional scoring aggregation rule _ could use a different score vector with ( and ) in order to increase the score of each exam paper by whenever it is ranked -th by a grader .  in our previous paper , we also considered several other aggregation rules such as a rule that we call random serial dictatorship ( rsd ) as well as rules that are based on appropriately defined markov chains .rsd is very slow in the computation of the final outcome and , even though it performs remarkably well with perfect graders , it has a poor performance in simulated exams with mallows graders. we will not consider it in the current paper ; actually , applying it with input from graders , which is the typical scenario we consider in this paper , is a computational challenge .the aggregation rules that are based on markov chains were defined in an unsuccessful attempt to distinguish between high and low quality graders and put more weight on the partial rankings of the former .these ideas are not considered in this work either .instead , we focus on much simpler aggregation rules .we will use the term _ type _ to refer to the grading result of an exam paper .the grading result for it consists of the ranks it gets from the graders that have it in their bundles .so , the type is a vector of integers from =\{1 , 2 , ... , k\} ] .we will usually identify an exam paper as a real number  ] with ( i.e. , exam paper has a better rank in the ground truth compared to exam paper ) and add one point if has a better type than according to the ordering , and half a point if both exam papers have the same type . in this last case ,the tie is resolved uniformly at random and the probability that the correct pairwise relation will be recovered is .hence , denoting the expected fraction of pairwise relations recovered by the rule by ( we will refine this notation in a while ) and the event that exam paper gets type after grading by , we have }+\frac{1}{2}\sum_{\sigma}{\pr[x \rhd \sigma \text { and } y \rhd \sigma]}\right){\,\mathrm{d}}y}{\,\mathrm{d}}x}\\ & = & \sum_{\sigma,\sigma':\sigma\succ\sigma'}{\int_0 ^ 1{\int_x^1{\pr[x \rhd \sigma \text { and } y \rhd \sigma']{\,\mathrm{d}}y}{\,\mathrm{d}}x}}+\frac{1}{2}\sum_{\sigma}{\int_0 ^ 1{\int_x^1{\pr[x \rhd \sigma \text { and } y \rhd \sigma]{\,\mathrm{d}}y}{\,\mathrm{d}}x}}\end{aligned}\ ] ] the first sum runs over all pairs of different types with order and the second sum runs over all types .the scary ( at first glance ) double integral can be hidden under the notation to obtain we will use the term _ weight _ to refer to the quantity . our assumption for an infinite number of students nullifies any dependencies between the rank vectors that exam papers and get after grading .so , the events and are independent and the definition of the weight becomes \cdot \pr[y \rhd \sigma']{\,\mathrm{d}}y}{\,\mathrm{d}}x}.\end{aligned}\ ] ]  let us now compute the probability that exam paper gets type . by considering all ways to distribute the entries of the type vector as ranks of an exam paper by the graders that handle it ( ignoring symmetries ) , there are ways that the exam paper can get type , where is the number of graders that have the exam paper ranked -th . again ,due to our assumption for infinitely many students and the uniform inclusion of them into bundles , the quality of each exam paper included in a bundle does not affect the quality of other exam papers ( in the same or different bundles ) .clearly , the grading by different students is performed without dependencies either . denoting by the event that exam paper is ranked -th in a bundle , the probability that is of type is & = & n(\sigma ) \prod_{i=1}^k { \pr[{\mathcal{e}}(x,\sigma_i ) ] } .\end{aligned}\ ] ] to compute  ] is equal to the number of ways we can choose exam papers to be ahead of times the probability that all of them will indeed be ahead of in the bundle times the probability that the rest exam papers in the bundle will have true ranks worse than .we use to denote the set of all -entry vectors with  ] is a univariate polynomial of degree .then , the double integral can be computed analytically .the computation is tedious but straightforward . in appendix[ app : sec : compute - integral ] , we show how to compute such integrals .   the approach in section [ subsec : framework ] suggests a general way of evaluating the performance of any type - ordering aggregation rule . in order to compute the expected number of correctly recovered pairwise relations, it suffices to use equations ( [ eq : sum - of - weights ] ) , ( [ eq : weight ] ) , and ( [ eq : prob - x - sigma - is - a - poly ] ) . equation ( [ eq : prob - x - sigma - is - a - poly ] ) can be used to obtain  ] which returns the importance of measuring a correctly recovered relation between two students and with . in the presentation of our framework in section [ subsec : framework ] , we have assumed such a function with for every student pair .the two scenarios of the previous paragraph can be captured by the function ( i ) when and otherwise , and ( ii ) when ( and ) and otherwise .many other performance objectives can be defined including ones in which the function returns fractional values between and .the only modification in the computation of section [ subsec : framework ] is in the computation of the weights which should become \cdot \pr[y \rhd \sigma']{\,\mathrm{d}}y}{\,\mathrm{d}}x}.\end{aligned}\ ] ] in order to capture the generality of the scenarios considered , we overload the notation for the performance measure to specify the bundle size , the aggregation rule , the noise matrix describing the grading behaviour , and the performance objective .consider a type - ordering aggregation rule that is applied on -sized partial rankings from an infinite population of students with grading behaviour that follows a noise matrix .then , the fraction of correctly recovered pairwise relations that satisfy the performance objective given by the bivariate function is where is given by ( [ eq : new - weight ] ) and  ] with \end{aligned}\ ] ] which represents a `` realistic '' noise model .the information in the matrix was obtained by measuring the frequency that the -th ranked exam paper by students should be correctly ranked at position .for example , out of students ranked third an exam paper in their bundles which should have been ranked fourth ; thus , .this matrix is used in the computation of the optimal type - ordering aggregation rules for realistic scenarios according to the methodology developed in sections [ subsec : framework ] and [ subsec : optimal ] . observe that matrix does not include any information about the correlation between the grading behaviour and the quality of the student that acts as grader ;actually , this is a feature that our framework completely neglects .surprisingly , the experimental results that we present in the following indicate that storing information about this relation is not necessary for making accurate performance predictions .our next step was to compute the optimal type - ordering aggregation rules for several scenarios .we have kept the same bundle size of in all scenarios and distinguish between the realistic and mallows noise models by using the corresponding matrices and defined in equations ( [ eq : real ] ) and ( [ eq : mallows ] ) , respectively . as performance objectives, we have considered the following :  * all2all : the total number of all correctly recovered pairwise relations ; * th-10% and th-50% : the total number of correctly recovered relations between pairs that include an exam paper that is ranked in the top and top in the ground truth , respectively ; * acc-2% and acc-5% : the total number of correctly recovered relations between pairs with positions that differ by at least and in the ground truth , respectively .we have used the theoretical framework presented in sections [ subsec : framework ] and [ subsec : optimal ] ; of course , computations have been automated .all the computational results that we report in the following have been obtained using an intel 12-core i7 machine with 32 gb of ram running windows 7 .our methods have been implemented in c using the gnu multiple precision arithmetic library ( gmp ) and in matlab r2013a . in particular , high precision is absolutely necessary in order to compute the weights even for bundles of size since , by inspecting equations ( [ eq : prob - x - sigma - is - a - poly ] ) and ( [ eq : new - weight ] ) carefully , we can see that there are products with more than factors and factorials of integers up to that are involved in the computations .a first remark is that the algorithm that we have used to solve fas exactly is very fast .this is due to the fact that strongly connected components have very small size . in all cases we considered , among the 462 different types that we can have for bundles of size , more than 97% of themform singleton components and the maximum component size never exceeded 26 ( for the -sample approximation of mallows that we discuss in section [ subsec : apx ] ) .their distribution for scenarios with realistic and mallows noise models for all performance objectives is depicted in table [ tab : component - size ] .  !1.5ptc|c|c|c!width 1.5ptc|c|c|c!width 1.5pt & & + size & all2all & th-50% & acc-2% & acc-5% & all2all & th-50% & acc-2% & acc-5% + 1 & 448 & 460 & 449 & 451 & 453 & 459 & 449 & 449 + 37 & 13 & 2 & 12 & 10 & 6 & 3 & 10 & 12 + 811 & 1 & 0 & 1 & 1 & 2 & 0 & 2 & 0 + 12 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 1 + max & 10 & 3 & 10 & 10 & 20 & 4 & 20 & 20 +  recall that our theoretical framework assumes an infinite number of students and , hence , neglects apparent dependencies between random variables that certainly exist in scenarios with finite number of students .so , the information in table [ tab : data ] is rather surprising and shows that , besides our assumptions , our theory provides extremely accurate predictions for the performance of aggregation rules in practice .note that the values in table [ tab : data ] are percentages and we never observed differences beyond the second decimal point between the theoretically predicted value and the experimental one . also , note that we have used students in our experiments .this number is lower than the vision for the most popular courses that will be offered by moocs in the near future ; the predictions become even more accurate for higher numbers of students .  ! width 1.5ptc!width1.5ptc|c!width 1.5ptc|c|c|c!width 1.5ptc|c|c|c!width 1.5pt noise & & & + setting & theory & & & & & + method & borda & borda & opt & borda & opt & borda & opt & borda & opt & borda + all2all & 92.01 & 92.02 & 80.01 & 79.57 & 80.09 & 79.57 & 85.15 & 84.38 & 85.16 & 84.39 + th-10% & 96.94 & 96.95 & 87.61 & 87.18 & 87.60 & 87.17 & 92.05 & 90.52 & 92.07 & 90.54 + th-50% & 94.13 & 94.14 & 83.62 & 83.43 & 83.62 & 83.43 & 88.39 & 87.80 & 88.40 & 87.81 + acc-2% & 93.57 & 93.57 & 81.27 & 80.73 & 81.27 & 80.74 & 86.52 & 85.72 & 86.52 & 85.73 + acc-5% & 95.47 & 95.47 & 82.97 & 82.42 & 82.97 & 82.42 & 88.42 & 87.61 & 88.42 & 87.62 +  table [ tab : data ] contains the expected values ( from 1000 simulations ) of the performance measure for each grading scenario and corresponding optimal aggregation rule ( as well as for borda ) . even though borda is never optimal in any imperfect grading scenario that we considered , its performance is always very close to the optimal rule . in particular , borda is never more than worse than the optimal rule in realistic scenarios ( for the acc-5% performance objective ); such differences can be up to in scenarios with mallows noise ( and the th-10% objective ) .figure [ fig : borda_vs_opt - all2all - acc5-th10 ] reports detailed information for all these simulations . clearly , the performance of the aggregation rules for all the objectives that we considered is sharply concentrated around the expected values .the clouds of points are very close to the diagonal ( corresponding to values with equal coordinates ) in the realistic scenarios and only marginally further from it in the mallows scenarios .  0.3  0.3  0.3  0.3  0.3  0.3  0.3  a final comment on the performance of the optimal type - ordering aggregation rules is that they are _ extremely robust_.even though they have been optimized with respect to a particular performance objective , they perform very well with respect to other objectives as well .figure [ fig : displ - int - top20 ] shows measurements of properties that can not be expressed as performance objectives under our framework .each plot shows data about borda and the optimal ( under the all2all objective ) type - ordering aggregation rules in scenarios with perfect , realistic , and mallows grading .borda in the perfect grading scenario achieves the best performance with respect to these objectives as well . actually , its performance in this scenario can serve as the optimistic barrier for every type - ordering aggregation rule in any ( imperfect ) grading scenario .more interestingly , borda has performance that is very close to the optimal rule for realistic grading ( the corresponding curves almost coincide in figure [ fig : displ - int - top20 ] ) and slightly worse for mallows grading .these results are in sync to those in table [ tab : data ] , figure [ fig : borda_vs_opt - all2all - acc5-th10 ] , and figure [ fig : displ - int - top20 ] and showcases the robustness of borda in different scenarios .0.3  0.3  0.3   the realistic noise model that we built in section [ subsec : real - exp ] is , by definition , an approximation of the students in our home institution . besides limitations that have to do with our modelling assumptions , it has the obvious drawback that it has been built using a very small fraction of our students .so far , the reader should have been convinced that the aggregation rules we have built are indeed optimal for a large population that inherits the quality and grading performance of this small fraction of students ; this has been the focus of our experiments with realistic grading .what is far from clear is whether these aggregation rules will perform equally well for the whole population of the students in our home institution . to see the importance of this question ,imagine it in the planetary scale that moocs envision .can we make safe predictions for huge student populations by sampling a tiny fraction of them , building a noise model as we did in section [ subsec : real - exp ] , and then select optimal type - ordering aggregation rules as we did in section [ subsec : optimal - exp ] ?  we give a positive answer to this question by considering mallows grading scenarios . with mallows , we have the luxury of a well - defined noise model for the grading behaviour of a huge student population and we have used it in order to compute optimal aggregation rules for this model. this information will be used only for assessing the approach presented in the following .instead , we will pretend that no information about grading behaviour is available and all we can do is to apply ( actually , to simulate ) a field experiment like the one we presented in section [ subsec : real - exp ] on a tiny fraction of the students in order to come up with a noise matrix . in this way, we will compute an approximation of the true noise model .we have followed this approach using samples of mallows graders of size and .the two noise model matrices we obtained are as follows : \ ] ] and \ ] ] the matrices have been used to compute the optimal type - ordering aggregation rules for the five performance objectives .interestingly , the instances of fas that we had to solve were slightly harder now , particularly for the -sample noise model , where strongly connected components of size up to emerged in the auxiliary graph .still , our methodology was applied smoothly and allowed us to compute optimal rules .  !1.5ptc|c!width 1.5ptc|c!width 1.5ptc|c!width 1.5pt # samples & & & + setting & theory & & theory & & theory & + all2all & 84.95 & 84.95 & 85.14 & 85.15 & 85.15 & 85.16 + th-10% & 91.82 & 91.85 & 92.05 & 92.04 & 92.05 & 92.07 + th-50% & 88.21 & 88.21 & 88.39 & 88.38 & 88.39 & 88.40 + acc-2% & 86.31 & 86.31 & 86.51 & 86.51 & 86.52 & 86.52 + acc-5% & 88.19 & 88.20 & 88.41 & 88.41 & 88.42 & 88.42 +  table [ tab : learning ] shows the theoretical prediction and observed performance values of these rules ( by applying the rules on 1000 simulated exams with mallows students ) .the performance of the aggregation rules that were computed using the -sample approximation of mallows are already amazingly close to those for mallows .for the rules that we computed using the -sample approximation , it is almost impossible to distinguish them from the mallows - optimal ones in terms of performance .a more refined graphical representation of these findings is given in figure [ fig : apx_vs_mallows ] ( best viewed in color ) .each plot contains a blue and a red cloud of points each corresponding to a single simulated exam with students .the blue points ( respectively , red points ) measure the performance of the optimal rule for the -sample ( respectively , -sample ) mallows approximation versus the mallows - optimal rule .the blue cloud almost coincides with the diagonal in each plot , indicating an optimal approximation of the mallows - optimal rule .the red cloud is distinct ( there is a negligible `` intersection '' of blue and red points only for the th-10% performance objective ) but very close . to realize how close the two clouds are , almost the whole cloud of points for borda ( from figures [ fig : borda_vs_opt - all2all - acc5-th10](d ) , [ fig : borda_vs_opt - all2all - acc5-th10](e ) , and [ fig : borda_vs_opt - all2all - acc5-th10](f ) )would be located outside the plot area of figure [ fig : apx_vs_mallows ] ( if we attempted to plot it ) .  0.3  0.3  0.3  we conclude the presentation of our experimental results with a comparison of the type orderings of the optimal aggregation rules for mallows and its -sample and -sample approximations ;these are presented in table [ tab : rules ] .therein , we can see that the optimal rule for the -sample noise model according to the all2all performance objective is very close ( but not identical ) to the mallows - optimal rule .the optimal rule for the -sample noise model is substantially different ( this difference is more apparent in lower positions of the ordering which can not be included here ) .an interesting characteristic of optimal rules for mallows and its approximations is that the orderings of types are not monotonic .for example , type is always ahead of .this justifies our decision to study type - ordering aggregation rules and ignore positional scoring rules ; clearly , no positional scoring rule would come up with non - monotonic orderings of types .  !1.5ptc!width 1.5ptc!width 1.5ptc!width 1.5ptc!width 1.5ptc!width 1.5pt pos . & mallows & -sample & -sample + & & approx . & approx .+ 1 & & & + 2 & & & + 3 & & & + 4 & & & + 5 & & & + 6 & & & + 7 & & & + 8 & & & + 9 & & & + 10 & & & + 11 & & & + 12 & & & + 13 & & & + 14 & & & +  10  n. alon . ranking tournaments . , 20(1):137142 , 2006 .f. brandt , v. conitzer , u. endriss , j. lang , and a. d. p. ( eds . ) . .cambridge university press , 2016 .i. caragiannis , g. a. krimpas , m. panteli , and a. a. voudouris .co - rank : an online tool for collectively deciding efficient rankings among peers . in _ proceedings of the 30th aaai conference on artificial intelligence ( aaai ) _ , 2016 , forthcoming .i. caragiannis , g. a. krimpas , and a. a. voudouris . aggregating partial rankings with applications to peer grading in massive online open courses . in _ proceedings of the 14th international conference on autonomous agents and multi - agent systems ( aamas )_ , pages 675683 , 2015 .i. caragiannis , a. d. procaccia , and n. shah .when do noisy votes reveal the truth ? in _ proceedings of the 14th acm conference on electronic commerce ( ec ) _ , pages 143160 , 2013 .i. caragiannis , a. d. procaccia , and n. shah .modal ranking : a uniquely robust voting rule . in _ proceedings of the 28th aaai conference on artificial intelligence _ , 2014 .f. chierichetti and j. m. kleinberg . voting with limited information and many alternatives ., 43(5):16151653 , 2014 .l. de alfaro and m. shavlovsky .rowdgrader : a tool for crowdsourcing the evaluation of homework assignments . in _ sigcse _ ,pages 415420 , 2014 .  c. kenyon - mathieu and w. schudy . how to rank with few errors . in _ proceedings of the 39th annual acm symposium on theory of computing ( stoc ) _ , pages 95103 , 2007 .c. kulkarni , k. p. wei , h. le , d. chia , k. papadopoulos , j. cheng , d. koller , and s. r. klemmer .peer and self assessment in massive online classes . , 20(6 ) , 2013 .e. law and l. von ahn . .synthesis lecture on artificial intelligence and machine learning .morgan & claypool , 2011 .  c. l. mallows .non - null ranking models . , 44:114130 , 1957 .  c. piech , j. huang , z. chen , c. do , a. ng , and d. koller .tuned models of peer assessment in moocs . in _ proceedings of the 6th international conference on educational data mining ( edm ) _ ,pages 153160 , 2013 .  k. raman and t. joachims .methods for ordinal peer grading . in _ proceedings of the 20th acm conference on knowledge discovery and data mining ( kdd ) _ , pages 10371046 , 2014 .n. b. shah , j. k. bradley , a. parekh , m. wainwright , and k. ramchandran .a case for ordinal peer - evaluation in moocs . in _ neural information processing systems ( nips ) : workshop on data driven education _ , 2013 .t. walsh .the peerrank method for peer assessment . in _ proceedings of the 21st european conference on artificial intelligence ( ecai ) _, pages 909914 , 2014 .we now elaborate on how to analytically compute the weight ; the following computations are implemented in algorithm [ alg : weights ] . recall that is given by equation ( [ eq : new - weight ] ) , i.e. , \cdot \pr[y \rhd \sigma']{\,\mathrm{d}}y}{\,\mathrm{d}}x } , \end{aligned}\ ] ] and that the performance objective bivariate function indicates whether a correctly recovered pairwise relation between two students and ( with ) should be accounted for or not .all perfomance objectives we consider in this paper can generically be described by such a function with when  ] for appropriate values of  ] is given by equation ( [ eq : prob - x - sigma - is - a - poly ] ) and is a univariate polynomial of degree .hence , it can be written as & = & \sum_{s=0}^{k^2-k } { c_s(\sigma ) x^s } , \end{aligned}\ ] ] where the coefficient with are computed by equation ( [ eq : prob - x - sigma - is - a - poly ] ) , and have been included for completeness in the first part of algorithm [ alg : weights ] .the inner integral in equation ( [ eq : weight - parameterized ] ) is computed as follows : } { \,\mathrm{d}}y & = & \int_{x+\gamma}^\delta {  \sum_{s=0}^{k^2-k } { c_s(\sigma ' ) y^s } } { \,\mathrm{d}}y \\ & = & \sum_{s=0}^{k^2-k } { c_s(\sigma ' ) \int_{x+\gamma}^\delta { y^s } { \,\mathrm{d}}y } \\ & = & \sum_{s=0}^{k^2-k } { \frac{c_s(\sigma')}{s+1 } \left ( \delta^{s+1 } - ( x+\gamma)^{s+1 } \right ) } \\ & = & \sum_{s=0}^{k^2-k } { \frac{c_s(\sigma')\delta^{s+1}}{s+1 } } - \sum_{s=1}^{k^2-k+1 } { \frac{c_s(\sigma')(x+\gamma)^s}{s } } .\end{aligned}\ ] ] using the fact that for , , and , we obtain } { \,\mathrm{d}}y & = & \sum_{s=0}^{k^2-k } { \frac{c_s(\sigma')\delta^{s+1}}{s+1 } } - \sum_{s=1}^{k^2-k+1}{c_s(\sigma')\sum_{i=0}^s { \frac{1}{s}{s \choose i}\gamma^{s - i } x^i } } .\end{aligned}\ ] ] observe that the inner integral is also a univariate polynomial of degree and it can be written as } { \,\mathrm{d}}y = \sum_{t=0}^{k^2-k+1 } { d_t(\sigma ) x^t } , \end{aligned}\ ] ] where the coefficient with are computed by equation ( [ eq : y - integral ] ) at the second part of algorithm [ alg : weights ] .  by substituting equations ( [ eq : x - poly - form ] ) and ( [ eq : y - poly - form ] ) in equation ( [ eq : weight - parameterized ] ) , we obtain \bigg ( \int_{x+\gamma}^\delta { \pr[y \rhd \sigma']{\,\mathrm{d}}y } \bigg ) { \,\mathrm{d}}x } \\ & = & \int_\alpha^\beta { \sum_{s=0}^{k^2-k } { c_s(\sigma ) x^s } \sum_{t=0}^{k^2-k+1 } { d_t(\sigma ' ) x^t } } { \,\mathrm{d}}z \\ & = & \sum_{s=0}^{k^2-k}\sum_{t=0}^{k^2-k+1 } { c_s(\sigma ) d_t(\sigma ' ) \int_\alpha^\beta { x^{s+t } } { \,\mathrm{d}}z } \\ & = & \sum_{s=0}^{k^2-k}\sum_{t=0}^{k^2-k+1 } { \frac{c_s(\sigma ) d_t(\sigma')}{s+t+1 } \bigg ( \beta^{s+t+1 } - \alpha^{s+t+1 } \bigg ) } .\end{aligned}\ ] ] this computation is described in the last part of algorithm [ alg : weights ] .

Abstract: ordinal peer grading has been proposed as a simple and scalable solution for computing reliable information about student performance in massive open online courses . the idea is to outsource the grading task to the students themselves as follows . after the end of an exam , each student is asked to rank in terms of quality a bundle of exam papers by fellow students . an aggregation rule will then combine the individual rankings into a global one that contains all students . we define a broad class of simple aggregation rules and present a theoretical framework for assessing their effectiveness . when statistical information about the grading behaviour of students is available , the framework can be used to compute the optimal rule from this class with respect to a series of performance objectives . for example , a natural rule known as borda is proved to be optimal when students grade correctly . in addition , we present extensive simulations and a field experiment that validate our theory and prove it to be extremely accurate in predicting the performance of aggregation rules even when only rough information about grading behaviour is available .


quantum measurement theory imposes fundamental limitations on the information extractable from a quantum system .although the evolution of quantum systems can be described deterministically , the measurement operation always leads to nondeterministic outcomes . in order to obtain a desired accuracy , measurement of a particular observable needs to be repeated over an ensemble of identical quantum systems .in addition , for systems with many degrees of freedom , one usually needs to measure a set of non - commuting observables corresponding to independent parameters of the system .characterization of state or dynamics of a quantum system can be achieved by a family of methods known as quantum tomography . in particular, quantum process tomography provides a general experimental procedure for estimating dynamics of a system which has an unknown interaction with its embedding environment for discrete or continuous degrees of freedom . in these methods ,the full information is obtained by a complete set of experimental settings associated with the set of required input states and non - commuting measurements . in recent developments , it has been demonstrated that the minimal number of required experimental settings can indeed be substantially reduced by using degrees of freedom of auxiliary quantum systems correlated with the system of interest .generally it is possible to completely characterize a quantum device with a single experimental setting . a correlated input state of the combined system and an ancilla is subjected to the unknown process , and a generalized measurement or positive operator valued measure ( povm ) is performed at the output .however , in order to realize such a generalized measurement one needs to effectively generate many - body interactions which are not naturally available and/or controllable .quantum simulation of such many - body interactions is in principle possible , but generally requires an exponentially large number of single- and two - body interactions with respect to system s degrees of freedom .an alternative method that circumvents the requirement for many - body interactions , yet allows simultaneous non - commuting observables through a single measurement setting , is known as direct characterization of quantum dynamics ( dcqd ) .the construction of the full information about the dynamical process is then possible via preparation of a set of mutually unbiased entangled input states over a subspace of the total hilbert space of the principle system and an ancilla .the dcqd approach was originally developed with the assumptions of ideal ( i.e. , error - free ) quantum state preparation , measurement , and ancilla channels .however , in a realistic estimation process , due to decoherence , limited preparation / measurement accuracies , or other imperfections , certain errors may occur hindering the overall procedure .in this work , we introduce an experimental procedure for using generic two - body interactions to perform quantum process estimation on a subsystem of interest .we employ this approach to generalize the dcqd scheme to the cases in which the preparations and measurements are realized with known systematic faulty operations .we demonstrate that in some specific , but physically motivated , noise models , such as the generalized depolarizing channels , only classical correlation between the system and ancilla suffice . moreover , for these situations the data processing overhead is fairly small in comparison to the ideal dcqd . given a noise model , one can find the optimal input states by minimizing the errors incurred through the inversion of experimental data .thus , we provide the optimal input states for reducing the inversion errors in the noiseless dcqd scheme .the structure of the paper is as follows . in sec .[ general ] , we set the framework for process tomography schemes where we have faulty rather than ideal faulty bell - state analyzers , emphasizing the dcqd approach .next , in sec . [ models ], we demonstrate the applicability of our framework through some simple , yet important examples of noise models .we conclude by summarizing the paper in sec .[ summary ] .let us consider a given quantum system composed of two correlated physical subsystems and for a time duration the two subsystems are decoupled , thus experiencing different quantum processes , and then they interact with each other again . the task is to characterize the unknown quantum process acting on the subsystem of interest , assuming we have prior knowledge about the dynamics of subsystem plus their initial and final correlations .another similar scenario can also be envisioned .given two controllable quantum systems and that are made to sufficiently interact before and after a time duration , we wish to estimate the unknown dynamics acting on system for such time interval , assuming the dynamics of the ancilla system and the interaction between two systems is known with a certain accuracy .much progress has been made in creating and characterizing two - body correlations in a variety of physical systems and interactions , including nuclear magnetic resonance ( nmr ) systems interacting through an ising hamiltonian together with refocusing or dynamical decoupling techniques , atoms / molecules in cavity quantum electrodynamics ( qed ) , trapped ions interacting via the jaynes - cummings hamiltonian driven by laser pulses and vibrational degrees of freedom , and photons correlated in one or many degrees of freedom , e.g. , generated by parametric - down conversion or four - wave mixing .other approaches include spin - coupled quantum dots , superconducting qubits controlled by external electric and/or magnetic fields , and chromophoric complexes coupled through forster / dexter interactions and monitored or controlled via ultra - fast nonlinear spectroscopy .however , in almost all of these systems , the entangled bell - state preparations and measurements , which generically are the basic building blocks of quantum information processing , hardly achieve high fidelities ; at the very least they are certain to be imperfect at some level , and this will limit their use for tomography . our goal is to determine the optimal states and measurement strategy that will minimize the deleterious effects of the nonidealities assumed known on process tomography .    ( 0,10 ) ( -113,70) ( -115,110) ( -168,60) ( -67,60)  we consider the cases where we can simulate initial or final two - body correlations in the above schemes by performing an ideal ( generalized ) bell - state preparation ( or measurement ) followed by a _ known _ faulty completely - positive ( cp ) quantum map acting on the entire systems involved . it should be noted that not every cp maps can be written as concatenation of two other cp maps . in other words , there exist cp maps that are `` indivisible '' , in the sense that , for such a map , there do not exist cp maps and such that , where neither or are unitary .nonetheless , all full - rank cp maps in the sense of the kraus representation  are divisible .  here, we include quantum maps acting on system in the preparation or measurement maps .this approach naturally provides a generalization of the dcqd scheme to the cases of faulty preparations , measurement and ancilla channels where the noise is already known see fig .[ gdcqd - fig ] . for simplicity , in this workwe concentrate only on one - qubit systems and the dcqd scheme ( summarized in table [ dcqd - tab ] ) . however , generalization of the framework is straightforward for other process estimation schemes and for dcqd on qudit systems with being a power of a prime ( according to the construction of ref . ) ..the ideal direct characterization of single - qubit .here , where and , and , , are eigenstates of the pauli operators , , and . is the projector on the bell state , and similarly for the other projectors .see refs .determination of optimal values of and is discussed in the text .[ cols="<,^,^ " , ]   [ dcqd - tab ]  let us consider the qubit of interest and the ancillary qubit prepared in the maximally entangled state .we first apply a _ known _ quantum error map to and : , where and are the identity and pauli operator for a single qubit .next , we apply the unknown quantum map to qubit a ; this is what we are trying to determine : =\sum_{mn}\chi _ { mn}\sigma _ { m}^{a}(\sum_{pqrs}\chi _ { pqrs}^{(\text{i})}\sigma _ { p}^{a}\sigma _ { q}^{b}\rho \sigma _ { r}^{b}\sigma _ { s}^{a})\sigma _ { n}^{a}. ] , , and for corresponds to the bell - states , , , and , respectively , where .( henceforth throughout this manuscript , superscripts refer to the bell - state basis and subscripts refer to the pauli operator basis . )the are known functions of , , , and .therefore , the overall output state can be rewritten as follows :   we now apply the standard dcqd data analysis to estimate the matrix elements of ( representing ) . after performing a bsm , i.e., measuring on this state , we obtain the bell - state with probability = \sum_{kk^{\prime } mn}\lambda _ { kk^{\prime } , mn}^{(j)}\chi _ { mn},\end{aligned}\]]where  ] }_{\varepsilon}}{\rightarrow } \frac{1-\varepsilon}{4}\openone\otimes\openone + \varepsilon\rho^{(i ) } , \\p^{jj}\overset{\mathcal{d}^{[2]}_{\varepsilon^{\prime } } } { \rightarrow } \frac{1-\varepsilon^{\prime } } { 4}\openone\otimes \openone + \varepsilon^{\prime jj},\end{aligned}\ ] ] where and could be independent of each other or correlated ( e.g. , ) .these errors result in the following noisy data processing of the measurement results of dcqd :   \rightarrow { }  \frac{(1-\varepsilon)(1-\varepsilon^{\prime } ) } { 16 } \text{tr}[\mathcal{e}(\openone ) \otimes\openone ] + \frac{\varepsilon^{\prime } ( 1-\varepsilon)}{4 } \text{tr}[\mathcal{e } ( \openone)\otimes \openone p^{jj } ] + \frac{\varepsilon(1-\varepsilon^{\prime } ) } { 4 } \text{tr}[\mathcal{e}(\rho^{(i ) } ) ] + \varepsilon\varepsilon^{\prime } \text{tr}[\mathcal{e}(\rho^{(i)})p^{jj } ] .\label{corr - noise}\end{aligned}\ ] ]  for the hamiltonian identification task , ( which is unital : , and trace - preserving : =1 ] , namely ,\text{tr}[\mathcal{e}(\rho^{(0)})p^{11}],\ldots , \text{tr}[\mathcal{e}(\rho^{(3)})p^{33}]\bigr)^t,\end{aligned}\ ] ] and the matrix , then ( [ noise ] ) can be written as the following linear matrix transformation ( see appendix [ app-1 ] ) : if we arrange the output elements as in table [ dcqd - tab ] , we will have where is the ( constant ) coefficient matrix , hence .we have provided a scheme for utilizing auxiliary quantum correlations to perform process estimation tasks with faulty quantum operations .we have demonstrated our approach via generalizing the ideal scheme of direct characterization of quantum dynamics ( dcqd ) where the required preparations and measurements could be noisy .it has been shown that when the systematic faulty operations are of the form of depolarizing channels , the overhead data processing is fairly simple . moreover, these examples have revealed that for the dcqd scheme , entanglement is secondary .this , in turn , broadens the range of applicability of our scheme to quantum systems with certain controllable classical correlations of their subsystems .therefore , our proposed method may have near - term applications to a variety of realistic quantum systems / devices with the current state - of - technology , such as trapped ions , liquid - state nmr , optical lattices , and entangled pairs of photons .we thank natural sciences and engineering research council of canada ( nserc ) , faculty of arts and sciences of harvard university , army research office ( aro ) [ project w911nf-07 - 1 - 0304 ] , mathematics of information technology and complex systems ( mitacs ) , pacific institute for mathemaical sciences ( pims ) , and the usc center for quantum information science and technology for funding .table [ dcqd - tab ] suggests that if , instead of the conventional bsms , we consider the expression \\\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(0 ) } ) ] \\\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(0 ) } ) ] \\ \mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(0 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(1)})]+\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(1 ) } ) ] \\\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(1)})]+\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(1 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(1)})]-\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(1 ) } ) ] \\\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(1)})]-\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(1 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(2)})]+\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(2 ) } ) ] \\\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(2)})]+\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(2 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(2)})]-\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(2 ) } ) ] \\\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(2)})]-\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(2 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(3)})]+\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(3 ) } ) ] \\\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(3)})]+\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(3 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(3)})]-\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(3 ) } ) ] \\\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(3)})]-\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(3)})]\end{smallmatrix } \right ) = \left(\begin{smallmatrix } 1 &  &  &  &  &  &  &  &  &  &  &  &  &  &  &  \\ & 1 &  &  &  &  &  &  &  &  &  &  &  &  &  &  \\ &  & 1 &  &  &  &  &  &  &  &  &  &  &  &  &  \\ &  &  & 1 &  &  &  &  &  &  &  &  &  &  &  &  \\ &  &  &  & 1 &  &  & 1 &  &  &  &  &  &  &  &  \\ &  &  &  &  & 1 & 1 &  &  &  &  &  &  &  &  &  \\ &  &  &  & 1 &  &  & -1 &  &  &  &  &  &  &  &  \\ &  &  &  &  & 1 & -1 &  &  &  &  &  &  &  &  &  \\ &  &  &  &  &  &  &  & 1 & 1 &  &  &  &  &  &  \\ &  &  &  &  &  &  &  &  &  & 1 & 1 &  &  &  &  \\ &  &  &  &  &  &  &  & 1 & -1 &  &  &  &  &  &  \\ &  &  &  &  &  &  &  &  &  & -1 & 1 &  &  &  &  \\ &  &  &  &  &  &  &  &  &  &  &  & 1 &  & 1 &  \\ &  &  &  &  &  &  &  &  &  &  &  &  & 1 &  & 1 \\ &  &  &  &  &  &  &  &  &  &  &  & 1 &  & -1 &  \\ &  &  &  &  &  &  &  &  &  &  &  &  & -1 &  & 1\end{smallmatrix}\right)\left(\begin{smallmatrix } \mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(0 ) } ) ] \\\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(0 ) } ) ] \\ \mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(0 ) } ) ] \\ \mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(0 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho^{(1 ) } ) ] \\ \mathrm{tr}[p^{11}\mathcal{e}(\rho^{(1 ) } ) ] \\ \mathrm{tr}[p^{22}\mathcal{e}(\rho^{(1 ) } ) ] \\ \mathrm{tr}[p^{33}\mathcal{e}(\rho^{(1 ) } ) ] \\ \mathrm{tr}[p^{00}\mathcal{e}(\rho^{(2 ) } ) ] \\\mathrm{tr}[p^{11}\mathcal{e}(\rho^{(2 ) } ) ] \\ \mathrm{tr}[p^{22}\mathcal{e}(\rho^{(2 ) } ) ] \\ \mathrm{tr}[p^{33}\mathcal{e}(\rho^{(2 ) } ) ] \\ \mathrm{tr}[p^{00}\mathcal{e}(\rho^{(3 ) } ) ] \\ \mathrm{tr}[p^{11}\mathcal{e}(\rho^{(3 ) } ) ] \\ \mathrm{tr}[p^{22}\mathcal{e}(\rho^{(3 ) } ) ] \\\mathrm{tr}[p^{33}\mathcal{e}(\rho^{(3)})]\end{smallmatrix}\right ) , \label{parametrization}\end{aligned}\ ] ] the results of the measurements can be related to the elements in a more straightforward fashion . here, the constant coefficient matrix ( ) relates the results of the new bsms ( ) to those of the conventional bsms ( ) .vs and . here , the coefficient matrix relates the experimental outcomes to the unknown elements of the superoperator .the input states for the standard dcqd , as defined in table [ dcqd - tab ] , e.g. , , are parameterized as and ( , ) .the optimal input states are associated to those parameters for which has its maximal value 1 , leading to a minimal statistical error.,width=207,height=151 ]  here , we find the optimal input states for the ideal dcqd . the idea is to choose the input states such that the ( linear ) inversion on the experimental data ( to read out matrix elements ) can be performed reliably .that is , the goal should be to make the coefficient matrix as far from singular matrices as possible . maximizing the determinant of this matrix is a sufficient condition to guarantee its reliable invertibility , hence in turn , minimal error propagation .the data obtained from the measurements ( bsms ) are $ ] , where corresponds to the first column of table [ dcqd - tab ] , respectively , for .we parameterize the input states by and ( , ) . using relation ( [ parametrization ] ), one can express eq .( [ vector - eq ] ) , for the standard dcqd , as the following : \\\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(1 ) } ) ] \\\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(1 ) } ) ] \\ \mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(1 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(2)})]+\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(2 ) } ) ] \\\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(2)})]+\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(2 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(2)})]-\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(2 ) } ) ] \\\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(2)})]-\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(2 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(3)})]+\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(3 ) } ) ] \\\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(3)})]+\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(3 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(3)})]-\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(3 ) } ) ] \\\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(3)})]-\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(3 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(4)})]+\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(4 ) } ) ] \\\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(4)})]+\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(4 ) } ) ] \\\mathrm{tr}[p^{00}\mathcal{e}(\rho ^{(4)})]-\mathrm{tr}[p^{22}\mathcal{e}(\rho ^{(4 ) } ) ] \\\mathrm{tr}[p^{33}\mathcal{e}(\rho ^{(4)})]-\mathrm{tr}[p^{11}\mathcal{e}(\rho ^{(4)})]\end{smallmatrix } \right ) = \left ( \begin{smallmatrix } 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \\ 1 & 0 & 0 & x & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & x & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 & 0 & 1 & -i x & 0 & 0 & i x & 1 & 0 & 0 & 0 & 0 & 0 \\ z & 0 & 0 & i y & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & -i y & 0 & 0 & -z \\ 0 & 0 & 0 & 0 & 0 & z & y & 0 & 0 & y & -z & 0 & 0 & 0 & 0 & 0 \\ 1 & x & 0 & 0 & x & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & -i x & 0 & 0 & i x & 1 \\ z & i y & 0 & 0 & -i y & -z & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & z & y & 0 & 0 & y & -z \\ 0 & 0 & 0 & 0 & 0 & 1 & 0 & -i x & 0 & 0 & 0 & 0 & 0 & i x & 0 & 1 \\ 1 & 0 & -x & 0 & 0 & 0 & 0 & 0 & -x & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & -z & 0 & -y & 0 & 0 & 0 & 0 & 0 & -y & 0 & z \\ -z & 0 & i y & 0 & 0 & 0 & 0 & 0 & -i y & 0 & z & 0 & 0 & 0 & 0 & 0\end{smallmatrix } \right ) \left(\begin{smallmatrix } \chi _ { 00 }\\ \chi _ { 01 } \\\chi _ { 02 } \\\chi _ { 03 } \\\chi _ { 10 } \\\chi _ { 11 } \\\chi _ { 12 } \\\chi _ { 13 } \\\chi _ { 20 }\\ \chi _ { 21 } \\\chi _ { 22 } \\\chi _ { 23 } \\\chi _ { 30 } \\\chi _ { 31 } \\\chi _ { 32 } \\\chi _ { 33}\end{smallmatrix}\right),\end{aligned}\ ] ] where in the coefficient matrix we have , , and .the determinant of this matrix is obtained as which attains its maximum value at ,  fig .[ maxima ] .therefore , the optimal input states for the standard dcqd are as in table [ dcqd - tab ] in which and are either of the pairs calculated from the above maximal set of and .a simple calculation shows that the amount of entanglement ( exactly speaking , concurrence ) of the optimal non - maximally entangled input states is ( independent of ) .a. bendersky , f. pastawski , and j. p. paz , phys .* 100 * , 190403 ( 2008 ) ; j. emerson , m. silva , o. moussa , c. ryan , m. laforest , j. baugh , d. g. cory , and r. laflamme , science * 317 * , 1893 ( 2007 ) .j. t. barreiro , n. k. langford , n. a. peters , and p. g. kwiat , phys . rev .* 95 * , 260501 ( 2005 ) ; p. kok , w. j. munro , k. nemoto , t. c. ralph , j. p. dowling , and g. j. milburn , rev .* 79 * , 135 ( 2007 ) .briegel , w. dr , j. i. cirac , and p. zoller , phys .lett . * 81 * , 5932 ( 1998 ) ; w. dr , h. -j .briegel , j. i. cirac , and p. zoller , phys .a * 59 * , 169 ( 1999 ) ; b. m. terhal , i. l. chuang , d. p. divincenzo , m. grassl , and j. a. smolin , phys .a * 60 * , 881 ( 1998 ) .

Abstract: performance of quantum process estimation is naturally limited to fundamental , random , and systematic imperfections in preparations and measurements . these imperfections may lead to considerable errors in the process reconstruction due to the fact that standard data analysis techniques presume ideal devices . here , by utilizing generic auxiliary quantum or classical correlations , we provide a framework for estimation of quantum dynamics via a single measurement apparatus . by construction , this approach can be applied to quantum tomography schemes with calibrated faulty state generators and analyzers . specifically , we present a generalization of direct characterization of quantum dynamics " [ m. mohseni and d. a. lidar , phys . rev . lett . * 97 * , 170501 ( 2006 ) ] with an imperfect bell - state analyzer . we demonstrate that , for several physically relevant noisy preparations and measurements , only classical correlations and small data processing overhead are sufficient to accomplish the full system identification . furthermore , we provide the optimal input states for which the error amplification due to inversion on the measurement data is minimal .