The performance of cluster computing depends on how concurrent jobs share multiple data center resource types like CPU, RAM and disk storage. Recent research has discussed efficiency and fairness requirements and identified a number of desirable scheduling objectives including so-called dominant resource fairness (DRF). We argue here that proportional fairness (PF), long recognized as a desirable objective in sharing network bandwidth between ongoing flows, is preferable to DRF. The superiority of PF is manifest under the realistic modelling assumption that the population of jobs in progress is a stochastic process. In random traffic the strategy-proof property of DRF proves unimportant while PF is shown by analysis and simulation to offer a significantly better efficiency-fairness tradeoff.
There is an abundant literature on sharing network bandwidth. PF and, more generally, the idea that bandwidth sharing between a fixed set of flows should maximize a sum of per-flow utilities were notions introduced by Kelly al in 1998 @cite_11 . In that work, and subsequent contributions by many authors on network utility maximization, the population of flows in progress is assumed fixed. We believe this to be inappropriate, like the assumption that the population of jobs is fixed when evaluating cluster sharing. It does not make much sense to measure the utility of a flow as a function of its rate (e.g., the log function for PF) on recognizing that the population of flows in progress, and consequently the rate of any ongoing flow, changes rapidly and significantly throughout the lifetime of that flow.
@cite_11
2159715570
This paper analyses the stability and fairness of two classes of rate control algorithm for communication networks. The algorithms provide natural generalisations to large-scale networks of simple additive increase multiplicative decrease schemes, and are shown to be stable about a system optimum characterised by a proportional fairness criterion. Stability is established by showing that, with an appropriate formulation of the overall optimisation problem, the network's implicit objective function provides a Lyapunov function for the dynamical system defined by the rate control algorithm. The network's optimisation problem may be cast in primal or dual form: this leads naturally to two classes of algorithm, which may be interpreted in terms of either congestion indication feedback signals or explicit rates based on shadow prices. Both classes of algorithm may be generalised to include routing control, and provide natural implementations of proportionally fair pricing.