1. A method for providing extensible centralized dynamic resource distribution, said method comprising: storing a plurality of partitions of a data set distributed across a cluster of computer nodes; designating one of said computer nodes to be a central distribution coordinator that collects information indicating a global state of said cluster and provides access to said global state; periodically analyzing the global state of the cluster by the distribution coordinator in order to determine whether changes should be made to distribution of said partitions among said computer nodes; generating, said distribution coordinator, a new distribution plan based on said changes to the distribution of said partitions, and providing access to said distribution plan to all of the cluster of computer nodes; and employing a distributed algorithm to independently determine by each node in the cluster how to perform individual partition transfers associated with said node in order to implement the new distribution plan.