<commit_before>* Parameter Server

Parameter server is distributed machine learning framework. It  targets
cloud-computing situations where machines are possibly unreliable,
jobs may get preempted, data may be lost, and where network latency
and temporary workloads lead to a much more diverse performance
profile. In other words, we
target real cloud computing scenarios applicable to Google, Baidu,
Amazon, Microsoft, etc. rather than low utilization-rate, exclusive
use, high performance supercomputer clusters.

** Features
- *Ease of use*. The globally shared parameters are represented as
  (potentially sparse) *vectors and matrices*, which are more convenient
  data structures for machine learning applications than the widely
  used (key,value) store or tables.  High-performance and convenient
  multi-threaded linear algebra operations, such as vector-matrix
  multiplication between parameters and local training data, are
  provided to facilitate developing applications.

- *Efficiency*. Communication between nodes is
  *asynchronous*. Importantly, synchronization does not block
  computation. This framework allows the algorithm designer to
  balance algorithmic convergence rate and system efficiency, where
  the best trade-off depends on data, algorithm, and hardware.

- *Elastic Scalability*. New nodes can be added without restarting
  the running framework. This property is desirable, e.g.
  for streaming sketches or when deploying a parameter server as an
  online service that must remain available for a long time.

- *Fault Tolerance and Durability*. Conversely, node failure is inevitable,
  particularly at large scale using commodity servers.  We use an optimized data
  replication architecture that efficiently stores data on multiple server nodes
  to enable fast (in less than 1 second) recovery from node failure.

** Architecture

The parameter server architecture has two classes of
nodes: Each *server* node maintains a partition of the globally
shared parameters.  They communicate with each other to replicate and/or to
migrate parameters for reliability and scaling.  The *client*
nodes perform the bulk of the computation. Each client
typically stores locally a portion of the training data, computing
local statistics such as gradients.  Clients communicate only with the
server nodes, updating and retrieving the shared parameters.  Clients
may be added or removed.

[[./doc/img/arch2.png]]

** APIs

#+BEGIN_SRC C++
  Status Push(const Header& h);
  Status Pull(const Header& h);
#+END_SRC

#+BEGIN_SRC C++


#+END_SRC<commit_msg>editing readme<commit_after>* Parameter Server

Parameter server is distributed machine learning framework. It  targets
cloud-computing situations where machines are possibly unreliable,
jobs may get preempted, data may be lost, and where network latency
and temporary workloads lead to a much more diverse performance
profile. In other words, we
target real cloud computing scenarios applicable to Google, Baidu,
Amazon, Microsoft, etc. rather than low utilization-rate, exclusive
use, high performance supercomputer clusters.

** Features
- *Ease of use*. The globally shared parameters are represented as
  (potentially sparse) *vectors and matrices*, which are more convenient
  data structures for machine learning applications than the widely
  used (key,value) store or tables.  High-performance and convenient
  multi-threaded linear algebra operations, such as vector-matrix
  multiplication between parameters and local training data, are
  provided to facilitate developing applications.

- *Efficiency*. Communication between nodes is
  *asynchronous*. Importantly, synchronization does not block
  computation. This framework allows the algorithm designer to
  balance algorithmic convergence rate and system efficiency, where
  the best trade-off depends on data, algorithm, and hardware.

- *Elastic Scalability*. New nodes can be added without restarting
  the running framework. This property is desirable, e.g.
  for streaming sketches or when deploying a parameter server as an
  online service that must remain available for a long time.

- *Fault Tolerance and Durability*. Conversely, node failure is inevitable,
  particularly at large scale using commodity servers.  We use an optimized data
  replication architecture that efficiently stores data on multiple server nodes
  to enable fast (in less than 1 second) recovery from node failure.

** Architecture

The parameter server architecture has two classes of
nodes: Each *server* node maintains a partition of the globally
shared parameters.  They communicate with each other to replicate and/or to
migrate parameters for reliability and scaling.  The *client*
nodes perform the bulk of the computation. Each client
typically stores locally a portion of the training data, computing
local statistics such as gradients.  Clients communicate only with the
server nodes, updating and retrieving the shared parameters.  Clients
may be added or removed.

[[./doc/img/arch2.png]]

** APIs

Parameter server provides several globally shared data structures, for example,
(sparse) vectors and matrices. The class =Container= is the base class of all
these structures. It provides two simple functions to do communication. =Push=
sends out local modifications to other nodes, while =Pull= retrieves
modifications from the others. These two functions are non-blocking, if the
user specified consistency model is satisfied. All control information is set in
=Header=.

#+BEGIN_SRC C++
  Status Push(const Header& h);
  Status Pull(const Header& h);
#+END_SRC

A particular data structure may provides more convenient functions to avoid
explicitly construct =Header=. For example, =Vectors= could store several
sparse vectors (namely a thin sparse matrix).

#+BEGIN_SRC C++
  Status PushPull(KeyRange key_range     = KeyRange::All(),
                  VecList  push_vec_list = {0},
                  bool     push_delta    = kDelta,
                  VecList  pull_vec_list = {0},
                  bool     pull_delta    = kValue,
                  uid_t    dest          = kServer);
#+END_SRC

A typical usage is that, we first construct 2 vectors, the first one stores the
gradient, while the second one has the weight. Then each client write local
computed gradient into the first one, push these values into all server nodes,
and then retrieve the modification of weight from the server. Then we can write
it as


#+BEGIN_SRC C++
  Vectors<double> W("my_vec", p, 2);
  // write local gradient into W.vec(0)
  W.PushPull(KeyRange::All(), {0}, kValue, {1}, kDelta, kServer);
#+END_SRC

{'fraction_non_alphanumeric': 0.05095733453942409, 'fraction_numerical': 0.001658374792703151, 'mean_word_length': 4.389114541023558, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6973991', 'n_tokens_mistral': 1630, 'n_tokens_neox': 1535, 'n_words': 867}