<commit_before>= SCAMP update

. The problem: distributed agreement
.. Solution: Paxos/Raft protocols
... What it is: Leader election, full cluster membership knowledge required, transactional cluster agreement
... CAP Rating: CA-
... Scalability comments: voting rounds require en masse communication from the leader to all followers (larger cluster = slower voting rounds and increasing liklihood of partitioning)
... When to use Paxos/Raft: if you need Consistency and Availability guarantees and are willing to tolerate downtime in the case of a Partition (network interruption) within the cluster
.. Solution: Gossip protocols
... What it is: the implementation of a metaphor
... CAP Rating: -AP
... Scalability comments: gossip protocols rely on redundant communication, 
. Naive implementation (November, 2015)
.. used TCP for comm, every node had membership list of entire cluster

> He was a scamp; he is a hero!
> - Les Misérables (Signet, p587)



SCAMP = SCAlable Membership Protocol

== Usage

`lein test`

== TODO

. Implement HyParView
. Evaluate results for clumping: https://arxiv.org/pdf/0906.0612.pdf

<commit_msg>[scamp] iterate on TALK to check rendering on github<commit_after>= SCAMP update

. The problem: distributed agreement
.. Solution: Paxos/Raft protocols
... What it is: Leader election, full cluster membership knowledge required, transactional cluster agreement
... CAP Rating: CA-
.... C = Consistency (Every read receives the most recent write or an error --wikipedia)
.... A = Availability (Every request receives a (non-error) response – without guarantee that it contains the most recent write --wikipedia)
.... P = Partition tolerance (The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes --wikipedia)
... Scalability comments: voting rounds require en masse communication from the leader to all followers (larger cluster = slower voting rounds and increasing liklihood of partitioning)
... When to use Paxos/Raft: if you need Consistency and Availability guarantees and are willing to tolerate downtime in the case of a Partition (network interruption) within the cluster
.. Solution: Gossip protocols
... What it is: the implementation of a metaphor
... CAP Rating: -AP
... Scalability comments: gossip protocols rely on redundant communication, but hopefully only among smaller sub-clusters of nodes
. Naive implementation (November, 2015)
.. used TCP for comm, every node had membership list of entire cluster, messages were passed from each node to all of its downstream nodes
.. Doing this exposed me quickly to the hard parts of designing a good gossip protocol
.. BIG PROBLEM: if every node needs to know the membership of the entire cluster, then we already need to solve the distributed agreement problem that Paxos and Raft address.
.. BIG QUESTION: How to maintain cluster membership in a way that yields any of the purported benefits of gossip?
.. Additional problem: scaling would be horrible, we have a mesh network requirement: _n_ nodes require _n^2^_ edges
. SCAMP = SCAlable Membership Protocol
.. Defined in \"Peer-to-Peer Membership Management for Gossip-Based Protocols\": https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/ieee_tocs.pdf
.. In SCAMP, each node 

== TODO

. Implement HyParView
. Evaluate results for clumping: https://arxiv.org/pdf/0906.0612.pdf


{'fraction_non_alphanumeric': 0.07082235879426398, 'fraction_numerical': 0.009950248756218905, 'mean_word_length': 4.226299694189603, 'pattern_counts': {'":': 1, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2104929', 'n_tokens_mistral': 902, 'n_tokens_neox': 821, 'n_words': 448}