<commit_before># Cancer Gene Trust Daemon

Stores steward submissions in a distributed, replicated and decentralized
content addressable database.  Provides an HTML interface and RESTful API to
add, list and authenticate submissions as well as the peering relationship
between stewards.

Submissions consist of a JSON manifest with a list of fields and files. Fields
typically include de-identified clinical data (i.e. tumor type).  Files
typically consist of somatic variant vcf files.  Manifest's and files are
stored and referenced by the multihash (https://github.com/jbenet/multihash) of
their content.

Each steward has a public private key pair which is used to authenticate their
submissions. A steward's address is the multihash of their public key.

Eash steward has a top level JSON index file containing it's dns domain, list
of submissions by multihash and list of peers by address. A steward's address
resolves to the multihash of the latest version of its index.

Updates to a steward's index file are signed using their private key.  This
provides authentication and authorization for its contents as well as any other
content referenced via multihash within it including all submissions.

The current underlying implementation leverages ipfs (http://ipfs.io) for
storage and replication and ipns for address resolution and public/private key
operations.  The server is implemented using python and flask
(http://flask.pocoo.org/)

# Running a Production Instance

Note: The only dependencies for the following is make and docker.

Start the ipfs server and store in data/ and generate a default
configuration and public/private key pair in data/config:

    make ipfs 

Reset the steward's index to no submissions, no peers, and a domain of
lorem.edu:

    DOMAIN=lorem.edu make reset<commit_msg>Update README to GA4GH language and link<commit_after># Cancer Gene Trust Daemon
The Cancer Gene Trust
(https://genomicsandhealth.org/work-products-demonstration-projects/cancer-gene-trust)
is a global online network for sharing somatic genomic and clinical data from
around the world, making the data accessible for research use in real time.

The Cancer Gene Trust Daemon (cgtd) stores steward submissions in a distributed,
replicated and decentralized content addressable database.  It provides an basic
HTML interface and RESTful API to add, list and authenticate submissions as well
as the peering relationship between stewards.

Submissions consist of a JSON manifest with a list of fields and files. Fields
typically include de-identified clinical data (i.e. tumor type).  Files
typically consist of somatic variant vcf files and gene expression tsv file.
Manifest's and files are stored and referenced by the multihash
(https://github.com/jbenet/multihash) of their content.

Eash steward has a top level JSON index file containing it's dns domain, list of
submissions by multihash and list of peers by address.  Each steward has a
public private key pair which is used to authenticate their submissions. A
steward's address is the multihash of their public key.  A steward's address
resolves to the multihash of the latest version of its index.  Updates to a
steward's index file are signed using their private key.  This provides
authentication and authorization for its contents as well as any other content
referenced via multihash within it including all submissions.

The current underlying implementation leverages ipfs (http://ipfs.io) for
storage and replication and ipns for address resolution and public/private key
operations.  The server is implemented using python and flask
(http://flask.pocoo.org/)

# Running a Production Instance

Note: The only dependencies for the following is make and docker.

Start the ipfs server and store in ./data/ and generate a default
configuration and public/private key pair in ./data/config:

    make ipfs 

Reset the steward's index to no submissions, no peers, set a domain:

    DOMAIN=lorem.edu make reset
{'fraction_non_alphanumeric': 0.038683747802059784, 'fraction_numerical': 0.00025119316754584274, 'mean_word_length': 4.616361071932299, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '152828', 'n_tokens_mistral': 1040, 'n_tokens_neox': 943, 'n_words': 569}