<jupyter_start><jupyter_text>Define a schema and the keysThe schema has two roles. Firstly, it defines the format of the data, such as the type of each column, minimum and maximum lengths, and permitted values. Secondly, it contains all the hashing settings, from the hash size, through global salts, to the tokenisation settings of each field.The schema is designed to be portable between Clkhash and Anonlink. Two runs of Clkhash with the same schema and the same keys on the same data will produce the exact same hashes, even if they use different versions of Clkhash.The keys can be thought of as a secret salt. They must be shared between organisations that are hashing their data, but must not be given to the authority performing the linkage.<jupyter_code>SCHEMA = {
  'version': 1,
  'clkConfig': {
    'l': 1024,
    'k': 20,
    'hash': {
      'type': 'doubleHash'
    },
    'kdf': {
      'type': 'HKDF',
      'hash': 'SHA256',
      'salt': 'SCbL2zHNnmsckfzchsNkZY9XoHk96P/G5nUBrM7ybymlEFsMV6PAeDZCNp3rfNUPCtLDMOGQHG4pCQpfhiHCyA==',
      'info': 'c2NoZW1hX2V4YW1wbGU=',
      'keySize': 64
    }
  },
  'features': [
    {
      'identifier': 'NAME freetext',
      'format': {
        'type': 'string',
        'encoding': 'utf-8',
        'case': 'mixed',
        'minLength': 3
      },
      'hashing': {
        'ngram': 2,
        'weight': 0.5
      }
    },
    {
      'identifier': 'DOB YYYY/MM/DD',
      'format': {
        'type': 'string',
        'encoding': 'ascii',
        'description': 'Numbers separated by slashes, in the year, month, day order',
        'pattern': '\\d\\d\\d\\d/\\d\\d/\\d\\d'
      },
      'hashing': {
        'ngram': 1,
        'positional': True
      }
    },
    {
      'identifier': 'GENDER M or F',
      'format': {
        'type': 'enum',
        'values': ['M', 'F']
      },
      'hashing': {
        'ngram': 1,
        'weight': 2
      }
    }
  ]
}
SECRET_KEY = b'<KEY>'<jupyter_output><empty_output><jupyter_text>Make new project<jupyter_code>r = requests.post(
    PREFIX + '/projects/',
    params=dict(
        project_id='demo-data',
        secret_key=base64.b64encode(SECRET_KEY)),
    json=SCHEMA)

print('Status code:', r.status_code)<jupyter_output>Status code: 201<jupyter_text>See all projects<jupyter_code>r = requests.get(
    PREFIX + '/projects')

print('Status code:', r.status_code)
print('Data:', r.json())<jupyter_output>Status code: 200
Data: {'projects': ['demo-data']}<jupyter_text>Upload private data<jupyter_code>csv_data = (
    'NAME freetext,DOB YYYY/MM/DD,GENDER M or F\n'
    '<NAME>,1968/05/19,F\n'
    '<NAME>,1998/12/20,M\n')

r = requests.post(
    PREFIX + '/projects/{project_id}/pii/'.format(
        project_id='demo-data'),
    params=dict(
        header='true',
        validate=True),
    data=csv_data)

json = r.json()
clk_range_start = json['dataIds']['rangeStart']
clk_range_end = json['dataIds']['rangeEnd']

print('Status code:', r.status_code)
print('Data:', r.json())<jupyter_output>Status code: 202
Data: {'dataIds': {'rangeEnd': 2, 'rangeStart': 0}}<jupyter_text>Get clks<jupyter_code>r = requests.get(
    PREFIX + '/projects/{project_id}/clks/'.format(
        project_id='demo-data',
        index_range_start=clk_range_start,
        index_range_end=clk_range_end))

print('Status code:', r.status_code)
print('Data:', r.json())<jupyter_output>Status code: 200
Data: {'clks': [{'index': 0, 'status': 'queued', 'errMsg': None, 'hash': None}, {'index': 1, 'status': 'queued', 'errMsg': None, 'hash': None}], 'responseMetadata': {'nextCursor': None}}<jupyter_text>Upload lots of private data!<jupyter_code>from clkhash import randomnames
import io

buffer = io.StringIO("")
buffer.write('NAME freetext,DOB YYYY/MM/DD,GENDER M or F\n')

synthetic_data = randomnames.NameList(100000).names
for entity in synthetic_data:
    buffer.write(','.join(entity[1:]) + '\n')
buffer.seek(0)
csv_data = buffer.read()
r = requests.post(
    PREFIX + '/projects/{project_id}/pii/'.format(
        project_id='demo-data'),
    params=dict(
        header='true'),
    data=csv_data)
print(r.text)

json = r.json()
clk_range_start = json['dataIds']['rangeStart']
clk_range_end = json['dataIds']['rangeEnd']

print('Status code:', r.status_code)
print('Data:', r.json())<jupyter_output>{
  "dataIds": {
    "rangeEnd": 100002,
    "rangeStart": 2
  }
}

Status code: 202
Data: {'dataIds': {'rangeEnd': 100002, 'rangeStart': 2}}<jupyter_text>Check job status<jupyter_code>r = requests.get(
    PREFIX + '/projects/{project_id}/clks/status'.format(
        project_id='demo-data'))

print('Status code:', r.status_code)
print('Data:', r.json())<jupyter_output>Status code: 200
Data: {'clksStatus': [{'status': 'done', 'rangeStart': 0, 'rangeEnd': 2}, {'status': 'queued', 'rangeStart': 2, 'rangeEnd': 100002}]}<jupyter_text>Delete clks or abort jobs If a particular clk has already been computed, then we simply delete it from the database. Otherwise, we cancel the scheduled job and delete the private data.<jupyter_code>r = requests.delete(
    PREFIX + '/projects/{project_id}/clks/'.format(
        project_id='demo-data'),
    params=dict(
        index_range_start=clk_range_start,
        index_range_end=clk_range_end))

print('Status code:', r.status_code)<jupyter_output>Status code: 204<jupyter_text>Delete project This will also delete all clks associated with the project and abort all hashing jobs.<jupyter_code>r = requests.delete(
    PREFIX + '/projects/{project_id}'.format(
        project_id='demo-data'))

print('Status code:', r.status_code)<jupyter_output>Status code: 204
{'fraction_non_alphanumeric': 0.15774446823697358, 'fraction_numerical': 0.020164168451106353, 'mean_word_length': 3.7540288379983036, 'pattern_counts': {'":': 3, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '356252', 'license': 'Apache-2.0', 'n_tokens_mistral': 2071, 'n_tokens_neox': 1954, 'n_words': 466, 'path': 'docs/demo.ipynb', 'repo_name': 'hardbyte/anonlink-encoding-service', 'size': '5628'}