<commit_before># K8s deployment notes for SMC

## What to do if things have gone to hell

See the section below about how to setup a machine to have the kubectl command.  Most importantly, if things go totally to hell, one option is to delete the entire k8s cluster and recreate it from scratch, which takes about 15 minutes.

```
alias c=./control.py
alias k=kubectl

# delete the cluster (5 min)
cd cluster
c delete-cluster

# if possible, you could now upgraded kubernetes by changing what tarball is in ~/kubernetes

# create the cluster (5 min)
c create-cluster --node-disk-size=60 --min-nodes=10 --max-nodes=12 --non-preemptible
```

Next, configure the cluster and start everything running.
Here's how to setup the test namespace; doing the prod one is
similar -- just allocate more resources (via -r):

```
# create the namespace
cd ~/smc/src/k8s/
c cluster namespace test

# start haproxy
cd haproxy/<commit_msg>k8s: additional note about cluster creation<commit_after># K8s deployment notes for SMC

## What to do if things have gone to hell

See the section below about how to setup a machine to have the kubectl command.  Most importantly, if things go totally to hell, one option is to delete the entire k8s cluster and recreate it from scratch, which takes about 15 minutes.

```
alias c=./control.py
alias k=kubectl

# delete the cluster (5 min)
cd cluster
c delete-cluster

# if possible, you could now upgraded kubernetes by changing what tarball is in ~/kubernetes

# create the cluster (5 min)
c create-cluster --node-disk-size=60 --min-nodes=1 --max-nodes=1 --non-preemptible
```

Immediately, once the cluster is running, add more nodes via the web UI or `c resize --size` or `c autoscale...`.   We recently hit  race condition in which during the initial cluster creation multiple nodes had the same Routes assigned (so `sudo ifconfig cbr0|grep inet` was repeated on multiple nodes).  This led to disaster.

Next, configure the cluster and start everything running.
Here's how to setup the test namespace; doing the prod one is
similar -- just allocate more resources (via -r):

```
# create the namespace
cd ~/smc/src/k8s/
c cluster namespace test

# start haproxy
cd haproxy/
{'fraction_non_alphanumeric': 0.07366984993178717, 'fraction_numerical': 0.01182355616189177, 'mean_word_length': 4.0, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4073425', 'n_tokens_mistral': 650, 'n_tokens_neox': 612, 'n_words': 327}