<commit_before># TOP PRIORITY

- [ ] - [ ] upgrade to sage-6.x:  git fetch git://git.sagemath.org/sage.git u/ohanar/ipython-upgrade
      - [ ] ensure clawpack fully works in new sage version
      - [x] make storage user have more sudo powers
      - [ ] put latest smc_storage.py in /home/storage/
      - [ ] push out a new vm base.
      - [ ] update vm base in services file.

- [ ] add 512G storage qcow2 image to every compute machine in services file

- [ ] restart vm's due to controlling vm.py failing: 5,6,7, 10, 17-20

- [ ] create a new ZFS pool on each compute machine called "storage" with filesystems:
        storage/images
        storage/streams

- [ ] implement storage.coffee Storage class that does:
       - snapshot
       - replicate
       - mount
       - close
      using new system if /storage/streams/project_id exists.

- [ ] storage3: better schedule for applying streams, which does the right thing when forks happen, multiple start times, etc.  I.e., no matter what even when we don't delete streams.   Basically, we start with the stream with the newest end time, then work back until hitting a stream with 0 interval.   Make *that* the only thing returned by the "def streams" function.

- [ ] storage: function that goes through and deletes all (non-fork) streams that would not be applied (except forks).

- [ ] cassandra -- redo to use ZFS -- e.g., 5,6,7 will save 100's of GB's of space (and we could use it there).

        fdisk /dev/vdc
        zpool create cassandra /dev/vdc
        zfs set compression=lz4 cassandra
        zfs set atime=off cassandra             # see http://www.slideshare.net/planetcassandra/c-summit-2013-practice-makes-perfect-extreme-cassandra-optimization-by-albert-tobey
        zfs set logbias=throughput cassandra
        rsync -axvH /mnt/cassandra /cassandra/
        # rm stuff

      On startup, right now I have to manually do this:

         sudo zpool import -f cassandra

      I could just stick this in the /etc/rc.local.

    Net result for cassandra5 -- it takes up 2.2G on disk instead of 150G :-)

        150G    cassandra5-cassandra.img
        2.2G    cassandra5-cassandra-zfs.img

    And migrating to an encrypted SSD device later will be easy (zfs send/recv).  Plus I can snapshot all nodes for a backup-in-place.
    And I can also make offsites of all nodes via zfs send.


- [ ] add functionality to vm.py to *add* a device dynamically, so I can create new pools (if linux supports this).


# OTHER

# DONE<commit_msg>planning...<commit_after># TOP PRIORITY

- [ ] transition the 8 compute machines at Google from Debian to Ubuntu:
       - [ ] setup compute-base-disk and run doctests
       - [ ] worry about startup script and switch 10.3.4.4 first, since no projects are running there.




- [ ] implement storage.coffee Storage class that does:
       - snapshot
       - replicate
       - mount
       - close
      using new system if /storage/streams/project_id exists.

- [ ] storage: better schedule for applying streams, which does the right thing when forks happen, multiple start times, etc.  I.e., no matter what even when we don't delete streams.   Basically, we start with the stream with the newest end time, then work back until hitting a stream with 0 interval.   Make *that* the only thing returned by the "def streams" function.

- [ ] storage: trim function that goes through and deletes all (non-fork) streams that would not be applied (except forks)


# OTHER

- [ ] add functionality to vm.py to *add* a device dynamically, so I can create new pools (if linux supports this).

# DONE


- [x] cassandra -- redo to use ZFS -- e.g., 5,6,7 will save 100's of GB's of space (and we could use it there).

DONE: 1,2,5,6,7,4,3,10,11,12,13,14,15,
TODO: 16,17,18,19,20,21


n=7
import admin; reload(admin); cloud = admin.Services('conf/deploy_cloud/'); h = lambda x: cloud._hosts('cassandra%s'%n, x, sudo=True, timeout=3600)
cloud.restart('vm',hostname='cassandra%s'%n)  # may take doing it multiple times...
cloud.wait_until_up('10.1.%s.2'%n)

h('zpool create -f cassandra /dev/vdc')
h('zfs set compression=lz4 cassandra')
h('zfs set atime=off cassandra')
h('zfs set logbias=throughput cassandra')
h('rsync -axH /mnt/cassandra/ /cassandra/')
h('umount /mnt/cassandra')
h('zfs set mountpoint=/mnt/cassandra cassandra')
h('zfs snapshot cassandra@init')
h('zpool export cassandra')

--> Which is the same as log in and...

    # fdisk /dev/vdc
    zpool create -f cassandra /dev/vdc
    zfs set compression=lz4 cassandra
    zfs set atime=off cassandra
    zfs set logbias=throughput cassandra
    time sudo rsync -axH /mnt/cassandra/ /cassandra/
    umount /mnt/cassandra
    zfs set mountpoint=/mnt/cassandra cassandra
    zfs snapshot cassandra@init
    zpool export cassandra


--> remove the old image from services file, then

import admin; reload(admin); cloud = admin.Services('conf/deploy_cloud/'); h = lambda x: cloud._hosts('cassandra%s'%n, x, sudo=True, timeout=3600)

cloud.restart('vm',hostname='cassandra%s'%n)
cloud.wait_until_up('10.1.%s.2'%n)
h('zpool import -f cassandra')
cloud.start('cassandra',host='cassandra%s'%n,wait=True)

print cloud._hosts('cassandra%s'%n, 'cd ~/salvus/salvus; . salvus-env; nodetool status', wait=True)


---> Once it is working, move the cassandra-cassandra.img to TRASH.

---
    Net result for cassandra5 -- it takes up 2.2G on disk instead of 150G :-)

        150G    cassandra5-cassandra.img
        2.2G    cassandra5-cassandra-zfs.img

    And migrating to an encrypted SSD device later will be easy (zfs send/recv).  Plus I can snapshot all nodes for a backup-in-place.
    And I can also make offsites of all nodes via zfs send.

---

- [x] build sage-6.2.beta with ipython update:  git fetch git://git.sagemath.org/sage.git u/ohanar/ipython-upgrade
            use system-wide atlas so it's fast to build and portable.
      - [x] ensure clawpack fully works in new sage version
      - [x] more useful crontab monitoring pools:
            */5 * * * * sudo zpool list projects > /home/storage/.zpool.list.projects && cp /home/storage/.zpool.list.projects /home/storage/zpool.list.projects
            */5 * * * * sudo zpool list storage > /home/storage/.zpool.list.storage&& cp /home/storage/.zpool.list.storage /home/storage/zpool.list.storage
            */5 * * * * sudo zpool list cassandra > /home/storage/.zpool.list.cassandra && cp /home/storage/.zpool.list.cassandra /home/storage/zpool.list.cassandra
      - [x] make storage user have more sudo powers:
            # to manage the zfs pool for other projects
            storage ALL=(ALL) NOPASSWD: /sbin/zfs *
            storage ALL=(ALL) NOPASSWD: /sbin/zpool *
            storage ALL=(ALL) NOPASSWD: /usr/bin/pkill *
            # to create users for projects
            storage ALL=(ALL) NOPASSWD: /usr/local/bin/create_project_user.py *
      - [x] automate put latest smc_storage.py in /home/storage/



   - [x] create a new ZFS pool on each compute machine called "storage" with filesystems:
        storage/images
        storage/streams

        zpool create storage -f /dev/sdc
        zfs set dedup=on storage
        zfs set compression=lz4 storage
        zfs create storage/images
        zfs create storage/streams
        chown storage. /storage/streams
        chown storage. /storage/images
        chmod og-rwx /storage/streams
        chmod og-rwx /storage/images
        zpool export storage

    (x) done at UW

- [x] create 425GB/each storage devices on all of 3.1.x.4 for x=1,2,..,8, attach and configure (with care!)
      (only 8 remains)

- [x] export and delete storage on 3.1.x.4 for x=1,2,3.
- [x] create zpool image ubuntu-base-salvus-qcow, and copy over data.


{'fraction_non_alphanumeric': 0.10187702265372169, 'fraction_numerical': 0.017864077669902913, 'mean_word_length': 2.807787087235091, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2047028', 'n_tokens_mistral': 2622, 'n_tokens_neox': 2431, 'n_words': 877}