<commit_before>
 - [x] 99 projects have >3 replicas in practice, though not (usually) in db -- these appear "lost" to users in some cases.
   For each:
       - copy all files into their current location
       - move files for projects not specifically listed in bup_last_save to a TO_DELETE location

  Plan:

    phase 1:
     - sync from *all* actual replicas to master
     - save from master to *all* replicas
    phase 2:
     - add all replicas to bup_locations in database.

  Update everything.

  Later -- reduce number of replicas in some cases.

 - [ ] after repairing cassandra data reduce the write consistency level when making new projects... maybe. (?)

 - [ ] write a post explaining what is new and awesome, and what the architecture is.

key points:

   - direct tcp connections instead of ssh tunnels (limits of sshd)
   - fix uid issue
   - project states
   - move/rename/copy file buttons
   - faster file listing
   - live files, with zfs snapshots every few minutes, which are not consistent across dc's, and will vanish if a project were moved in a dc
       - dedup'd across projects on a given host
       - compressed
       - quota
   - set bup repo of snapshots that are consistent across dc's -- highly deduped and compressed; easy to sync around; git-based so branches are possible; dynamic fuse mounting
   - /scratch
   - sync to other dc's is done via rsync
   - daemon that runs on compute vm's and starts/stops projects, sets quotas, replicates, etc., but knows nothing global (e.g., no database).


 - [ ] (in progress on cloud3) create a full offsite-able backup of all bup repos of projects in dc1, and also the database nodes in dc1.

 - [ ] run through and do "bup ls master" on every repo in an offline archive, and investigate/fix ones that don't work, if any.

 - [ ] i observe two bup saves happening at once -- that should be *impossible*, and could result in corruption.
 root@compute8dc2:/bup/bups/4cff8798-41d0-4d9b-b516-ba106ba89c57/objects# ps ax |grep 4cff8798-41d0-4d9b-b516-ba106ba89c57|grep bup
 8792 ?        S      0:00 sudo /usr/local/bin/bup_storage.py save --targets=10.1.17.5,10.1.1.5 4cff8798-41d0-4d9b-b516-ba106ba89c57
 8793 ?        S      0:00 python /usr/local/bin/bup_storage.py save --targets=10.1.17.5,10.1.1.5 4cff8798-41d0-4d9b-b516-ba106ba89c57
10309 ?        S      2:27 bup-save --strip -n master -d 1397161051 /projects/4cff8798-41d0-4d9b-b516-ba106ba89c57
11668 ?        Ss     0:20 bup-fuse -o --uid 632382271 --gid 632382271 /projects/4cff8798-41d0-4d9b-b516-ba106ba89c57/.snapshots
12748 ?        S      0:00 sudo /usr/local/bin/bup_storage.py save --targets=10.1.17.5,10.1.1.5 4cff8798-41d0-4d9b-b516-ba106ba89c57
12749 ?        S      0:00 python /usr/local/bin/bup_storage.py save --targets=10.1.17.5,10.1.1.5 4cff8798-41d0-4d9b-b516-ba106ba89c57
13104 ?        S      2:16 bup-save --strip -n master -d 1397161256 /projects/4cff8798-41d0-4d9b-b516-ba106ba89c57
ALSO, when a file vanishes between index and save, we get an error, but still there is a new commit -- we should always remount the snapshots.

 - [ ] fix gce boot -- right now it boots up but doesn't mount the zfs pool -- or rather it starts getting rsync'd too before finishing the mount (?).  This is very bad.  Maybe don't go on VPN until /projects is mounted to avoid potential data loss.

 - [ ] setup so that cqlsh doesnt' need env variable but uses .cqlshrc

 - [ ] test ui changes on other browsers.

 - [ ] hourly or rolling snapshots of new *compute vm's* filesystems:
         - https://github.com/zfsnap/zfsnap/tree/legacy

 - [ ] test/fix ui changes on other browsers.

 - [ ] add a bigger (?) timeout between vm stop/start (?)

 - [ ] function to "truly" move a project within a given data center

 - [ ] write clean() -- for each project on a given host that hasn't been used in the last n days, delete .sagemathcloud, etc., directories

 - [ ] install something randy needs:  I think this will be possible in the release planned for this summer, but for now it would be nice to use Jake's mpld3 package, which doesn't seem to be installed.  I tried downloading and following the instructions at   https://github.com/jakevdp/mpld3 but didn't have permissions.  Is this something you could install globally?

  - [ ] make this standard  -- https://github.com/biocore/scikit-bio   -- see https://mail.google.com/mail/u/0/?shva=1#inbox/1454ce211132e2bf



 - [ ] MAYBE -- or maybe not -- change bup_storage to never delete account: it's very useful for linking projects and sharing files to have account available at all times.  will make, e.g., persistent sshfs possible; make sure .ssh is not ssh excluded from rsync

- [ ] have stable ipv6 project ip addresses be would be a huge *win*.  LXC would make that possible.

- [ ] deal with the exception around this - codecs.open(self.save_log,'a',"utf-8-sig").write(json.dumps(r)+'\n')

- [ ] go through and chown/sync every project systematically; evidently I didn't in the current migration, so I'll just put a chown in the start script for now -- this slows things down, but is temporary.

- [ ] update quota information using du script and re-enable quotas

- [ ] write code that cleans up /bup/projects fs by removing .sagemathcloud directories, etc., of projects not used for a while

- [ ] make it so move is never automatic but prompted?

- [ ] automated rolloing snapshots of bup/projects

- [ ] add bup quota as a standard part of settings, and refuse to make further snapshots if bup usage exceeds 3 times user disk quota.  This will avoid a horrible edge case.   Critical that this produces an error that the user learns about.  This will happen for some users.  Alternatively, I could periodically rebuild those bup repos with many snapshots deleted - that would be much nicer and is totally do-able.

- [ ] script to cleanup bup repos, e.g., delete tmp files, maybe recompact, etc.

- [ ] manual project move system -- bring it back...


======


- [x] read consistency issue: if db say bup_last_save empty, but it isn't, we destroy everything and completely loose project (?!)

 - project_id=48099d72-a9f0-4090-a4a5-a5681b612222
  bup_last_save *should* be these,
 - 10.1.4.5 10.1.15.5 10.3.7.4
  but suddenly became these:  10.1.5.5, 10.1.21.5, 10.3.4.4

  Only explanation, after reading the code is that I better run nodetool repair.



 - [x] change status to not bail when value is False; otherwise, broken sage ==> can't access project!


 - [x] gce booting configuration:

        fdisk /dev/sdd
        mkfs.ext4 /dev/sdd1 && mkdir /mnt/conf2 && mount /dev/sdd1 /mnt/conf2 && rsync -axvH /mnt/conf/ /mnt/conf2/ && ls /mnt/conf2 && sync


 - [x] all proxy server stuff is broken, so ipython, latex, file download, etc.

    - try again using the testing server

 - [x] gce nodes -- need to fix /bup/projects2 issue,


 - [x] include some more anti-bitcoin mining measures.
<commit_msg>planning<commit_after>- [x] frontend: don't include "a" in rsync option for recovering/copying files -- just use r

- [ ] get GCE restart to actually work

- [ ] project folder connections (?)

       zfs set sharenfs=on bup/projects
       sudo zfs set sharenfs='rw=@10.1.1.0/16',no_subtree_check,async,no_root_squash bup/projects
       apt-get install  nfs-kernel-server

   Seems very flaky, and only mildly faster or maybe even *SLOWER* than sshfs, at least over our network.

   This seems very nice... and works fantastically!

      sshfs -o cache_timeout=10 -o kernel_cache -o auto_cache -o uid=1959631043 -o gid=1959631043 -o allow_other -o default_permissions 10.1.1.5:/projects/test/sage compute1


      cd /projects/3702601d-9fbc-4e4e-b7ab-c10a79e34d3b; mkdir -p projects/edf7b34d-8ef9-49ad-b83f-8fa4cde53380; sshfs -o cache_timeout=10 -o kernel_cache -o auto_cache -o uid=1959631043 -o gid=1959631043 -o allow_other -o default_permissions 10.1.3.5:/projects/edf7b34d-8ef9-49ad-b83f-8fa4cde53380 projects/edf7b34d-8ef9-49ad-b83f-8fa4cde53380

      fusermount -u edf7b34d-8ef9-49ad-b83f-8fa4cde53380

    # mounting student projects

    coffee> x={};require('bup_server').global_client(cb:(e,c)->x.c=c)
    coffee> p=x.c.get_project('cc96c0e6-8daf-467d-b8d2-354f9c5144a5')
    coffee> p.get_location_pref(console.log)
    undefined 'b9cd6c52-059d-44e1-ace0-be0a26568713'
    coffee> x.c.servers.by_id['b9cd6c52-059d-44e1-ace0-be0a26568713'].host
    '10.1.15.5'

    # then at the shell

    export project_id=cc96c0e6-8daf-467d-b8d2-354f9c5144a5; export host=10.1.15.5; export uid=447893796

    mkdir -p students/$project_id && sshfs -o cache_timeout=10 -o kernel_cache -o auto_cache -o uid=$uid -o gid=$uid -o allow_other -o default_permissions $host:/projects/$project_id students/$project_id; chown $uid:$uid students/$project_id

- [ ] rewrite `bup_server` to use a local sqlite database; then state is preserved upon restart/crash/reboot/etc.

- [ ] "pip install --user pymc": https://mail.google.com/mail/u/0/?shva=1#search/Carlos+Rodriguez/14541f56e95e0756

- [ ] code to "rebuild/repair a node" -- hmm; because of this maybe need some way to know when a project was last sync'd based on filesystem

- [ ] --delete and --update together with rsync - what happens? -- we might as well make the replication actually a merge of newest files!

 - [ ] after repairing cassandra data reduce the write consistency level when making new projects... maybe. (?)

 - [ ] I'm also trying to install pymc (python montecarlo) but when I run it, it complains that the ver of numpy is too old... any tips on how to upgrade numpy or how to make pymc work?....; github ticket #2

 - [ ] put project creation date in project

 - [ ] write a post explaining what is new and awesome, and what the architecture is.


key points:

   - direct tcp connections instead of ssh tunnels (limits of sshd)
   - fix uid issue
   - project states
   - move/rename/copy file buttons
   - faster file listing
   - live files, with zfs snapshots every few minutes, which are not consistent across dc's, and will vanish if a project were moved in a dc
       - dedup'd across projects on a given host
       - compressed
       - quota
   - set bup repo of snapshots that are consistent across dc's -- highly deduped and compressed; easy to sync around; git-based so branches are possible; dynamic fuse mounting
   - /scratch
   - sync to other dc's is done via rsync
   - daemon that runs on compute vm's and starts/stops projects, sets quotas, replicates, etc., but knows nothing global (e.g., no database).


 - [ ] (in progress on cloud3) create a full offsite-able backup of all bup repos of projects in dc1, and also the database nodes in dc1.

 - [ ] run through and do "bup ls master" on every repo in an offline archive, and investigate/fix ones that don't work, if any.

 - [ ] i observe two bup saves happening at once -- that should be *impossible*, and could result in corruption.
 root@compute8dc2:/bup/bups/4cff8798-41d0-4d9b-b516-ba106ba89c57/objects# ps ax |grep 4cff8798-41d0-4d9b-b516-ba106ba89c57|grep bup
 8792 ?        S      0:00 sudo /usr/local/bin/bup_storage.py save --targets=10.1.17.5,10.1.1.5 4cff8798-41d0-4d9b-b516-ba106ba89c57
 8793 ?        S      0:00 python /usr/local/bin/bup_storage.py save --targets=10.1.17.5,10.1.1.5 4cff8798-41d0-4d9b-b516-ba106ba89c57
10309 ?        S      2:27 bup-save --strip -n master -d 1397161051 /projects/4cff8798-41d0-4d9b-b516-ba106ba89c57
11668 ?        Ss     0:20 bup-fuse -o --uid 632382271 --gid 632382271 /projects/4cff8798-41d0-4d9b-b516-ba106ba89c57/.snapshots
12748 ?        S      0:00 sudo /usr/local/bin/bup_storage.py save --targets=10.1.17.5,10.1.1.5 4cff8798-41d0-4d9b-b516-ba106ba89c57
12749 ?        S      0:00 python /usr/local/bin/bup_storage.py save --targets=10.1.17.5,10.1.1.5 4cff8798-41d0-4d9b-b516-ba106ba89c57
13104 ?        S      2:16 bup-save --strip -n master -d 1397161256 /projects/4cff8798-41d0-4d9b-b516-ba106ba89c57
ALSO, when a file vanishes between index and save, we get an error, but still there is a new commit -- we should always remount the snapshots.

 - [ ] fix gce boot -- right now it boots up but doesn't mount the zfs pool -- or rather it starts getting rsync'd too before finishing the mount (?).  This is very bad.  Maybe don't go on VPN until /projects is mounted to avoid potential data loss.

 - [ ] setup so that cqlsh doesnt' need env variable but uses .cqlshrc

 - [ ] test ui changes on other browsers.

 - [ ] hourly or rolling snapshots of new *compute vm's* filesystems:
         - https://github.com/zfsnap/zfsnap/tree/legacy

 - [ ] test/fix ui changes on other browsers.

 - [ ] add a bigger (?) timeout between vm stop/start (?)

 - [ ] function to "truly" move a project within a given data center

 - [ ] write clean() -- for each project on a given host that hasn't been used in the last n days, delete .sagemathcloud, etc., directories

 - [ ] install something randy needs:  I think this will be possible in the release planned for this summer, but for now it would be nice to use Jake's mpld3 package, which doesn't seem to be installed.  I tried downloading and following the instructions at   https://github.com/jakevdp/mpld3 but didn't have permissions.  Is this something you could install globally?

  - [ ] make this standard  -- https://github.com/biocore/scikit-bio   -- see https://mail.google.com/mail/u/0/?shva=1#inbox/1454ce211132e2bf



 - [ ] MAYBE -- or maybe not -- change bup_storage to never delete account: it's very useful for linking projects and sharing files to have account available at all times.  will make, e.g., persistent sshfs possible; make sure .ssh is not ssh excluded from rsync

- [ ] have stable ipv6 project ip addresses be would be a huge *win*.  LXC would make that possible.

- [ ] deal with the exception around this - codecs.open(self.save_log,'a',"utf-8-sig").write(json.dumps(r)+'\n')

- [ ] go through and chown/sync every project systematically; evidently I didn't in the current migration, so I'll just put a chown in the start script for now -- this slows things down, but is temporary.

- [ ] update quota information using du script and re-enable quotas

- [ ] write code that cleans up /bup/projects fs by removing .sagemathcloud directories, etc., of projects not used for a while

- [ ] make it so move is never automatic but prompted?

- [ ] automated rolloing snapshots of bup/projects

- [ ] add bup quota as a standard part of settings, and refuse to make further snapshots if bup usage exceeds 3 times user disk quota.  This will avoid a horrible edge case.   Critical that this produces an error that the user learns about.  This will happen for some users.  Alternatively, I could periodically rebuild those bup repos with many snapshots deleted - that would be much nicer and is totally do-able.

- [ ] script to cleanup bup repos, e.g., delete tmp files, maybe recompact, etc.

- [ ] manual project move system -- bring it back...


======

 - [x] delete all lines in compute vm's /etc/passwd files that include /bup/projects.



 - [x] 99 projects have >3 replicas in practice, though not (usually) in db -- these appear "lost" to users in some cases.
   For each:
       - copy all files into their current location
       - move files for projects not specifically listed in bup_last_save to a TO_DELETE location

  Plan:

    phase 1:
     - sync from *all* actual replicas to master
     - save from master to *all* replicas
    phase 2:
     - add all replicas to bup_locations in database.

  Update everything.

  Later -- reduce number of replicas in some cases.

- [x] read consistency issue: if db say bup_last_save empty, but it isn't, we destroy everything and completely loose project (?!)

 - project_id=48099d72-a9f0-4090-a4a5-a5681b612222
  bup_last_save *should* be these,
 - 10.1.4.5 10.1.15.5 10.3.7.4
  but suddenly became these:  10.1.5.5, 10.1.21.5, 10.3.4.4

  Only explanation, after reading the code is that I better run nodetool repair.



 - [x] change status to not bail when value is False; otherwise, broken sage ==> can't access project!


 - [x] gce booting configuration:

        fdisk /dev/sdd
        mkfs.ext4 /dev/sdd1 && mkdir /mnt/conf2 && mount /dev/sdd1 /mnt/conf2 && rsync -axvH /mnt/conf/ /mnt/conf2/ && ls /mnt/conf2 && sync


 - [x] all proxy server stuff is broken, so ipython, latex, file download, etc.

    - try again using the testing server

 - [x] gce nodes -- need to fix /bup/projects2 issue,


 - [x] include some more anti-bitcoin mining measures.

{'fraction_non_alphanumeric': 0.09264305177111716, 'fraction_numerical': 0.06533454435361792, 'mean_word_length': 3.145582329317269, 'pattern_counts': {'":': 1, '<': 3, '<?xml version=': 0, '>': 12, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 33, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7417148', 'n_tokens_mistral': 6073, 'n_tokens_neox': 5283, 'n_words': 2066}