<commit_before>
### http://metalearning.ml/2018/#submission-instructions

Some of the fundamental questions that this workshop aims to address are:

*  What are the fundamental differences in the learning “task” compared to traditional “non-meta” learners?
*  Is there a practical limit to the number of meta-learning layers (e.g., would a meta-meta-meta-learning algorithm be of practical use)?
*  How can we design more sample-efficient meta-learning methods?
*  How can we exploit our domain knowledge to effectively guide the meta-learning process?
*  What are the meta-learning processes in nature (e.g, in humans), and how can we take inspiration from them?
*  Which ML approaches are best suited for meta-learning, in which circumstances, and why?
*  What principles can we learn from meta-learning to help us design the next generation of learning systems?


### https://r2learning.github.io/

Topics include, but are not limited to:

*  Algorithmic approaches: E.g., probabilistic generative models, message-passing neural networks, embedding methods, dimensionality reduction techniques, group-invariant architectures etc. for relational data
*  Theoretical aspects: E.g., when and why do learned representations aid relational reasoning? How does the non-i.i.d. nature of relational data conflict with our current understanding of representation learning?
*  Optimization and scalability challenges due to the inherent discreteness and curse of dimensionality of relational datasets
*  Evaluation of learned relational representations
*  Security and privacy challenges
*  Domain-specific applications
*  Any other topic of interest


Zero-Shot Relation Extraction via Reading Comprehension
*  https://arxiv.org/pdf/1706.04115.pdf
*  http://nlp.cs.washington.edu/zeroshot/  # Data
*  https://bitbucket.org/omerlevy/bidaf_no_answer  # Code

Graph Convolution over Pruned Dependency Trees Improves Relation Extraction
*  https://arxiv.org/abs/1809.10185

Scene Graph Parsing as Dependency Parsing  
*  https://arxiv.org/pdf/1803.09189.pdf
*  https://github.com/Yusics/bist-parser/tree/sgparser


BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
*  https://arxiv.org/pdf/1810.04805.pdf
  
## Other datasets

SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals
*  http://aclweb.org/anthology/S/S10/S10-1006.pdf
*  http://www.kozareva.com/downloads.html  # SemEval-2010 Task 8 Dataset

Teaching Machines to Read and Comprehend
*  https://arxiv.org/abs/1506.03340
*  NER-anonamised Daily Mail stories with questions
*  https://github.com/deepmind/rc-data # Data creator
*  https://cs.nyu.edu/~kcho/DMQA/  # Data raw download


## Pull in data

cd REPO/notebooks/work-in-progress/2018-10_ZeroShotRelationships

wget http://nlp.cs.washington.edu/zeroshot/evaluate.py

ln -s /mnt/rdai/reddragon/2018-10_ZeroShotRelationships_orig orig

cd orig
#git clone https://bitbucket.org/omerlevy/bidaf_no_answer
#hg pull && hg update default
wget https://bitbucket.org/omerlevy/bidaf_no_answer/get/2e9868b224e4.zip  # 96Kb
unzip 2e9868b224e4.zip
cd omerlevy-bidaf_no_answer-2e9868b224e4/


wget http://nlp.cs.washington.edu/zeroshot/relation_splits.tar.bz2
bunzip2 relation_splits.tar.bz2  # Output .tar : 1,936,209,920
tar -xf relation_splits.tar 
## rm relation_splits.tar  # should be fine

#wget http://nlp.cs.washington.edu/zeroshot/raw_data.tar.bz2    # 1.3Gb
#bunzip2 relation_splits.tar.bz2  # Output .tar : 6,909,859,840
#tar -xf relation_splits.tar 





# OK, so ignore the previous paper's model : It wants TF 0.11 ...
#### IGNORE START  #### 

# Inaccurate (negative_example): 
# IUCN conservation status	What is the endangered status of XXX?	Bawean deer	The Bawean deer (Hyelaphus kuhlii), also known as Kuhl's hog deer or Bawean hog deer, is a highly threatened species of deer found only in the island 
of Bawean (Gresik Regency) in Indonesia.

# Need to get 'glove.6B.100d.txt' into current directory...


. ~/env3/bin/activate

# ./run_prep.sh relation_splits  # Bad paths... so do the following : 
run_name=relation_splits
python -m zeroshot.zeroshot2squad ${run_name}/train ${run_name}/train-v1.1.json
python -m zeroshot.zeroshot2squad ${run_name}/test ${run_name}/dev-v1.1.json
python -m squad.prepro -s ${run_name} -t ${run_name}

num=1
run_squad=${run_name}-squad/${num}
mkdir -p ${run_squad}
python -m zeroshot.zeroshot2squad ${run_name}/train.${num} ${run_squad}/train-v1.1.json
python -m zeroshot.zeroshot2squad ${run_name}/test.${num} ${run_squad}/dev-v1.1.json
python -m squad.prepro -s ${run_squad} --glove_dir . -t ${run_squad}  # This will take a while...

# my/tensorflow/nn.py : First lines should be :
"""
#from tensorflow.python.ops.rnn_cell import _linear
from tensorflow.contrib.rnn.python.ops import core_rnn_cell
linear = core_rnn_cell._linear
"""

# my/tensorflow/rnn.py : Near top should be :
"""
# from tensorflow.python.ops.rnn import bidirectional_rnn as _bidirectional_rnn
from tensorflow.python.ops.rnn import static_bidirectional_rnn as _bidirectional_rnn
"""

python -m basic.cli --mode train --noload --debug

# The model was trained with NVidia Titan X (Pascal Architecture, 2016). 
# The model requires at least 12GB of GPU RAM. 
# If your GPU RAM is smaller than 12GB, you can either decrease batch size (performance might degrade), 
# or you can use multi GPU (see below). 
# The training converges at ~18k steps, and it took ~4s per step (i.e. ~20 hours).

python -m basic.cli --mode train --noload --batch_size 400 --sent_size_th 60 --num_steps 0 --num_epochs 3 \
                    --len_opt --cluster --num_gpus 1 --run_id 1 --data_dir ${run_squad} --eval_period 500
# When something starts to work ...
#Loaded 835951/840000 examples from train
#Loaded 11870/12000 examples from dev

#### IGNORE END #### 


cd  ../../orig
git clone https://github.com/openai/finetune-transformer-lm.git
rm -rf  finetune-transformer-lm/.git
git clone https://github.com/huggingface/pytorch-openai-transformer-lm.git

# cp --no-clobber orig/pytorch-openai-transformer-lm/text_utils.py ..  # Their text_utils needs updating : Bring it into my repo

# Need to go to the Rocstories website to get download links to files - put them in 'rocstories_dataset'
ls -l rocstories_dataset/
-rw-rw-r--. 1 andrewsm andrewsm   574480 Oct 14 18:58 'cloze_test_test__spring2016 - cloze_test_ALL_test.csv'
-rw-rw-r--. 1 andrewsm andrewsm   575510 Oct 14 18:56 'cloze_test_val__spring2016 - cloze_test_ALL_val.csv'
-rw-rw-r--. 1 andrewsm andrewsm 14537764 Oct 14 18:52 'ROCStories_winter2017 - ROCStories_winter2017.csv'

python train.py --dataset rocstories --desc rocstories --submit --analysis \
                --data_dir rocstories_dataset \
                --bpe_path=../finetune-transformer-lm/model/vocab_40000.bpe \
                --encoder_path=../finetune-transformer-lm/model/encoder_bpe_40000.json 
# Works




cd REPO/notebooks/work-in-progress/2018-10_ZeroShotRelationships

. ~/env3/bin/activate

python relation_split_to_hdf5.py --phase=train
python relation_split_to_hdf5.py --phase=dev,test --save_bpe


# TODO : 
#   Create HDF5 dataset reader per blog post.  Assume that ids=ALL if not given
#   Create additional 'head' with attention-direction loss fn
#   Make sure that something happens when training on the dev set (model loading, training steps, model saving, etc)
#   Kick off training with attn_loss=0 (leave running?)
#   Think about GCP P100 for additional training...  Maybe multiple P100s?  or Multiple instances?
#   Create a test script that can evaluate the results on a saved model
#   Write paper...


# Try out code with ...
python train_on_onestep.py --relation_hdf5=dev.1_all.hdf5 --stub=base --dep_fac=0.0

# Real deal :
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=base --dep_fac=0.0

Saving Checkpoint : './checkpoints/model-stepwise_base_0000-150048.pth', loss_recent=1.0170
...
Saving Checkpoint : './checkpoints/model-stepwise_base_0000-650208.pth', loss_recent=0.5590
Saving Checkpoint : './checkpoints/model-stepwise_base_0000-700224.pth', loss_recent=0.5383
Saving Checkpoint : './checkpoints/model-stepwise_base_0000-750240.pth', loss_recent=0.5181
Saving Checkpoint : './checkpoints/model-stepwise_base_0000-800256.pth', loss_recent=0.5061
Saving Checkpoint : './checkpoints/model-stepwise_base_0001-011136.pth', loss_recent=0.5059
Saving Checkpoint : './checkpoints/model-stepwise_base_0001-061152.pth', loss_recent=0.4592
Saving Checkpoint : './checkpoints/model-stepwise_base_0001-211200.pth', loss_recent=0.4179
...
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-077984.pth', loss_recent=0.2855
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-128000.pth', loss_recent=0.2753
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-178016.pth', loss_recent=0.2605
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-236992.pth', loss_recent=0.2605
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-337024.pth', loss_recent=0.2382
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-393664.pth', loss_recent=0.2382
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-494624.pth', loss_recent=0.2294
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-594656.pth', loss_recent=0.2141
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-703968.pth', loss_recent=0.2090
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-753984.pth', loss_recent=0.2079
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-804000.pth', loss_recent=0.2041


-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 04:20 model-stepwise_base_0000-050016.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 04:41 model-stepwise_base_0000-100032.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 05:02 model-stepwise_base_0000-150048.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 05:23 model-stepwise_base_0000-200064.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 05:44 model-stepwise_base_0000-250080.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 06:05 model-stepwise_base_0000-300096.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 06:27 model-stepwise_base_0000-350112.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 06:48 model-stepwise_base_0000-400128.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 07:09 model-stepwise_base_0000-450144.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 07:30 model-stepwise_base_0000-500160.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 07:51 model-stepwise_base_0000-550176.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 08:13 model-stepwise_base_0000-600192.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 08:34 model-stepwise_base_0000-650208.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 08:55 model-stepwise_base_0000-700224.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 09:16 model-stepwise_base_0000-750240.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 09:37 model-stepwise_base_0000-800256.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 09:59 model-stepwise_base_0001-011136.pth


#Time per 1000.  ~ >>> (3*60+10)*60./(800256-350112)*1000.
# 25.325229259970143 secs on TitanX

morning
Factor hints (class_loss=  4.5217, deps_loss= 5344.4214, fac=0.00084607)
Factor hints (class_loss=  8.4208, deps_loss= 6178.9131, fac=0.00136283)
Factor hints (class_loss= 34.4343, deps_loss= 6007.8652, fac=0.00573154)
Factor hints (class_loss=  6.2002, deps_loss= 5839.6787, fac=0.00106173)
Factor hints (class_loss= 14.6076, deps_loss= 6349.5327, fac=0.00230058)
Factor hints (class_loss= 13.9885, deps_loss= 6815.1782, fac=0.00205256)
Factor hints (class_loss=  6.0767, deps_loss= 5829.8833, fac=0.00104233)
Factor hints (class_loss=  9.1029, deps_loss= 5803.7627, fac=0.00156844)
Factor hints (class_loss=  1.2308, deps_loss= 5650.0684, fac=0.00021784)
Factor hints (class_loss=  9.0883, deps_loss= 5750.9033, fac=0.00158032)

evening
Factor hints (class_loss=  1.3447, deps_loss= 5558.3813, fac=0.00024193)
Factor hints (class_loss= 18.4624, deps_loss= 6056.8135, fac=0.00304820)
Factor hints (class_loss=  0.1400, deps_loss= 6122.2803, fac=0.00002287)
Factor hints (class_loss=  0.0526, deps_loss= 5790.4863, fac=0.00000908)
Factor hints (class_loss= 21.1384, deps_loss= 5803.6270, fac=0.00364228)
Factor hints (class_loss=  3.1123, deps_loss= 6002.5112, fac=0.00051850)
Factor hints (class_loss= 60.7331, deps_loss= 5879.1123, fac=0.01033032)
Factor hints (class_loss=  1.3860, deps_loss= 5668.1182, fac=0.00024452)
Factor hints (class_loss=  3.6776, deps_loss= 6045.1016, fac=0.00060836)
Factor hints (class_loss= 15.7819, deps_loss= 6411.4414, fac=0.00246152)
Factor hints (class_loss= 21.9232, deps_loss= 5676.2529, fac=0.00386227)
Factor hints (class_loss=  0.3627, deps_loss= 6797.3110, fac=0.00005336)
Factor hints (class_loss= 16.1702, deps_loss= 5596.6489, fac=0.00288926)
Factor hints (class_loss=  0.0677, deps_loss= 5657.2944, fac=0.00001196)
Factor hints (class_loss=  0.7334, deps_loss= 5572.8364, fac=0.00013160)
Factor hints (class_loss=  0.7809, deps_loss= 6325.0166, fac=0.00012346)
Factor hints (class_loss=  9.2523, deps_loss= 5544.1602, fac=0.00166883)

# Locally test the test script
python train_on_onestep.py --relation_hdf5=dev.1_all.hdf5 --stub=base --predict --checkpoint=./checkpoints/model-stepwise_base_0002-804000.pth

# Locally run the test script
python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=base --predict --checkpoint=./checkpoints/model-stepwise_base_0002-804000.pth

#Assessment via test_on_onestep : 
##  #precision=37.56% recall=40.10% F1=38.79%  # intersection/ans
##  precision=30.97% recall=33.06% F1=31.98%  # intersection/union of a_start_best, a_end_best

#Paper result for 'Multiple Templates' : (1_Allen_ZeroShotRelationExtraction-ReadingComprehension_1706.04115.pdf - Table 3)
##  precision=43.61% recall=36.45% F1=39.61%


# https://cloud.google.com/compute/docs/gpus/add-gpus
# Modify 'rdai-tts-p100-vm' 
#  to have 4*P100, and 
#    $0.43 USD per GPU per hour  (confirmed to have 16Gb of memory each)
#  attach a 200Gb persistent disk : 'zero-shot-relationships'
#    https://cloud.google.com/persistent-disk/

# https://cloud.google.com/tpu/docs/pricing
#   Potential : $1.35 USD per TPU per hour (preemptible)


export INSTANCE_NAME="rdai-tts-p100-vm"
gcloud compute instances start $INSTANCE_NAME

gcloud compute ssh $INSTANCE_NAME

cd ~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/
mkdir orig # ... Follow instructions above ...


gcloud compute scp *.hdf5 rdai-tts-p100-vm:~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/orig/omerlevy-bidaf_no_answer-2e9868b224e4/relation_splits/


cd ~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/orig/omerlevy-bidaf_no_answer-2e9868b224e4/relation_splits/
md5sum *
6e8fe51b869fecb6acd241df9fab07d8  dev.1_all.hdf5
de44a1feb2392ce91b03f3d7b5feba35  test.1_all.hdf5
e735c9a89f98613e2e64af0ac794bf5e  train.1_all.hdf5

# locally :
md5sum *.hdf5
6e8fe51b869fecb6acd241df9fab07d8  dev.1_all.hdf5
de44a1feb2392ce91b03f3d7b5feba35  test.1_all.hdf5
e735c9a89f98613e2e64af0ac794bf5e  train.1_all.hdf5


# Mount new persistent disk : 
#   https://devopscube.com/mount-extra-disks-on-google-cloud/
sudo lsblk
#NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
#sda      8:0    0   30G  0 disk 
#└─sda1   8:1    0   30G  0 part /
#sdb      8:16   0  200G  0 disk 


cd ~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/
mkdir checkpoints

## sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb

cd ~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/
sudo mount -o discard,defaults /dev/sdb checkpoints
sudo chmod a+x checkpoints



. ~/env3/bin/activate

python train_on_onestep.py --relation_hdf5=dev.1_all.hdf5 --stub=base --dep_fac=0.0
# Functions (after fixing :: grep 'f"' orig/pytorch-openai-transformer-lm/*.py) 3.6 string formatting

python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=base --dep_fac=0.0

Factor hints (class_loss=29012.3672, deps_loss=23580.5684, fac=1.23035063)
Factor hints (class_loss=455.4864, deps_loss=23485.1406, fac=0.01939466)
Factor hints (class_loss=271.1743, deps_loss=23552.4355, fac=0.01151364)
Factor hints (class_loss=217.7633, deps_loss=25605.2051, fac=0.00850465)
Time used for 0.06 of epoch 0: 300.5 seconds
  Time per 1000 lines : 6.005 seconds
  Expected finish time : Wednesday, October 17, 2018 12:21:37  (server)
  Expected finish time : Wednesday, October 17, 2018 12:21:37 +08+0800 (local)
Factor hints (class_loss=176.8094, deps_loss=23848.9160, fac=0.00741373)
Factor hints (class_loss=173.7655, deps_loss=22794.8516, fac=0.00762301)
Factor hints (class_loss=119.6408, deps_loss=25449.4863, fac=0.00470111)
Factor hints (class_loss=162.4608, deps_loss=24370.5605, fac=0.00666627)


screen -S QAZRE
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac1.0 --dep_fac=1.0


# Remotely test the test script
python train_on_onestep.py --relation_hdf5=dev.1_all.hdf5 --stub=fac1.0 --predict --checkpoint=./checkpoints/model-stepwise_fac1.0_0001-760576.pth

# Remotely run the test script
python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=fac1.0 --predict --checkpoint=./checkpoints/model-stepwise_fac1.0_0001-760576.pth

export REMOTE_BASE="rdai-tts-p100-vm:~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships"
export REL_PATH="orig/omerlevy-bidaf_no_answer-2e9868b224e4/relation_splits"
## mv ${REL_PATH}/test.1_all.hdf5_base.npz ${REL_PATH}/test.1_all.hdf5_fac1.0.npz
gcloud compute scp ${REMOTE_BASE}/${REL_PATH}/test.1_all.hdf5_fac1.0.npz  ${REL_PATH}/

# Assessment via test_on_onestep : 
## precision=24.67% recall=43.13% F1=31.39%




# Next run : 
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1 --checkpoint=./checkpoints/model-stepwise_fac0.1_0000-600192.pth

# Update with 'no answer' forced labels
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1 --checkpoint=./checkpoints/model-stepwise_fac0.1_0001-400128.pth

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44                 Driver Version: 396.44                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   76C    P0   193W / 250W |  10601MiB / 16280MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  Off  | 00000000:00:05.0 Off |                    0 |
| N/A   71C    P0   187W / 250W |   8699MiB / 16280MiB |     97%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P100-PCIE...  Off  | 00000000:00:06.0 Off |                    0 |
| N/A   70C    P0   189W / 250W |   8699MiB / 16280MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P100-PCIE...  Off  | 00000000:00:07.0 Off |                    0 |
| N/A   71C    P0   203W / 250W |   8699MiB / 16280MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1374      C   python                                     10591MiB |
|    1      1374      C   python                                      8689MiB |
|    2      1374      C   python                                      8689MiB |
|    3      1374      C   python                                      8689MiB |
+-----------------------------------------------------------------------------+

# Start again with 'no answer' forced labels
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1


Saving Checkpoint : './checkpoints/model-stepwise_fac0.1_0001-760576.pth', loss_recent=2.2232
Factor hints (class_loss=199.1927, deps_loss=  617.2078, fac=0.32273209)
Time used for 0.92 of epoch 1: 4506.6 seconds
  Time per 1000 lines : 5.843 seconds
  Expected finish in : 1.47 hours
  Expected finish time : Wednesday, October 17, 2018 23:37:18  (server)
  Expected finish time : Thursday, October 18, 2018 07:37:18 +08+0800 (local)
Factor hints (class_loss=215.0347, deps_loss=  622.1494, fac=0.34563187)
Factor hints (class_loss=199.8563, deps_loss=  689.3589, fac=0.28991617)
Factor hints (class_loss=215.8770, deps_loss=  829.4865, fac=0.26025383)
Factor hints (class_loss=221.7887, deps_loss=  877.3491, fac=0.25279410)
Time used for 0.98 of epoch 1: 4807.2 seconds
  Time per 1000 lines : 5.841 seconds
  Expected finish in : 1.39 hours
  Expected finish time : Wednesday, October 17, 2018 23:37:15  (server)
  Expected finish time : Thursday, October 18, 2018 07:37:15 +08+0800 (local)
Factor hints (class_loss=206.2093, deps_loss=  751.4877, fac=0.27440149)




Saving Checkpoint : './checkpoints/model-stepwise_fac0.1_0002-800256.pth', loss_recent=1.9000


pip install ftfy, spacy
python -m spacy download en
python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=fac0.1 --predict --checkpoint=./checkpoints/model-stepwise_fac0.1_0001-839936_end-epoch.pth

gcloud compute scp ${REMOTE_BASE}/${REL_PATH}/test.1_all.hdf5_fac0.1.npz  ${REL_PATH}/

# Assessment via test_on_onestep : 
## precision=55.43% recall=19.11% F1=28.42%  # using p_ij


python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1 --checkpoint=./checkpoints/model-stepwise_fac0.1_0001-839936_end-epoch.pth --n_epoch=5

# Relevant tabs :
https://arxiv.org/pdf/1810.04805.pdf
https://arxiv.org/pdf/1803.09189.pdf
https://arxiv.org/pdf/1706.04115.pdf
http://nlp.cs.washington.edu/zeroshot/
https://bitbucket.org/omerlevy/bidaf_no_answer/src/2e9868b224e4?at=default
https://arxiv.org/pdf/1809.10185.pdf
https://arxiv.org/pdf/1611.01603.pdf
https://allennlp.org/elmo
https://arxiv.org/pdf/1802.05365.pdf
https://tfhub.dev/google/elmo/2
https://github.com/huggingface/pytorch-openai-transformer-lm
https://arxiv.org/abs/1706.04115
https://arxiv.org/pdf/1806.08730.pdf

Factor hints (class_loss=210.2790, deps_loss=  634.4976, fac=0.33141021)
Factor hints (class_loss=192.0340, deps_loss=  551.7582, fac=0.34804023)
Saving End-epoch checkpoint : './checkpoints/model-stepwise_fac0.1_0001-839936_end-epoch.pth'

python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1 --checkpoint=./checkpoints/model-stepwise_fac0.1_0002-839936_end-epoch.pth --n_epoch=5



python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.02 --dep_fac=0.02

Factor hints (class_loss=223.2570, deps_loss= 1481.2102, fac=0.15072607)
Factor hints (class_loss=211.9604, deps_loss= 1401.0105, fac=0.15129106)
Factor hints (class_loss=196.1872, deps_loss= 1312.2306, fac=0.14950667)
Factor hints (class_loss=204.6235, deps_loss= 1759.9146, fac=0.11626901)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0001-160384.pth', loss_recent=2.0039

Factor hints (class_loss=234.9280, deps_loss= 1539.1934, fac=0.15263058)
Factor hints (class_loss=198.6711, deps_loss= 1532.9457, fac=0.12960086)
Factor hints (class_loss=222.6590, deps_loss= 1567.9419, fac=0.14200719)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0001-360448.pth', loss_recent=1.9098

Factor hints (class_loss=216.5104, deps_loss= 1278.1627, fac=0.16939190)
Factor hints (class_loss=202.8433, deps_loss= 1384.3292, fac=0.14652819)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0001-560512.pth', loss_recent=1.8569
Factor hints (class_loss=201.3995, deps_loss= 1333.3656, fac=0.15104597)
Factor hints (class_loss=219.0672, deps_loss= 1239.7926, fac=0.17669667)

Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0001-760576.pth', loss_recent=1.8242
Factor hints (class_loss=189.2207, deps_loss= 1184.1119, fac=0.15979971)
Factor hints (class_loss=217.4455, deps_loss= 1151.4514, fac=0.18884469)
Factor hints (class_loss=196.9379, deps_loss= 1192.1925, fac=0.16518969)

Saving End-epoch checkpoint : './checkpoints/model-stepwise_fac0.02_0001-839936_end-epoch.pth'

Factor hints (class_loss=194.4754, deps_loss= 1045.6996, fac=0.18597633)
Factor hints (class_loss=187.8143, deps_loss= 1010.2685, fac=0.18590533)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0002-120704.pth', loss_recent=1.7843
Factor hints (class_loss=199.0331, deps_loss= 1381.5564, fac=0.14406438)

Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0002-320768.pth', loss_recent=1.7356
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0002-520832.pth', loss_recent=1.7111

Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0002-720896.pth', loss_recent=1.6952
Saving End-epoch checkpoint : './checkpoints/model-stepwise_fac0.02_0002-839936_end-epoch.pth'


python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=fac0.02 --predict --checkpoint=./checkpoints/model-stepwise_fac0.02_0002-839936_end-epoch.pth

gcloud compute scp ${REMOTE_BASE}/${REL_PATH}/test.1_all.hdf5_fac0.02.npz  ${REL_PATH}/

# Assessment via test_on_onestep : 
## precision=55.43% recall=19.11% F1=28.42%  # using p_ij  #  fac0.1
## 





# Now running this with the baseno [0:2] ~ 4,3 training data...
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=baseno --dep_fac=0.0 --n_epoch=5

Factor hints (class_loss=217.6262, deps_loss=25597.0469, fac=0.00850201)
Factor hints (class_loss=195.7768, deps_loss=25347.2812, fac=0.00772378)
Factor hints (class_loss=206.4771, deps_loss=24933.8594, fac=0.00828099)
Time used for 0.85 of epoch 2: 4809.7 seconds
  Time per 1000 lines : 6.735 seconds
  Expected finish in : 3.38 hours
  Expected finish time : Friday, October 19, 2018 02:15:36 +08+0800 (local)
Factor hints (class_loss=199.8273, deps_loss=23559.3867, fac=0.00848185)
Saving Checkpoint : './checkpoints/model-stepwise_baseno_0002-720896.pth', loss_recent=1.5658
Saving Checkpoint : './checkpoints/model-stepwise_baseno_0003-081024.pth', loss_recent=1.5545
Saving Checkpoint : './checkpoints/model-stepwise_baseno_0004-441472.pth', loss_recent=1.4768
Factor hints (class_loss=204.8316, deps_loss=24427.4961, fac=0.00838529)
Factor hints (class_loss=183.4029, deps_loss=24861.5137, fac=0.00737698)
Saving Checkpoint : './checkpoints/model-stepwise_baseno_0004-641536.pth', loss_recent=1.4739
Factor hints (class_loss=184.2379, deps_loss=25274.0234, fac=0.00728961)
Factor hints (class_loss=184.8552, deps_loss=23345.1270, fac=0.00791836)

# model-stepwise_baseno_0004-641536.pth COMPLAINS, use instead : model-stepwise_baseno_0004-441472.pth

python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=baseno --predict --checkpoint=./checkpoints/model-stepwise_baseno_0004-441472.pth
gcloud compute scp ${REMOTE_BASE}/${REL_PATH}/test.1_all.hdf5_baseno.npz  ${REL_PATH}/


##  Now move over to (no-ans at token_clf) training

# On GCP : 
git pull
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=base-clf-extra --dep_fac=0.0 --n_epoch=4 --extra_block
  Expected finish time : Friday, October 19, 2018 10:46:59 +08+0800 (Asia/Singapore)
Factor hints (class_loss= 42.5011, deps_loss=24650.1445, fac=0.00172417)
Saving Checkpoint : './checkpoints/model-stepwise_base-clf-extra_0000-600192.pth', loss_recent=0.6231
Factor hints (class_loss= 97.1845, deps_loss=23209.7129, fac=0.00418723)
Factor hints (class_loss= 71.8025, deps_loss=24324.6543, fac=0.00295184)
...
Factor hints (class_loss= 75.3395, deps_loss=23493.6504, fac=0.00320680)
Saving End-epoch checkpoint : './checkpoints/model-stepwise_base-clf-extra_0000-839936_end-epoch.pth'
Factor hints (class_loss= 78.9366, deps_loss=23639.1445, fac=0.00333923)
Factor hints (class_loss= 60.5935, deps_loss=23108.5254, fac=0.00262213)
Factor hints (class_loss= 53.5047, deps_loss=23190.4785, fac=0.00230718)
Factor hints (class_loss= 45.4149, deps_loss=24913.8945, fac=0.00182288)
...
Saving End-epoch checkpoint : './checkpoints/model-stepwise_base-clf-extra_0001-839936_end-epoch.pth'
Factor hints (class_loss= 55.6968, deps_loss=23727.3867, fac=0.00234736)
Factor hints (class_loss= 43.2192, deps_loss=23148.8574, fac=0.00186701)
Factor hints (class_loss= 23.4774, deps_loss=23160.7090, fac=0.00101367)
Factor hints (class_loss= 16.8954, deps_loss=25083.6582, fac=0.00067356)
Time used for 0.05 of epoch 2: 300.5 seconds
...
Saving End-epoch checkpoint : './checkpoints/model-stepwise_base-clf-extra_0002-839936_end-epoch.pth'
Factor hints (class_loss= 42.1381, deps_loss=23662.4453, fac=0.00178080)
Factor hints (class_loss= 22.2152, deps_loss=23115.2637, fac=0.00096106)
Factor hints (class_loss= 14.2607, deps_loss=23218.6602, fac=0.00061419)
Factor hints (class_loss= 16.2253, deps_loss=25105.4043, fac=0.00064629)
...
Factor hints (class_loss= 17.5010, deps_loss=23592.5371, fac=0.00074180)
Factor hints (class_loss=  4.5477, deps_loss=24901.6953, fac=0.00018262)
Saving Checkpoint : './checkpoints/model-stepwise_base-clf-extra_0003-281088.pth', loss_recent=0.1168
Factor hints (class_loss=  3.9022, deps_loss=24384.4844, fac=0.00016003)
...
Factor hints (class_loss= 22.9843, deps_loss=24748.4570, fac=0.00092872)
Factor hints (class_loss= 18.2148, deps_loss=25440.7285, fac=0.00071597)
Saving Checkpoint : './checkpoints/model-stepwise_base-clf-extra_0003-481152.pth', loss_recent=0.1085
Factor hints (class_loss=  5.0076, deps_loss=24175.3359, fac=0.00020714)
Factor hints (class_loss=  4.6461, deps_loss=24355.6562, fac=0.00019076)
Factor hints (class_loss= 52.0994, deps_loss=25632.4824, fac=0.00203255)
Saving Checkpoint : './checkpoints/model-stepwise_base-clf-extra_0003-681216.pth', loss_recent=0.1024
Factor hints (class_loss=  5.5085, deps_loss=25472.1855, fac=0.00021625)
Factor hints (class_loss=  1.7764, deps_loss=25121.1641, fac=0.00007071)
Time used for 0.85 of epoch 3: 5108.6 seconds
  Time per 1000 lines : 7.192 seconds
  Expected finish in : 0.26 hours
  Expected finish time : Friday, October 19, 2018 10:45:10 +08+0800 (Asia/Singapore)
Factor hints (class_loss= 18.1204, deps_loss=23622.1484, fac=0.00076709)
Factor hints (class_loss= 21.3947, deps_loss=24128.2852, fac=0.00088670)
Factor hints (class_loss= 12.1556, deps_loss=24368.9648, fac=0.00049882)
Time used for 0.90 of epoch 3: 5409.3 seconds
  Time per 1000 lines : 7.191 seconds
  Expected finish in : 0.18 hours
  Expected finish time : Friday, October 19, 2018 10:45:09 +08+0800 (Asia/Singapore)
Factor hints (class_loss=  3.1787, deps_loss=23763.5664, fac=0.00013376)
Factor hints (class_loss=  5.3035, deps_loss=26530.5723, fac=0.00019990)
Factor hints (class_loss= 19.4625, deps_loss=24494.5547, fac=0.00079456)
Factor hints (class_loss=  1.7563, deps_loss=23389.7383, fac=0.00007509)
Time used for 0.95 of epoch 3: 5710.0 seconds
  Time per 1000 lines : 7.189 seconds
  Expected finish in : 0.09 hours
  Expected finish time : Friday, October 19, 2018 10:45:07 +08+0800 (Asia/Singapore)
Factor hints (class_loss=  1.7182, deps_loss=24661.6191, fac=0.00006967)
Factor hints (class_loss=  6.0365, deps_loss=25655.3418, fac=0.00023529)
Factor hints (class_loss= 39.1545, deps_loss=23551.1094, fac=0.00166253)
Time used for 1.00 of epoch 3: 6010.6 seconds
  Time per 1000 lines : 7.188 seconds
  Expected finish in : 0.01 hours
  Expected finish time : Friday, October 19, 2018 10:45:06 +08+0800 (Asia/Singapore)
Saving End-epoch checkpoint : './checkpoints/model-stepwise_base-clf-extra_0003-839936_end-epoch.pth'


# On local :
git pull
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1-clf-extra --dep_fac=0.1 --n_epoch=1 --extra_block
  Expected finish time : Friday, October 19, 2018 10:26:38 +08+0800 (Asia/Singapore)
Factor hints (class_loss= 19.5906, deps_loss=  244.4604, fac=0.08013825)
Factor hints (class_loss= 11.7347, deps_loss=  211.1650, fac=0.05557138)
Factor hints (class_loss=  9.3514, deps_loss=  199.4653, fac=0.04688250)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.1-clf-extra_0000-600096.pth', loss_recent=1.3007
Factor hints (class_loss= 21.3773, deps_loss=  199.2804, fac=0.10727230)
Factor hints (class_loss= 19.8534, deps_loss=  191.3389, fac=0.10376021)
Factor hints (class_loss=  2.9317, deps_loss=  185.7273, fac=0.01578493)
Time used for 1.00 of epoch 0: 23436.6 seconds
  Time per 1000 lines : 27.976 seconds
  Expected finish in : 0.02 hours
  Expected finish time : Friday, October 19, 2018 10:26:42 +08+0800 (Asia/Singapore)
Factor hints (class_loss= 30.0499, deps_loss=  242.6052, fac=0.12386338)
Saving End-epoch checkpoint : './checkpoints/model-stepwise_fac0.1-clf-extra_0000-839968_end-epoch.pth'


# TODO On GCP : 
git pull
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.02-clf-extra --dep_fac=0.02 --n_epoch=5 --extra_block

<commit_msg>Results for fac0.02 with clf no-answers and extra_block<commit_after>
### http://metalearning.ml/2018/#submission-instructions

Some of the fundamental questions that this workshop aims to address are:

*  What are the fundamental differences in the learning “task” compared to traditional “non-meta” learners?
*  Is there a practical limit to the number of meta-learning layers (e.g., would a meta-meta-meta-learning algorithm be of practical use)?
*  How can we design more sample-efficient meta-learning methods?
*  How can we exploit our domain knowledge to effectively guide the meta-learning process?
*  What are the meta-learning processes in nature (e.g, in humans), and how can we take inspiration from them?
*  Which ML approaches are best suited for meta-learning, in which circumstances, and why?
*  What principles can we learn from meta-learning to help us design the next generation of learning systems?


### https://r2learning.github.io/

Topics include, but are not limited to:

*  Algorithmic approaches: E.g., probabilistic generative models, message-passing neural networks, embedding methods, dimensionality reduction techniques, group-invariant architectures etc. for relational data
*  Theoretical aspects: E.g., when and why do learned representations aid relational reasoning? How does the non-i.i.d. nature of relational data conflict with our current understanding of representation learning?
*  Optimization and scalability challenges due to the inherent discreteness and curse of dimensionality of relational datasets
*  Evaluation of learned relational representations
*  Security and privacy challenges
*  Domain-specific applications
*  Any other topic of interest


Zero-Shot Relation Extraction via Reading Comprehension
*  https://arxiv.org/pdf/1706.04115.pdf
*  http://nlp.cs.washington.edu/zeroshot/  # Data
*  https://bitbucket.org/omerlevy/bidaf_no_answer  # Code

Graph Convolution over Pruned Dependency Trees Improves Relation Extraction
*  https://arxiv.org/abs/1809.10185

Scene Graph Parsing as Dependency Parsing  
*  https://arxiv.org/pdf/1803.09189.pdf
*  https://github.com/Yusics/bist-parser/tree/sgparser


BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
*  https://arxiv.org/pdf/1810.04805.pdf
  
## Other datasets

SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals
*  http://aclweb.org/anthology/S/S10/S10-1006.pdf
*  http://www.kozareva.com/downloads.html  # SemEval-2010 Task 8 Dataset

Teaching Machines to Read and Comprehend
*  https://arxiv.org/abs/1506.03340
*  NER-anonamised Daily Mail stories with questions
*  https://github.com/deepmind/rc-data # Data creator
*  https://cs.nyu.edu/~kcho/DMQA/  # Data raw download


## Pull in data

cd REPO/notebooks/work-in-progress/2018-10_ZeroShotRelationships

wget http://nlp.cs.washington.edu/zeroshot/evaluate.py

ln -s /mnt/rdai/reddragon/2018-10_ZeroShotRelationships_orig orig

cd orig
#git clone https://bitbucket.org/omerlevy/bidaf_no_answer
#hg pull && hg update default
wget https://bitbucket.org/omerlevy/bidaf_no_answer/get/2e9868b224e4.zip  # 96Kb
unzip 2e9868b224e4.zip
cd omerlevy-bidaf_no_answer-2e9868b224e4/


wget http://nlp.cs.washington.edu/zeroshot/relation_splits.tar.bz2
bunzip2 relation_splits.tar.bz2  # Output .tar : 1,936,209,920
tar -xf relation_splits.tar 
## rm relation_splits.tar  # should be fine

#wget http://nlp.cs.washington.edu/zeroshot/raw_data.tar.bz2    # 1.3Gb
#bunzip2 relation_splits.tar.bz2  # Output .tar : 6,909,859,840
#tar -xf relation_splits.tar 





# OK, so ignore the previous paper's model : It wants TF 0.11 ...
#### IGNORE START  #### 

# Inaccurate (negative_example): 
# IUCN conservation status	What is the endangered status of XXX?	Bawean deer	The Bawean deer (Hyelaphus kuhlii), also known as Kuhl's hog deer or Bawean hog deer, is a highly threatened species of deer found only in the island 
of Bawean (Gresik Regency) in Indonesia.

# Need to get 'glove.6B.100d.txt' into current directory...


. ~/env3/bin/activate

# ./run_prep.sh relation_splits  # Bad paths... so do the following : 
run_name=relation_splits
python -m zeroshot.zeroshot2squad ${run_name}/train ${run_name}/train-v1.1.json
python -m zeroshot.zeroshot2squad ${run_name}/test ${run_name}/dev-v1.1.json
python -m squad.prepro -s ${run_name} -t ${run_name}

num=1
run_squad=${run_name}-squad/${num}
mkdir -p ${run_squad}
python -m zeroshot.zeroshot2squad ${run_name}/train.${num} ${run_squad}/train-v1.1.json
python -m zeroshot.zeroshot2squad ${run_name}/test.${num} ${run_squad}/dev-v1.1.json
python -m squad.prepro -s ${run_squad} --glove_dir . -t ${run_squad}  # This will take a while...

# my/tensorflow/nn.py : First lines should be :
"""
#from tensorflow.python.ops.rnn_cell import _linear
from tensorflow.contrib.rnn.python.ops import core_rnn_cell
linear = core_rnn_cell._linear
"""

# my/tensorflow/rnn.py : Near top should be :
"""
# from tensorflow.python.ops.rnn import bidirectional_rnn as _bidirectional_rnn
from tensorflow.python.ops.rnn import static_bidirectional_rnn as _bidirectional_rnn
"""

python -m basic.cli --mode train --noload --debug

# The model was trained with NVidia Titan X (Pascal Architecture, 2016). 
# The model requires at least 12GB of GPU RAM. 
# If your GPU RAM is smaller than 12GB, you can either decrease batch size (performance might degrade), 
# or you can use multi GPU (see below). 
# The training converges at ~18k steps, and it took ~4s per step (i.e. ~20 hours).

python -m basic.cli --mode train --noload --batch_size 400 --sent_size_th 60 --num_steps 0 --num_epochs 3 \
                    --len_opt --cluster --num_gpus 1 --run_id 1 --data_dir ${run_squad} --eval_period 500
# When something starts to work ...
#Loaded 835951/840000 examples from train
#Loaded 11870/12000 examples from dev

#### IGNORE END #### 


cd  ../../orig
git clone https://github.com/openai/finetune-transformer-lm.git
rm -rf  finetune-transformer-lm/.git
git clone https://github.com/huggingface/pytorch-openai-transformer-lm.git

# cp --no-clobber orig/pytorch-openai-transformer-lm/text_utils.py ..  # Their text_utils needs updating : Bring it into my repo

# Need to go to the Rocstories website to get download links to files - put them in 'rocstories_dataset'
ls -l rocstories_dataset/
-rw-rw-r--. 1 andrewsm andrewsm   574480 Oct 14 18:58 'cloze_test_test__spring2016 - cloze_test_ALL_test.csv'
-rw-rw-r--. 1 andrewsm andrewsm   575510 Oct 14 18:56 'cloze_test_val__spring2016 - cloze_test_ALL_val.csv'
-rw-rw-r--. 1 andrewsm andrewsm 14537764 Oct 14 18:52 'ROCStories_winter2017 - ROCStories_winter2017.csv'

python train.py --dataset rocstories --desc rocstories --submit --analysis \
                --data_dir rocstories_dataset \
                --bpe_path=../finetune-transformer-lm/model/vocab_40000.bpe \
                --encoder_path=../finetune-transformer-lm/model/encoder_bpe_40000.json 
# Works




cd REPO/notebooks/work-in-progress/2018-10_ZeroShotRelationships

. ~/env3/bin/activate

python relation_split_to_hdf5.py --phase=train
python relation_split_to_hdf5.py --phase=dev,test --save_bpe


# TODO : 
#   Create HDF5 dataset reader per blog post.  Assume that ids=ALL if not given
#   Create additional 'head' with attention-direction loss fn
#   Make sure that something happens when training on the dev set (model loading, training steps, model saving, etc)
#   Kick off training with attn_loss=0 (leave running?)
#   Think about GCP P100 for additional training...  Maybe multiple P100s?  or Multiple instances?
#   Create a test script that can evaluate the results on a saved model
#   Write paper...


# Try out code with ...
python train_on_onestep.py --relation_hdf5=dev.1_all.hdf5 --stub=base --dep_fac=0.0

# Real deal :
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=base --dep_fac=0.0

Saving Checkpoint : './checkpoints/model-stepwise_base_0000-150048.pth', loss_recent=1.0170
...
Saving Checkpoint : './checkpoints/model-stepwise_base_0000-650208.pth', loss_recent=0.5590
Saving Checkpoint : './checkpoints/model-stepwise_base_0000-700224.pth', loss_recent=0.5383
Saving Checkpoint : './checkpoints/model-stepwise_base_0000-750240.pth', loss_recent=0.5181
Saving Checkpoint : './checkpoints/model-stepwise_base_0000-800256.pth', loss_recent=0.5061
Saving Checkpoint : './checkpoints/model-stepwise_base_0001-011136.pth', loss_recent=0.5059
Saving Checkpoint : './checkpoints/model-stepwise_base_0001-061152.pth', loss_recent=0.4592
Saving Checkpoint : './checkpoints/model-stepwise_base_0001-211200.pth', loss_recent=0.4179
...
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-077984.pth', loss_recent=0.2855
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-128000.pth', loss_recent=0.2753
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-178016.pth', loss_recent=0.2605
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-236992.pth', loss_recent=0.2605
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-337024.pth', loss_recent=0.2382
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-393664.pth', loss_recent=0.2382
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-494624.pth', loss_recent=0.2294
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-594656.pth', loss_recent=0.2141
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-703968.pth', loss_recent=0.2090
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-753984.pth', loss_recent=0.2079
Saving Checkpoint : './checkpoints/model-stepwise_base_0002-804000.pth', loss_recent=0.2041


-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 04:20 model-stepwise_base_0000-050016.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 04:41 model-stepwise_base_0000-100032.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 05:02 model-stepwise_base_0000-150048.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 05:23 model-stepwise_base_0000-200064.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 05:44 model-stepwise_base_0000-250080.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 06:05 model-stepwise_base_0000-300096.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 06:27 model-stepwise_base_0000-350112.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 06:48 model-stepwise_base_0000-400128.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 07:09 model-stepwise_base_0000-450144.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 07:30 model-stepwise_base_0000-500160.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 07:51 model-stepwise_base_0000-550176.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 08:13 model-stepwise_base_0000-600192.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 08:34 model-stepwise_base_0000-650208.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 08:55 model-stepwise_base_0000-700224.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 09:16 model-stepwise_base_0000-750240.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 09:37 model-stepwise_base_0000-800256.pth
-rw-rw-r--. 1 andrewsm andrewsm 1409989614 Oct 17 09:59 model-stepwise_base_0001-011136.pth


#Time per 1000.  ~ >>> (3*60+10)*60./(800256-350112)*1000.
# 25.325229259970143 secs on TitanX

morning
Factor hints (class_loss=  4.5217, deps_loss= 5344.4214, fac=0.00084607)
Factor hints (class_loss=  8.4208, deps_loss= 6178.9131, fac=0.00136283)
Factor hints (class_loss= 34.4343, deps_loss= 6007.8652, fac=0.00573154)
Factor hints (class_loss=  6.2002, deps_loss= 5839.6787, fac=0.00106173)
Factor hints (class_loss= 14.6076, deps_loss= 6349.5327, fac=0.00230058)
Factor hints (class_loss= 13.9885, deps_loss= 6815.1782, fac=0.00205256)
Factor hints (class_loss=  6.0767, deps_loss= 5829.8833, fac=0.00104233)
Factor hints (class_loss=  9.1029, deps_loss= 5803.7627, fac=0.00156844)
Factor hints (class_loss=  1.2308, deps_loss= 5650.0684, fac=0.00021784)
Factor hints (class_loss=  9.0883, deps_loss= 5750.9033, fac=0.00158032)

evening
Factor hints (class_loss=  1.3447, deps_loss= 5558.3813, fac=0.00024193)
Factor hints (class_loss= 18.4624, deps_loss= 6056.8135, fac=0.00304820)
Factor hints (class_loss=  0.1400, deps_loss= 6122.2803, fac=0.00002287)
Factor hints (class_loss=  0.0526, deps_loss= 5790.4863, fac=0.00000908)
Factor hints (class_loss= 21.1384, deps_loss= 5803.6270, fac=0.00364228)
Factor hints (class_loss=  3.1123, deps_loss= 6002.5112, fac=0.00051850)
Factor hints (class_loss= 60.7331, deps_loss= 5879.1123, fac=0.01033032)
Factor hints (class_loss=  1.3860, deps_loss= 5668.1182, fac=0.00024452)
Factor hints (class_loss=  3.6776, deps_loss= 6045.1016, fac=0.00060836)
Factor hints (class_loss= 15.7819, deps_loss= 6411.4414, fac=0.00246152)
Factor hints (class_loss= 21.9232, deps_loss= 5676.2529, fac=0.00386227)
Factor hints (class_loss=  0.3627, deps_loss= 6797.3110, fac=0.00005336)
Factor hints (class_loss= 16.1702, deps_loss= 5596.6489, fac=0.00288926)
Factor hints (class_loss=  0.0677, deps_loss= 5657.2944, fac=0.00001196)
Factor hints (class_loss=  0.7334, deps_loss= 5572.8364, fac=0.00013160)
Factor hints (class_loss=  0.7809, deps_loss= 6325.0166, fac=0.00012346)
Factor hints (class_loss=  9.2523, deps_loss= 5544.1602, fac=0.00166883)

# Locally test the test script
python train_on_onestep.py --relation_hdf5=dev.1_all.hdf5 --stub=base --predict --checkpoint=./checkpoints/model-stepwise_base_0002-804000.pth

# Locally run the test script
python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=base --predict --checkpoint=./checkpoints/model-stepwise_base_0002-804000.pth

#Assessment via test_on_onestep : 
##  #precision=37.56% recall=40.10% F1=38.79%  # intersection/ans
##  precision=30.97% recall=33.06% F1=31.98%  # intersection/union of a_start_best, a_end_best

#Paper result for 'Multiple Templates' : (1_Allen_ZeroShotRelationExtraction-ReadingComprehension_1706.04115.pdf - Table 3)
##  precision=43.61% recall=36.45% F1=39.61%


# https://cloud.google.com/compute/docs/gpus/add-gpus
# Modify 'rdai-tts-p100-vm' 
#  to have 4*P100, and 
#    $0.43 USD per GPU per hour  (confirmed to have 16Gb of memory each)
#  attach a 200Gb persistent disk : 'zero-shot-relationships'
#    https://cloud.google.com/persistent-disk/

# https://cloud.google.com/tpu/docs/pricing
#   Potential : $1.35 USD per TPU per hour (preemptible)


export INSTANCE_NAME="rdai-tts-p100-vm"
gcloud compute instances start $INSTANCE_NAME

gcloud compute ssh $INSTANCE_NAME

cd ~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/
mkdir orig # ... Follow instructions above ...


gcloud compute scp *.hdf5 rdai-tts-p100-vm:~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/orig/omerlevy-bidaf_no_answer-2e9868b224e4/relation_splits/


cd ~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/orig/omerlevy-bidaf_no_answer-2e9868b224e4/relation_splits/
md5sum *
6e8fe51b869fecb6acd241df9fab07d8  dev.1_all.hdf5
de44a1feb2392ce91b03f3d7b5feba35  test.1_all.hdf5
e735c9a89f98613e2e64af0ac794bf5e  train.1_all.hdf5

# locally :
md5sum *.hdf5
6e8fe51b869fecb6acd241df9fab07d8  dev.1_all.hdf5
de44a1feb2392ce91b03f3d7b5feba35  test.1_all.hdf5
e735c9a89f98613e2e64af0ac794bf5e  train.1_all.hdf5


# Mount new persistent disk : 
#   https://devopscube.com/mount-extra-disks-on-google-cloud/
sudo lsblk
#NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
#sda      8:0    0   30G  0 disk 
#└─sda1   8:1    0   30G  0 part /
#sdb      8:16   0  200G  0 disk 


cd ~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/
mkdir checkpoints

## sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb

cd ~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships/
sudo mount -o discard,defaults /dev/sdb checkpoints
sudo chmod a+x checkpoints



. ~/env3/bin/activate

python train_on_onestep.py --relation_hdf5=dev.1_all.hdf5 --stub=base --dep_fac=0.0
# Functions (after fixing :: grep 'f"' orig/pytorch-openai-transformer-lm/*.py) 3.6 string formatting

python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=base --dep_fac=0.0

Factor hints (class_loss=29012.3672, deps_loss=23580.5684, fac=1.23035063)
Factor hints (class_loss=455.4864, deps_loss=23485.1406, fac=0.01939466)
Factor hints (class_loss=271.1743, deps_loss=23552.4355, fac=0.01151364)
Factor hints (class_loss=217.7633, deps_loss=25605.2051, fac=0.00850465)
Time used for 0.06 of epoch 0: 300.5 seconds
  Time per 1000 lines : 6.005 seconds
  Expected finish time : Wednesday, October 17, 2018 12:21:37  (server)
  Expected finish time : Wednesday, October 17, 2018 12:21:37 +08+0800 (local)
Factor hints (class_loss=176.8094, deps_loss=23848.9160, fac=0.00741373)
Factor hints (class_loss=173.7655, deps_loss=22794.8516, fac=0.00762301)
Factor hints (class_loss=119.6408, deps_loss=25449.4863, fac=0.00470111)
Factor hints (class_loss=162.4608, deps_loss=24370.5605, fac=0.00666627)


screen -S QAZRE
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac1.0 --dep_fac=1.0


# Remotely test the test script
python train_on_onestep.py --relation_hdf5=dev.1_all.hdf5 --stub=fac1.0 --predict --checkpoint=./checkpoints/model-stepwise_fac1.0_0001-760576.pth

# Remotely run the test script
python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=fac1.0 --predict --checkpoint=./checkpoints/model-stepwise_fac1.0_0001-760576.pth

export REMOTE_BASE="rdai-tts-p100-vm:~/deep-learning-workshop/notebooks/work-in-progress/2018-10_ZeroShotRelationships"
export REL_PATH="orig/omerlevy-bidaf_no_answer-2e9868b224e4/relation_splits"
## mv ${REL_PATH}/test.1_all.hdf5_base.npz ${REL_PATH}/test.1_all.hdf5_fac1.0.npz
gcloud compute scp ${REMOTE_BASE}/${REL_PATH}/test.1_all.hdf5_fac1.0.npz  ${REL_PATH}/

# Assessment via test_on_onestep : 
## precision=24.67% recall=43.13% F1=31.39%




# Next run : 
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1 --checkpoint=./checkpoints/model-stepwise_fac0.1_0000-600192.pth

# Update with 'no answer' forced labels
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1 --checkpoint=./checkpoints/model-stepwise_fac0.1_0001-400128.pth

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44                 Driver Version: 396.44                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   76C    P0   193W / 250W |  10601MiB / 16280MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  Off  | 00000000:00:05.0 Off |                    0 |
| N/A   71C    P0   187W / 250W |   8699MiB / 16280MiB |     97%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P100-PCIE...  Off  | 00000000:00:06.0 Off |                    0 |
| N/A   70C    P0   189W / 250W |   8699MiB / 16280MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P100-PCIE...  Off  | 00000000:00:07.0 Off |                    0 |
| N/A   71C    P0   203W / 250W |   8699MiB / 16280MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1374      C   python                                     10591MiB |
|    1      1374      C   python                                      8689MiB |
|    2      1374      C   python                                      8689MiB |
|    3      1374      C   python                                      8689MiB |
+-----------------------------------------------------------------------------+

# Start again with 'no answer' forced labels
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1


Saving Checkpoint : './checkpoints/model-stepwise_fac0.1_0001-760576.pth', loss_recent=2.2232
Factor hints (class_loss=199.1927, deps_loss=  617.2078, fac=0.32273209)
Time used for 0.92 of epoch 1: 4506.6 seconds
  Time per 1000 lines : 5.843 seconds
  Expected finish in : 1.47 hours
  Expected finish time : Wednesday, October 17, 2018 23:37:18  (server)
  Expected finish time : Thursday, October 18, 2018 07:37:18 +08+0800 (local)
Factor hints (class_loss=215.0347, deps_loss=  622.1494, fac=0.34563187)
Factor hints (class_loss=199.8563, deps_loss=  689.3589, fac=0.28991617)
Factor hints (class_loss=215.8770, deps_loss=  829.4865, fac=0.26025383)
Factor hints (class_loss=221.7887, deps_loss=  877.3491, fac=0.25279410)
Time used for 0.98 of epoch 1: 4807.2 seconds
  Time per 1000 lines : 5.841 seconds
  Expected finish in : 1.39 hours
  Expected finish time : Wednesday, October 17, 2018 23:37:15  (server)
  Expected finish time : Thursday, October 18, 2018 07:37:15 +08+0800 (local)
Factor hints (class_loss=206.2093, deps_loss=  751.4877, fac=0.27440149)




Saving Checkpoint : './checkpoints/model-stepwise_fac0.1_0002-800256.pth', loss_recent=1.9000


pip install ftfy, spacy
python -m spacy download en
python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=fac0.1 --predict --checkpoint=./checkpoints/model-stepwise_fac0.1_0001-839936_end-epoch.pth

gcloud compute scp ${REMOTE_BASE}/${REL_PATH}/test.1_all.hdf5_fac0.1.npz  ${REL_PATH}/

# Assessment via test_on_onestep : 
## precision=55.43% recall=19.11% F1=28.42%  # using p_ij


python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1 --checkpoint=./checkpoints/model-stepwise_fac0.1_0001-839936_end-epoch.pth --n_epoch=5

# Relevant tabs :
https://arxiv.org/pdf/1810.04805.pdf
https://arxiv.org/pdf/1803.09189.pdf
https://arxiv.org/pdf/1706.04115.pdf
http://nlp.cs.washington.edu/zeroshot/
https://bitbucket.org/omerlevy/bidaf_no_answer/src/2e9868b224e4?at=default
https://arxiv.org/pdf/1809.10185.pdf
https://arxiv.org/pdf/1611.01603.pdf
https://allennlp.org/elmo
https://arxiv.org/pdf/1802.05365.pdf
https://tfhub.dev/google/elmo/2
https://github.com/huggingface/pytorch-openai-transformer-lm
https://arxiv.org/abs/1706.04115
https://arxiv.org/pdf/1806.08730.pdf

Factor hints (class_loss=210.2790, deps_loss=  634.4976, fac=0.33141021)
Factor hints (class_loss=192.0340, deps_loss=  551.7582, fac=0.34804023)
Saving End-epoch checkpoint : './checkpoints/model-stepwise_fac0.1_0001-839936_end-epoch.pth'

python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1 --dep_fac=0.1 --checkpoint=./checkpoints/model-stepwise_fac0.1_0002-839936_end-epoch.pth --n_epoch=5



python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.02 --dep_fac=0.02

Factor hints (class_loss=223.2570, deps_loss= 1481.2102, fac=0.15072607)
Factor hints (class_loss=211.9604, deps_loss= 1401.0105, fac=0.15129106)
Factor hints (class_loss=196.1872, deps_loss= 1312.2306, fac=0.14950667)
Factor hints (class_loss=204.6235, deps_loss= 1759.9146, fac=0.11626901)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0001-160384.pth', loss_recent=2.0039

Factor hints (class_loss=234.9280, deps_loss= 1539.1934, fac=0.15263058)
Factor hints (class_loss=198.6711, deps_loss= 1532.9457, fac=0.12960086)
Factor hints (class_loss=222.6590, deps_loss= 1567.9419, fac=0.14200719)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0001-360448.pth', loss_recent=1.9098

Factor hints (class_loss=216.5104, deps_loss= 1278.1627, fac=0.16939190)
Factor hints (class_loss=202.8433, deps_loss= 1384.3292, fac=0.14652819)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0001-560512.pth', loss_recent=1.8569
Factor hints (class_loss=201.3995, deps_loss= 1333.3656, fac=0.15104597)
Factor hints (class_loss=219.0672, deps_loss= 1239.7926, fac=0.17669667)

Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0001-760576.pth', loss_recent=1.8242
Factor hints (class_loss=189.2207, deps_loss= 1184.1119, fac=0.15979971)
Factor hints (class_loss=217.4455, deps_loss= 1151.4514, fac=0.18884469)
Factor hints (class_loss=196.9379, deps_loss= 1192.1925, fac=0.16518969)

Saving End-epoch checkpoint : './checkpoints/model-stepwise_fac0.02_0001-839936_end-epoch.pth'

Factor hints (class_loss=194.4754, deps_loss= 1045.6996, fac=0.18597633)
Factor hints (class_loss=187.8143, deps_loss= 1010.2685, fac=0.18590533)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0002-120704.pth', loss_recent=1.7843
Factor hints (class_loss=199.0331, deps_loss= 1381.5564, fac=0.14406438)

Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0002-320768.pth', loss_recent=1.7356
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0002-520832.pth', loss_recent=1.7111

Saving Checkpoint : './checkpoints/model-stepwise_fac0.02_0002-720896.pth', loss_recent=1.6952
Saving End-epoch checkpoint : './checkpoints/model-stepwise_fac0.02_0002-839936_end-epoch.pth'


python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=fac0.02 --predict --checkpoint=./checkpoints/model-stepwise_fac0.02_0002-839936_end-epoch.pth

gcloud compute scp ${REMOTE_BASE}/${REL_PATH}/test.1_all.hdf5_fac0.02.npz  ${REL_PATH}/

# Assessment via test_on_onestep : 
## precision=55.43% recall=19.11% F1=28.42%  # using p_ij  #  fac0.1
## 





# Now running this with the baseno [0:2] ~ 4,3 training data...
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=baseno --dep_fac=0.0 --n_epoch=5

Factor hints (class_loss=217.6262, deps_loss=25597.0469, fac=0.00850201)
Factor hints (class_loss=195.7768, deps_loss=25347.2812, fac=0.00772378)
Factor hints (class_loss=206.4771, deps_loss=24933.8594, fac=0.00828099)
Time used for 0.85 of epoch 2: 4809.7 seconds
  Time per 1000 lines : 6.735 seconds
  Expected finish in : 3.38 hours
  Expected finish time : Friday, October 19, 2018 02:15:36 +08+0800 (local)
Factor hints (class_loss=199.8273, deps_loss=23559.3867, fac=0.00848185)
Saving Checkpoint : './checkpoints/model-stepwise_baseno_0002-720896.pth', loss_recent=1.5658
Saving Checkpoint : './checkpoints/model-stepwise_baseno_0003-081024.pth', loss_recent=1.5545
Saving Checkpoint : './checkpoints/model-stepwise_baseno_0004-441472.pth', loss_recent=1.4768
Factor hints (class_loss=204.8316, deps_loss=24427.4961, fac=0.00838529)
Factor hints (class_loss=183.4029, deps_loss=24861.5137, fac=0.00737698)
Saving Checkpoint : './checkpoints/model-stepwise_baseno_0004-641536.pth', loss_recent=1.4739
Factor hints (class_loss=184.2379, deps_loss=25274.0234, fac=0.00728961)
Factor hints (class_loss=184.8552, deps_loss=23345.1270, fac=0.00791836)

# model-stepwise_baseno_0004-641536.pth COMPLAINS, use instead : model-stepwise_baseno_0004-441472.pth

python train_on_onestep.py --relation_hdf5=test.1_all.hdf5 --stub=baseno --predict --checkpoint=./checkpoints/model-stepwise_baseno_0004-441472.pth
gcloud compute scp ${REMOTE_BASE}/${REL_PATH}/test.1_all.hdf5_baseno.npz  ${REL_PATH}/


##  Now move over to (no-ans at token_clf) training

# On GCP : 
git pull
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=base-clf-extra --dep_fac=0.0 --n_epoch=4 --extra_block
  Expected finish time : Friday, October 19, 2018 10:46:59 +08+0800 (Asia/Singapore)
Factor hints (class_loss= 42.5011, deps_loss=24650.1445, fac=0.00172417)
Saving Checkpoint : './checkpoints/model-stepwise_base-clf-extra_0000-600192.pth', loss_recent=0.6231
Factor hints (class_loss= 97.1845, deps_loss=23209.7129, fac=0.00418723)
Factor hints (class_loss= 71.8025, deps_loss=24324.6543, fac=0.00295184)
...
Factor hints (class_loss= 75.3395, deps_loss=23493.6504, fac=0.00320680)
Saving End-epoch checkpoint : './checkpoints/model-stepwise_base-clf-extra_0000-839936_end-epoch.pth'
Factor hints (class_loss= 78.9366, deps_loss=23639.1445, fac=0.00333923)
Factor hints (class_loss= 60.5935, deps_loss=23108.5254, fac=0.00262213)
Factor hints (class_loss= 53.5047, deps_loss=23190.4785, fac=0.00230718)
Factor hints (class_loss= 45.4149, deps_loss=24913.8945, fac=0.00182288)
...
Saving End-epoch checkpoint : './checkpoints/model-stepwise_base-clf-extra_0001-839936_end-epoch.pth'
Factor hints (class_loss= 55.6968, deps_loss=23727.3867, fac=0.00234736)
Factor hints (class_loss= 43.2192, deps_loss=23148.8574, fac=0.00186701)
Factor hints (class_loss= 23.4774, deps_loss=23160.7090, fac=0.00101367)
Factor hints (class_loss= 16.8954, deps_loss=25083.6582, fac=0.00067356)
Time used for 0.05 of epoch 2: 300.5 seconds
...
Saving End-epoch checkpoint : './checkpoints/model-stepwise_base-clf-extra_0002-839936_end-epoch.pth'
Factor hints (class_loss= 42.1381, deps_loss=23662.4453, fac=0.00178080)
Factor hints (class_loss= 22.2152, deps_loss=23115.2637, fac=0.00096106)
Factor hints (class_loss= 14.2607, deps_loss=23218.6602, fac=0.00061419)
Factor hints (class_loss= 16.2253, deps_loss=25105.4043, fac=0.00064629)
...
Factor hints (class_loss= 17.5010, deps_loss=23592.5371, fac=0.00074180)
Factor hints (class_loss=  4.5477, deps_loss=24901.6953, fac=0.00018262)
Saving Checkpoint : './checkpoints/model-stepwise_base-clf-extra_0003-281088.pth', loss_recent=0.1168
Factor hints (class_loss=  3.9022, deps_loss=24384.4844, fac=0.00016003)
...
Factor hints (class_loss= 22.9843, deps_loss=24748.4570, fac=0.00092872)
Factor hints (class_loss= 18.2148, deps_loss=25440.7285, fac=0.00071597)
Saving Checkpoint : './checkpoints/model-stepwise_base-clf-extra_0003-481152.pth', loss_recent=0.1085
Factor hints (class_loss=  5.0076, deps_loss=24175.3359, fac=0.00020714)
Factor hints (class_loss=  4.6461, deps_loss=24355.6562, fac=0.00019076)
Factor hints (class_loss= 52.0994, deps_loss=25632.4824, fac=0.00203255)
Saving Checkpoint : './checkpoints/model-stepwise_base-clf-extra_0003-681216.pth', loss_recent=0.1024
Factor hints (class_loss=  5.5085, deps_loss=25472.1855, fac=0.00021625)
Factor hints (class_loss=  1.7764, deps_loss=25121.1641, fac=0.00007071)
Time used for 0.85 of epoch 3: 5108.6 seconds
  Time per 1000 lines : 7.192 seconds
  Expected finish in : 0.26 hours
  Expected finish time : Friday, October 19, 2018 10:45:10 +08+0800 (Asia/Singapore)
Factor hints (class_loss= 18.1204, deps_loss=23622.1484, fac=0.00076709)
Factor hints (class_loss= 21.3947, deps_loss=24128.2852, fac=0.00088670)
Factor hints (class_loss= 12.1556, deps_loss=24368.9648, fac=0.00049882)
Time used for 0.90 of epoch 3: 5409.3 seconds
  Time per 1000 lines : 7.191 seconds
  Expected finish in : 0.18 hours
  Expected finish time : Friday, October 19, 2018 10:45:09 +08+0800 (Asia/Singapore)
Factor hints (class_loss=  3.1787, deps_loss=23763.5664, fac=0.00013376)
Factor hints (class_loss=  5.3035, deps_loss=26530.5723, fac=0.00019990)
Factor hints (class_loss= 19.4625, deps_loss=24494.5547, fac=0.00079456)
Factor hints (class_loss=  1.7563, deps_loss=23389.7383, fac=0.00007509)
Time used for 0.95 of epoch 3: 5710.0 seconds
  Time per 1000 lines : 7.189 seconds
  Expected finish in : 0.09 hours
  Expected finish time : Friday, October 19, 2018 10:45:07 +08+0800 (Asia/Singapore)
Factor hints (class_loss=  1.7182, deps_loss=24661.6191, fac=0.00006967)
Factor hints (class_loss=  6.0365, deps_loss=25655.3418, fac=0.00023529)
Factor hints (class_loss= 39.1545, deps_loss=23551.1094, fac=0.00166253)
Time used for 1.00 of epoch 3: 6010.6 seconds
  Time per 1000 lines : 7.188 seconds
  Expected finish in : 0.01 hours
  Expected finish time : Friday, October 19, 2018 10:45:06 +08+0800 (Asia/Singapore)
Saving End-epoch checkpoint : './checkpoints/model-stepwise_base-clf-extra_0003-839936_end-epoch.pth'


# On local :
git pull
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.1-clf-extra --dep_fac=0.1 --n_epoch=1 --extra_block
  Expected finish time : Friday, October 19, 2018 10:26:38 +08+0800 (Asia/Singapore)
Factor hints (class_loss= 19.5906, deps_loss=  244.4604, fac=0.08013825)
Factor hints (class_loss= 11.7347, deps_loss=  211.1650, fac=0.05557138)
Factor hints (class_loss=  9.3514, deps_loss=  199.4653, fac=0.04688250)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.1-clf-extra_0000-600096.pth', loss_recent=1.3007
Factor hints (class_loss= 21.3773, deps_loss=  199.2804, fac=0.10727230)
Factor hints (class_loss= 19.8534, deps_loss=  191.3389, fac=0.10376021)
Factor hints (class_loss=  2.9317, deps_loss=  185.7273, fac=0.01578493)
Time used for 1.00 of epoch 0: 23436.6 seconds
  Time per 1000 lines : 27.976 seconds
  Expected finish in : 0.02 hours
  Expected finish time : Friday, October 19, 2018 10:26:42 +08+0800 (Asia/Singapore)
Factor hints (class_loss= 30.0499, deps_loss=  242.6052, fac=0.12386338)
Saving End-epoch checkpoint : './checkpoints/model-stepwise_fac0.1-clf-extra_0000-839968_end-epoch.pth'


# RUNNING On GCP : 
git pull
python train_on_onestep.py --relation_hdf5=train.1_all.hdf5 --stub=fac0.02-clf-extra --dep_fac=0.02 --n_epoch=5 --extra_block
Factor hints (class_loss=128.0616, deps_loss= 2245.1643, fac=0.05703887)
Factor hints (class_loss= 76.1013, deps_loss= 2231.0840, fac=0.03410956)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02-clf-extra_0000-400128.pth', loss_recent=1.1983
Factor hints (class_loss= 58.7096, deps_loss= 2217.0830, fac=0.02648055)
Time used for 0.49 of epoch 0: 3005.3 seconds
  Time per 1000 lines : 7.240 seconds
  Expected finish in : 7.61 hours
  Expected finish time : Friday, October 19, 2018 19:29:38 +08+0800 (Asia/Singapore)
Factor hints (class_loss= 79.3488, deps_loss= 2049.4536, fac=0.03871705)
Factor hints (class_loss= 98.8306, deps_loss= 2370.9326, fac=0.04168429)
Factor hints (class_loss= 96.3353, deps_loss= 2136.8147, fac=0.04508360)
...
Factor hints (class_loss= 71.2041, deps_loss= 1708.2195, fac=0.04168322)
Factor hints (class_loss= 45.5585, deps_loss= 1169.6462, fac=0.03895065)
Saving Checkpoint : './checkpoints/model-stepwise_fac0.02-clf-extra_0001-360448.pth', loss_recent=0.5567
Factor hints (class_loss= 81.4754, deps_loss= 1168.1217, fac=0.06974903)
Factor hints (class_loss= 29.0369, deps_loss= 1348.5143, fac=0.02153248)
Factor hints (class_loss= 37.8925, deps_loss= 1938.2567, fac=0.01954981)
Factor hints (class_loss= 22.0796, deps_loss= 1345.1158, fac=0.01641465)
Time used for 0.45 of epoch 1: 2704.5 seconds
  Time per 1000 lines : 7.177 seconds
  Expected finish in : 5.95 hours
  Expected finish time : Friday, October 19, 2018 19:25:42 +08+0800 (Asia/Singapore)
Saving End-epoch checkpoint : './checkpoints/model-stepwise_fac0.02-clf-extra_0001-839936_end-epoch.pth'
Factor hints (class_loss= 60.4158, deps_loss= 1315.0813, fac=0.04594071)
Factor hints (class_loss= 38.5199, deps_loss=  984.4031, fac=0.03913019)
Factor hints (class_loss= 22.1790, deps_loss= 1378.7698, fac=0.01608610)


{'fraction_non_alphanumeric': 0.1411083683397572, 'fraction_numerical': 0.16342586577743132, 'mean_word_length': 3.9573493631029053, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 60, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 8, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5563595', 'n_tokens_mistral': 33629, 'n_tokens_neox': 26398, 'n_words': 5569}