<jupyter_start><jupyter_text>DelCloseBotGoal is to make a DelCloseBot using old youtube footage. Will have live actor filmed with webcam during show whose pose gets transformed into real time DelClose Poses shown to audience, with voice impressonation reading lines scripted from dialogue AI. *   Using https://github.com/datitran/face2face-demo*   youtube video here https://www.youtube.com/watch?v=DjbQaSPPsqA*   saved into .mp4 with https://www.onlinevideoconverter.com/youtube-converter Instructions:* Connect colab to google drive * save the .mp4 video into a folder, * download the facial landmark model from http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2* after uncompressing locally and saving to google drive Tactics:* Convert video into side by side images of video and face structure* Train model on these side by side images to predict DelClose from Face pose* Use live webam to get net face structure* Convert new face structure to DelClose pose with model in real time  Prepare Environment * Instal git repo* Install anaconda and setup to use* Create conda environment and activate it* Install some needed packages>   ISSUE - dlib install takes about 10 minutes to compile, find way to speed up?* Mount your google drive to load and save from > Note you have to follow the link here and get the activation code before proceeding<jupyter_code>#mount google drive to load own images, follow link and get code and paste in below
from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)

!git clone https://github.com/datitran/face2face-demo.git --recursive
# Clone the repo from <NAME>'s pix2pix TensorFlow implementation
!git clone https://github.com/affinelayer/pix2pix-tensorflow.git

!wget -c https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
!chmod +x Anaconda3-5.1.0-Linux-x86_64.sh
!bash ./Anaconda3-5.1.0-Linux-x86_64.sh -b -f -p /usr/local

import sys
sys.path.append('/usr/local/lib/python3.6/site-packages/')

!conda env create -f face2face-demo/environment.yml 
#not sure this actually works in colab:
!source activate face2face-demo
#since I have to do this after:
!pip install opencv-python
!pip install imutils==0.4.3
!pip install tensorflow==1.2.1<jupyter_output>--2019-01-27 22:26:32--  https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
Resolving repo.continuum.io (repo.continuum.io)... 172.16.31.10, 192.168.127.12, fdf8:f53e:61e4::18:130a, ...
Connecting to repo.continuum.io (repo.continuum.io)|172.16.31.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 577996269 (551M) [application/x-sh]
Saving to: ‘Anaconda3-5.1.0-Linux-x86_64.sh’

Anaconda3-5.1.0-Lin 100%[===================>] 551.22M  99.2MB/s    in 5.1s    

2019-01-27 22:26:38 (109 MB/s) - ‘Anaconda3-5.1.0-Linux-x86_64.sh’ saved [577996269/577996269]

PREFIX=/usr/local
installing: python-3.6.4-hc3d631a_1 ...
Python 3.6.4 :: Anaconda, Inc.
installing: ca-certificates-2017.08.26-h1d4fec5_0 ...
installing: conda-env-2.6.0-h36134e3_1 ...
installing: intel-openmp-2018.0.0-hc7b2577_8 ...
installing: libgcc-ng-7.2.0-h7cc24e2_2 ...
installing: libgfortran-ng-7.2.0-h9f7466a_2 ...
installing: libstdcxx-ng-7.2.0-h7a57d05_2 ...
installing: bzip2-1.0.6-h9a117a8_4[...]<jupyter_text>only if doing stimulus generation<jupyter_code>!pip install dlib<jupyter_output><empty_output><jupyter_text>Processes the source video to find frames with faces, and pull out the face structure, save side by side image<jupyter_code>print('Generating faces from video')
!python face2face-demo/generate_train_data.py \
  --file /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/DelCloseMono.mp4 \
  --num 4000 \
  --landmark-model /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/shape_predictor_68_face_landmarks.dat

print('Moving Images to Google Drive')
# Move the original and landmarks folder into the google drive folder
!mkdir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos
!cp -r landmarks -r original /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos

print('Resizing Images')
# Resize original images
!python pix2pix-tensorflow/tools/process.py \
  --input_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/original \
  --operation resize \
  --output_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/original_resized
  
# Resize landmark images
!python pix2pix-tensorflow/tools/process.py \
  --input_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/landmarks \
  --operation resize \
  --output_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/landmarks_resized

print('Combining Images')
# Combine both resized original and landmark images
!python pix2pix-tensorflow/tools/process.py \
  --input_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/landmarks_resized \
  --b_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/original_resized \
  --operation combine \
  --output_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/combined
  
print('Spliting into training and val')
# Split into train/val set
!python pix2pix-tensorflow/tools/split.py \
  --dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/combined<jupyter_output>Generating faces from video
Traceback (most recent call last):
  File "face2face-demo/generate_train_data.py", line 96, in <module>
    predictor = dlib.shape_predictor(args.face_landmark_shape_file)
KeyboardInterrupt
Moving Images to Google Drive
mkdir: cannot create directory ‘/content/gdrive/My Drive/DeepLearning/DelCloseBot/photos’: File exists
Resizing Images
python: can't open file 'pix2pix-tensorflow/tools/process.py': [Errno 2] No such file or directory
python: can't open file 'pix2pix-tensorflow/tools/process.py': [Errno 2] No such file or directory
Combining Images
python: can't open file 'pix2pix-tensorflow/tools/process.py': [Errno 2] No such file or directory
Spliting into training and val
python: can't open file 'pix2pix-tensorflow/tools/split.py': [Errno 2] No such file or directory<jupyter_text>Train the model for the first time<jupyter_code># Train the model on the data for the first time
!python pix2pix-tensorflow/pix2pix.py \
  --mode train \
  --output_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/face2face-model \
  --max_epochs 10000 \
  --input_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/combined/train \
  --which_direction AtoB \
  --display_freq 50 \
  --save_freq 200<jupyter_output>aspect_ratio = 1.0
batch_size = 1
beta1 = 0.5
checkpoint = None
display_freq = 50
flip = True
gan_weight = 1.0
input_dir = /content/gdrive/My Drive/DeepLearning/DelCloseBot/photos/combined/train
l1_weight = 100.0
lab_colorization = False
lr = 0.0002
max_epochs = 10000
max_steps = None
mode = train
ndf = 64
ngf = 64
output_dir = /content/gdrive/My Drive/DeepLearning/DelCloseBot/face2face-model
output_filetype = png
progress_freq = 50
save_freq = 200
scale_size = 286
seed = 230210217
separable_conv = False
summary_freq = 100
trace_freq = 0
which_direction = AtoB
examples count = 1719
2019-01-26 05:22:56.547765: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-26 05:22:56.547825: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine [...]<jupyter_text>Train the model from checkpoint<jupyter_code># Train the model on the data from a checkpoint
!python pix2pix-tensorflow/pix2pix.py \
  --mode train \
  --output_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/face2face-model \
  --max_epochs 200 \
  --input_dir /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/photos/combined/train \
  --which_direction AtoB \
  --display_freq 50 \
  --checkpoint /content/gdrive/My\ Drive/DeepLearning/DelCloseBot/face2face-model \
  --save_freq 200<jupyter_output>aspect_ratio = 1.0
batch_size = 1
beta1 = 0.5
checkpoint = /content/gdrive/My Drive/DeepLearning/DelCloseBot/face2face-model
display_freq = 50
flip = True
gan_weight = 1.0
input_dir = /content/gdrive/My Drive/DeepLearning/DelCloseBot/photos/combined/train
l1_weight = 100.0
lab_colorization = False
lr = 0.0002
max_epochs = 200
max_steps = None
mode = train
ndf = 64
ngf = 64
output_dir = /content/gdrive/My Drive/DeepLearning/DelCloseBot/face2face-model
output_filetype = png
progress_freq = 50
save_freq = 200
scale_size = 286
seed = 699686393
separable_conv = False
summary_freq = 100
trace_freq = 0
which_direction = AtoB
examples count = 320
2019-01-14 03:18:18.719988: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-14 03:18:18.720049: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE[...]<jupyter_text>Now we export the model, by reducing and freezing it<jupyter_code>#Reduce the model to save space during webcam inference
!python face2face-demo/reduce_model.py \
  --model-input gdrive/My\ Drive/DeepLearning/DelCloseBot/face2face-model \
  --model-output gdrive/My\ Drive/DeepLearning/DelCloseBot/face2face-model-reduced

#Freeze the Reduced Model |saves a file frozen_model.pb, download this or use one in link below|
!python face2face-demo/freeze_model.py --model-folder gdrive/My\ Drive/DeepLearning/DelCloseBot/face2face-model-reduced<jupyter_output>2019-01-27 22:33:00.045876: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-27 22:33:00.045929: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-27 22:33:00.045946: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2019-01-27 22:33:00.045970: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-27 22:33:00.046000: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't[...]<jupyter_text>Instructions to run locally with webcam Now run the demo - need to download model and do localdownload the face predictor model here:http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2download frozen model above or this one:https://drive.google.com/open?id=1DkGbtIGmEdoOB_Ar4e3Tem8j9V7MemGbSave them and modify the locations shown belowopen terminalNote that you will create a conda environment, with an older version of tensor flow that needs to match the one used above to generate the model, errors can occur if the versions do not match first time* show 0 to show the live webcam too, * show 2 to show large output image only, * show 1 is face tracking```git clone https://github.com/kylemath/face2face-demo.gitcd face2face-democonda env create -f environment.ymlsource activate face2face-demopython run_webcam_only.py \  --source 0 \  --show 0 \  --landmark-model /Users/kylemathewson/face2face-demo/shape_predictor_68_face_landmarks.dat \  --tf-model /Users/kylemathewson/face2face-demo/frozen_model.pb  ``` subsequent times* show 0 to show the live webcam too, * show 2 to show large output image only, * show 1 is face tracking```cd face2face-demosource activate face2face-demopython run_webcam_only.py \  --source 0 \  --show 2 \  --landmark-model /Users/kylemathewson/face2face-demo/shape_predictor_68_face_landmarks.dat \  --tf-model /Users/kylemathewson/face2face-demo/frozen_model.pb  ```<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09954413593037713, 'fraction_numerical': 0.056029838375466226, 'mean_word_length': 4.69419537517697, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 29, 'https://': 9, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '332218', 'license': 'MIT', 'n_tokens_mistral': 4463, 'n_tokens_neox': 3942, 'n_words': 1114, 'path': 'notebooks/DelCloseBot.ipynb', 'repo_name': 'kylemath/Apparition', 'size': '12071'}