<jupyter_start><jupyter_text>Getting started on Colab Colab is a hosted python notebook service, based on Jupyter, that provides free access to computing resources (including GPUs and TPUs). Colab allows for quick and easy sharing with collaborators without requiring downloads or python environment setup.  Executing Python Code<jupyter_code><jupyter_output><empty_output><jupyter_text>Colab environment comes with a number of pre-installed  scientific and machine learning packages such as numpy, scipy, pandas, tensorflow, and pytorch.We can check the installed python packages using `pip freeze`<jupyter_code><jupyter_output><empty_output><jupyter_text>Add code snippets To add pre-written code snippets (useful for plotting), open the code snippets tab- Insert -> Code Snippets  Runtimes Each Colab instance (runtime) runs on an individual Virtual Machine(VM).  To view active sessions, select Runtime > Manage Sessions. Here you will be able to view all your open notebooks (and terminate them). ---By Default, Colab instances have access to CPUs, however notebooks have access to GPU and TPU resources.  To initialize a runtime with a GPU, 1.   Select 'Runtime' > 'Change Runtime Type'2.   Select 'GPU' in the drop down menu and click 'Save' We'll test the GPU using the [example](https://colab.research.google.com/notebooks/gpu.ipynb) provided by colab.<jupyter_code>%tensorflow_version 2.x
import tensorflow as tf
import timeit

device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  print(
      '\n\nThis error most likely means that this notebook is not '
      'configured to use a GPU.  Change this in Notebook Settings via the '
      'command palette (cmd/ctrl-shift-P) or the Edit menu.\n\n')
  raise SystemError('GPU device not found')

def cpu():
  with tf.device('/cpu:0'):
    random_image_cpu = tf.random.normal((100, 100, 100, 3))
    net_cpu = tf.keras.layers.Conv2D(32, 7)(random_image_cpu)
    return tf.math.reduce_sum(net_cpu)

def gpu():
  with tf.device('/device:GPU:0'):
    random_image_gpu = tf.random.normal((100, 100, 100, 3))
    net_gpu = tf.keras.layers.Conv2D(32, 7)(random_image_gpu)
    return tf.math.reduce_sum(net_gpu)
  
# We run each op once to warm up; see: https://stackoverflow.com/a/45067900
cpu()
gpu()

# Run the op several times.
print('Time (s) to convolve 32x7x7x3 filter over random 100x100x100x3 images '
      '(batch x height x width x channel). Sum of five runs.')
print('CPU (s):')
cpu_time = timeit.timeit('cpu()', number=5, setup="from __main__ import cpu")
print(cpu_time)
print('GPU (s):')
gpu_time = timeit.timeit('gpu()', number=5, setup="from __main__ import gpu")
print(gpu_time)
print('GPU speedup over CPU: {}x'.format(int(cpu_time/gpu_time)))<jupyter_output>Time (s) to convolve 32x7x7x3 filter over random 100x100x100x3 images (batch x height x width x channel). Sum of five runs.
CPU (s):
1.650525636999987
GPU (s):
0.018746201999988443
GPU speedup over CPU: 88x<jupyter_text>Install Conda using conda-colabGoogle Colab does contain a pre-installed version of conda. Conda is a package manager and environment manager.This will allow you to use conda to install packages<jupyter_code><jupyter_output><empty_output><jupyter_text>To simplify the conda installation, we use [condacolab](https://pypi.org/project/condacolab/).<jupyter_code><jupyter_output><empty_output><jupyter_text>After running condacolab.install(), conda and mamba are installed. Conda and Mamba are functionally equivalent, however conda is written in python, whilst Mamba is written in C. In our lesson, we use Mamba to leverage its speed (as compared to Conda). However, we note that Mamba can be more prone to bugs, as it is relatively new and less popular.<jupyter_code><jupyter_output><empty_output><jupyter_text>For compatibility with colab, we will need to ensure the numpy version is unchanged when we install packages (such as pymatgen).To identify the numpy version factory installed on colab:<jupyter_code><jupyter_output><empty_output><jupyter_text>The current version of numpy is v1.19.5, thus we specify mamba to install this specific version.<jupyter_code><jupyter_output><empty_output><jupyter_text>Setup environment using conda constructor.---1.   First, we want to remove our previous installations by resetting our runtime to factory defaults. This process will revert the environment to its original state, removing any added packages or restoring deleted system files.  *   Runtime > Factory reset runtime 2.   Next, we install conda colab<jupyter_code><jupyter_output><empty_output><jupyter_text>3.   Finally, we install using the constructor we have built for the [workshop](https://github.com/materialsproject/workshop)  Link to the [release](https://github.com/materialsproject/workshop/releases/tag/2021.08.09)  - https://github.com/materialsproject/workshop/releases/download/2021.08.09/condacolab-0.1-Linux-x86_64.sh<jupyter_code><jupyter_output><empty_output><jupyter_text>The environment is now setup with pymatgen, fireworks, custodian, atomate, and matminer.  Building a conda constructorTo build your own environment constructor (to save time when restarting notebook)Adapted from this [notebook](https://colab.research.google.com/github/jaimergp/condacolab/blob/main/constructor-example/condacolab_constructor_tutorial.ipynbscrollTo=-RDpyAYfXk9L) 1. Download the templates constructor<jupyter_code>!wget -q https://raw.githubusercontent.com/materialsproject/workshop/master/workshop/primer/04_Python_environments/env_installer/construct.yaml
!wget -q https://raw.githubusercontent.com/materialsproject/workshop/master/workshop/primer/04_Python_environments/env_installer/pip-dependencies.sh<jupyter_output><empty_output><jupyter_text>2. Modify the construct.yaml as desired. Here, we will add:   - numpy 1.19.5  - pymatgen  - custodian  - fireworks  - atomate  - matminer 3. Install condacolab, conda, mamba, and constructor<jupyter_code>!pip install -q condacolab

# Install conda and mamba
import condacolab
condacolab.install()

# Install constructor
!mamba install -q constructor<jupyter_output>⏬ Downloading https://github.com/jaimergp/miniforge/releases/latest/download/Mambaforge-colab-Linux-x86_64.sh...
📦 Installing...
📌 Adjusting configuration...
🩹 Patching environment...
⏲ Done in 0:00:38
🔁 Restarting kernel...
  Package             Version  Build           Channel                    Size
────────────────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────────────────

[32m  conda-standalone[00m     4.10.3  ha770c72_0      conda-forge/linux-64       9 MB
[32m  constructor     [00m      3.2.1  py37h89c1867_1  conda-forge/linux-64     162 KB

  Upgrade:
────────────────────────────────────────────────────────────────────────────────

[31m  ca-certificates [00m  2020.12.5  ha878542_0      installed                      
[32m  ca-certificates [00m  2021.5.30  ha878542_0      conda-forge/linux-64     136 KB
[31m  certifi         [00m  2020.12.5  py37h89c1867_1  installed      [...]<jupyter_text>5. Run constructor to build .sh template file<jupyter_code>!constructor .<jupyter_output>platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
fetching: python-3.7.0-hd21baee_1006.tar.bz2
fetching: _libgcc_mutex-0.1-conda_forge.tar.bz2
fetching: ca-certificates-2021.5.30-ha878542_0.tar.bz2
fetching: libgfortran4-7.5.0-h14aa051_19.tar.bz2
fetching: libstdcxx-ng-11.1.0-h56837e0_5.tar.bz2
fetching: libgfortran-ng-7.5.0-h14aa051_19.tar.bz2
fetching: libgomp-11.1.0-hc902ee8_5.tar.bz2
fetching: _openmp_mutex-4.5-1_gnu.tar.bz2
fetching: libgcc-ng-11.1.0-hc902ee8_5.tar.bz2
fetching: bzip2-1.0.8-h7f98852_4.tar.bz2
fetching: expat-2.4.1-h9c3ff4c_0.tar.bz2
fetching: gmp-6.2.1-h58526e2_0.tar.bz2
fetching: icu-68.1-h58526e2_0.tar.bz2
fetching: jpeg-9d-h36c2ea0_0.tar.bz2
fetching: jsoncpp-1.8.4-hc9558a2_1002.tar.bz2
fetching: libffi-3.2.1-he1b5a44_1007.tar.bz2
fetching: libiconv-1.16-h516909a_0.tar.bz2
fetching: libopenblas-0.3.12-pthreads_hb3c22a3_1.tar.bz2
fetching: libsodium-1.0.18-h36c2ea0_1.tar.bz2
fetching: libuuid-2.32.1-h7f98852_1000.tar.[...]<jupyter_text>5. Download .sh constructor and upload to any file hosting site (github releases can be used)<jupyter_code>from google.colab import files
installer = !ls *-Linux-x86_64.sh
files.download(installer[0])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1258575822096049, 'fraction_numerical': 0.056186420629287914, 'mean_word_length': 5.000709723207949, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 58, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '315675', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3235, 'n_tokens_neox': 2836, 'n_words': 755, 'path': 'workshop/primer/04_Python_environments/2_google_colab_blank.ipynb', 'repo_name': 'materialsproject/workshop', 'size': '8454'}