<jupyter_start><jupyter_text>JCAP images & spectra hackathon Intro and motivationThis notebook gives an introduction to the dataset used in the research presented in Stein et al. Chem. Sci., 2019, 10, 47-55 https://doi.org/10.1039/C8SC03077D to generate embeddings from printed images and use them to predict full UV-Vis absorption spectra. The bigger picture for this scientific endeavour is the quest to find materials that are able to split water under irradiation of sunlight (solar water splitting) to generate renewable "solar" fuels. The scientists in the high-throughput experiementation group (HTE) at the Joint Center for Artificial Photosynthesis (JCAP - https://solarfuelshub.org/) therefore utilize ink-jet printing to deposit metal nitrate precursors onto transparent conducting oxides (TCO) that are annealed at high temperatures to form metal oxides. The images in this study (already encoded using a neural network in the dataset here) are optical microscopy images from a consumer grade photo scanner that is able to scan tens of thousands of materials images (ca. 4mm by 4mm) a minute. The information in these images is classic RGB CMOS data. Being able to tell from this very low-cost measurement how the full absorption spectra would qualitatively and quantitaively look like would not just enable the scientists to accelerate their discovery process but to also allow them to use the giant amount of materials images that have never been measured using UV-Vis spectroscopy.The measurement workflow is shown below. Technical introduction / reference**THIS IS FOR REFERENCE, THERE IS NO NEED TO DOWNLOAD FROM THE EXTERNAL SITE** In their publication Stein et al. used a convolutional variational autoencoder to generate image embeddings, that for the purpose of this notebook, shall function as a surrogate representation of the images. What you have in this notebook is the embeddings for a subset of the over 100k images described by Stein et al. in https://doi.org/10.1038/s41597-019-0019-4 that can be downloaded at https://data.caltech.edu/records/1152 with code to wrangle with the dataset available at https://github.com/helgestein/materials-images-spectra. The embeddings for images are already generated for you as this process requires training of a larger deep neural network and takes about 30 min to complete. **For the bandgap prediction a word of precaution should be made:** We calculate the bandgap only for those spectra where reasonable values were found for both the predicted and regressed bandgaps. The used algorithm for bandgap extraction is not very stable and can predict bandgaps outside a physically meaningful range. Also, we know that most materials are multiphase thus exhibit more than one bandgap. The algorithm finds the most dominant one.  Load packages and files<jupyter_code>%matplotlib inline
import os
import boto3
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from tqdm import tqdm<jupyter_output><empty_output><jupyter_text>Download files to SageMaker instance from S3.<jupyter_code>jcap_pickle = 'jcap_optical_encoding.pickle'
energy_npy = 'energy_ev.npy'
def hook(t):
  def inner(bytes_amount):
    t.update(bytes_amount)
  return inner
#session = boto3.session.Session(profile_name='hack2020')
#s3 = session.client('s3')
s3 = boto3.client('s3')
for filename in [jcap_pickle, energy_npy]:
    filesize = boto3.resource('s3').Object('hackathon2020-prod', 'data/' + filename).content_length
    with tqdm(total=filesize, unit='B', unit_scale=True, desc=jcap_pickle) as t:
        s3.download_file('hackathon2020-prod', 'data/' + filename, filename, Callback=hook(t))<jupyter_output>jcap_optical_encoding.pickle: 100%|██████████| 154M/154M [00:01<00:00, 151MB/s]  
jcap_optical_encoding.pickle: 100%|██████████| 1.01k/1.01k [00:00<00:00, 30.4kB/s]<jupyter_text>Load local files into arrays or Pandas dataframe.<jupyter_code>energy_ev = np.load(energy_npy)
jcap_df = pd.read_pickle(jcap_pickle)
jcap_df.shape<jupyter_output><empty_output><jupyter_text>Look at the first few rows of the dataframe.<jupyter_code>jcap_df.head()<jupyter_output><empty_output><jupyter_text>The dataframe is indexed with an entry id associated with a unique composition. The bandgap, 100-dimensional encoding from variational autoencoder in the reference, the optical-UV spectrum, and the atomic fraction of each element are listed for each entry.<jupyter_code>jcap_df.columns<jupyter_output><empty_output><jupyter_text>Here we plot a histogram of all the bandgaps over the entire dataset.<jupyter_code>jcap_df['bandgap'].plot.hist(bins=40,range=[0,3])
plt.xlabel('bandgap [eV]')
plt.savefig('bandgap_hist.png')
jcap_df['bandgap'].plot.hist(bins=40,range=[0,1.2])
plt.xlabel('bandgap [eV]')
plt.savefig('bandgap_hist_zoom.png')<jupyter_output><empty_output><jupyter_text>Here we inspect the properties of a single entry (106). The optical-UV spectrum is shown over the range in array `energy_ev`, a histogram of coefficients from the encoding is plotted, and we print the elements with non-zero atomic fraction.<jupyter_code>entry_id = 101
plt.plot(energy_ev, jcap_df.loc[entry_id, 'spectrum'])
plt.show()
plt.hist(jcap_df.loc[entry_id, 'vae'], bins=40)
plt.show()
for col in jcap_df.columns:
    if len(col) > 2:
        continue
    if jcap_df.loc[entry_id, col] > 1e-6:
        print(entry_id, col, jcap_df.loc[entry_id, col])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07932115845784911, 'fraction_numerical': 0.02545655783065855, 'mean_word_length': 5.652760736196319, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '69804', 'license': 'Apache-2.0', 'n_tokens_mistral': 1728, 'n_tokens_neox': 1550, 'n_words': 625, 'path': 'tutorial/datasets/jcap.ipynb', 'repo_name': 'lerandc/tri-hackathon-2020-cobalt', 'size': '5421'}