@inproceedings{
gruver2023finetuned,
title={Fine-Tuned Language Models Generate Stable Inorganic Materials as Text},
author={Nate Gruver and Anuroop Sriram and Andrea Madotto and Andrew Gordon Wilson and C. Lawrence Zitnick and Zachary Ward Ulissi},
booktitle={AI for Accelerated Materials Design - NeurIPS 2023 Workshop},
year={2023},
url={https://openreview.net/forum?id=0r5DE2ZSwJ}
}
Deep learning models have drastically accelerated materials discovery by accelerating predictive computational simulations like density functional theory (DFT). Large open computational materials databases such as the Materials Project or OQMD contain O($10^6$) known structures, and it is now straightforward to search those databases for materials with exciting properties. However, these databases are limited to experimentally known materials or candidates discovered in high-throughput computational campaigns. Many state-of-the-art engineering advances in  solar photovaltaics, battery electrodes, and catalysts are made by discovering materials with outstanding properties that have not yet been discovered. Generative models are a natural solution to expand families of interest through sampling. While popular methods are typically constructed from variational autoencoders or diffusion models, we propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable, with around 90\% of sampled structures obeying physical constraints on atom positions and charges. Using energy of hull calculations from both learned ML potentials and gold-standard DFT calculations, we show that our strongest model (fine-tuned  LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49\% vs 28\%) of CDVAE, a competing diffusion model. Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.
Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
Nate Gruver Anuroop Sriram Andrea Madotto Andrew Gordon Wilson C. Lawrence Zitnick Zachary Ward Ulissi