<jupyter_start><jupyter_text>Introduction¶This notebook demonstrates how to use the defects modules within Pymatgen and PyCDT v2.0.Written using:- pymatgen==2019.5.1, pycdt==2.0.0Relevant links:      - http://pymatgen.org      - https://github.com/materialsproject/pymatgen      - https://bitbucket.org/mbkumar/pycdt/src/master/Author: <NAME> (last updated: 5/8/19)This Jupyter notebook is not intended to teach one about the physics of point defect calculations. It instead intends to show how one can use the tools within the above mentioned codes to expedite the performance of these calculations for an intelligent researcher. For references on how to perform defect calculations, we suggest:- <NAME> et al. (2018): https://doi.org/10.1016/j.cpc.2018.01.004- <NAME> et al. (2017): https://doi.org/10.1016/j.commatsci.2016.12.040- <NAME> et al. (2014): https://doi.org/10.1103/RevModPhys.86.253- S. Lany et al. (2008): https://doi.org/10.1103/PhysRevB.78.235104- <NAME> et al. (2004): https://doi.org/10.1063/1.1682673  Pymatgen defect-related code Many features that originally existed within the Python Charged Defect Toolkit (PyCDT) now exist in pymatgen. While working with many of these features is useful for writing your own code, you may find that using Atomate or PyCDT scripts expedite the process of working with these defect modules. Regardless, it is still important to understand the fundamental building blocks of the pymatgen Defects module for writing your own code.  Example P1: Creating and using Defect objects (pymatgen) Pymatgen has several defect object types available for use: Vacancy, Substitution, and Interstitial. Substitutions is used for both intrinsic (antisites) and extrinsic (dopants) types of defects. Note that defect-complexes are not currently possible to execute within pymatgen. It is possible to manually create Defect objects or to generate them using pymatgen's defect generator classes. Generators are particularly useful when dealing with Interstitial sites, which can occasionally be difficult to identify. There are multiple possible routines available for interstitial generation (InFit, Voronoi, and Midpoint) as well as a MasterInterstitialGenerator which considers all the generation methods simultaneously. There is also a SimpleChargeGenerator routine which can be used for very simplistic guesses of charge states to consider.<jupyter_code># manually create defect objects and generate their supercell structures
from pymatgen.analysis.defects.core import Vacancy, Substitution, Interstitial
from pymatgen.io.vasp import Poscar
from pymatgen import MPRester, PeriodicSite

with MPRester() as mp:
    bulk_structure = mp.get_structure_by_material_id('mp-661') #AlN wurtzite structure 
    
    
defect_site = PeriodicSite(bulk_structure.sites[0].specie, bulk_structure.sites[0].frac_coords, 
                           bulk_structure.lattice)
vac = Vacancy(bulk_structure, defect_site, charge=-3)
print("Creating Defect {} in charge state {}:".format( vac.name, vac.charge))
print(vac.generate_defect_structure( supercell=(2, 1, 1)))


defect_site = PeriodicSite("Be", 
                           #note Be is the specie to sit on the final site in the defect structure
                           bulk_structure.sites[0].frac_coords, 
                           bulk_structure.lattice)
sub = Substitution(bulk_structure, defect_site, charge=-2)
print("\nCreated Defect {} in charge {}:".format( sub.name, sub.charge))
print(sub.generate_defect_structure( supercell=(2, 1, 1)))


defect_site = PeriodicSite("H", [0.666, 0.333, 0.69], 
                           bulk_structure.lattice)
inter = Interstitial(bulk_structure, defect_site, charge=0.)
print("\nCreated Defect {} in charge {}:".format( inter.name, inter.charge))
print(inter.generate_defect_structure( supercell=(2, 1, 1)))
# use generators to create defects. Note that generators always create neutral 
# defects which can be charged after being created
from pymatgen.analysis.defects.generators import VacancyGenerator, SubstitutionGenerator, \
                                                VoronoiInterstitialGenerator

print('Vacancy Generator:')
for vac_defect in VacancyGenerator( bulk_structure):
    print("\tCreated Defect {} at site {}".format( vac_defect.name, vac_defect.site))
    
print('Substitution Generator:')
for sub_defect in SubstitutionGenerator( bulk_structure, 'Be'):
    print("\tCreated Defect {} at site {}".format( sub_defect.name, sub_defect.site))
    
print('Voronoi Generator:')
for inter_defect in VoronoiInterstitialGenerator( bulk_structure, 'H'):
    print("\tCreated Defect {} at site {}".format( inter_defect.name, inter_defect.site))
# use SimpleChargeGenerator to consider a simple set of charges to run for a given defect
from pymatgen.analysis.defects.generators import VacancyGenerator, SimpleChargeGenerator

defects_to_run = []
for vac_defect in VacancyGenerator( bulk_structure):
    print("Created Defect {}".format( vac_defect.name))
    for charged_defect in SimpleChargeGenerator( vac_defect):
        print("\tcreated defect with charge {}".format( charged_defect.charge))
        defects_to_run.append( charged_defect.copy())
# note that multiplicites are automatically determined based 
# on the number of equivalent sites in the bulk_structure's sublattice

for defect in [vac, sub, inter]:
    print("{} chg {} has multiplicity = {}".format(defect.name, defect.charge, defect.multiplicity))<jupyter_output><empty_output><jupyter_text>One advantage of using pymatgen Defect objects is the ability to compare equivalent defects which do not have identical cartesian coordinates. This comes in useful when comparing defects with different cell sizes or defects created at different sites on an indentical sublattice. This comparison is done via the PointDefectComparator module.<jupyter_code># compare equivalent defect objects
from pymatgen.analysis.structure_matcher import PointDefectComparator

# same sublattice, different cartesian coordinates, different charge (with check_charge = False)
vac_1 = Vacancy(bulk_structure, bulk_structure.sites[0], charge=2.)
vac_2 = Vacancy(bulk_structure, bulk_structure.sites[1], charge=1.)

pdc = PointDefectComparator( check_charge=False, check_primitive_cell=False,
                             check_lattice_scale=False)
print("(1) Are these defects equivalent? {}".format( pdc.are_equal( vac_1, vac_2)))

# same site, different charge (with check_charge = True)
vac_3 = vac_1.copy()
vac_3.set_charge(0.)
pdc = PointDefectComparator( check_charge=True)
print("(2) Are these defects equivalent? {}".format( pdc.are_equal( vac_1, vac_3)))

# same sublattice, different coordinates, different lattice (with check_lattice_scale = False/True)
scaled_bulk_struct = bulk_structure.copy()
scaled_bulk_struct.scale_lattice(bulk_structure.volume * 0.95)
vac_4 = Vacancy(scaled_bulk_struct, scaled_bulk_struct.sites[1], charge=2.)
pdc = PointDefectComparator( check_lattice_scale=False)
print("(3) Are these defects equivalent? {}".format( pdc.are_equal( vac_1, vac_4)))

pdc = PointDefectComparator( check_lattice_scale=True)
print("(4) Are these defects equivalent? {}".format( pdc.are_equal( vac_1, vac_4)))<jupyter_output><empty_output><jupyter_text>Example P2: Working with DefectEnergy objects (pymatgen) DefectEntry objects are  similar to Pymatgen ComputedEntry objects, but with additional features useful for Defect analysis. The minimum requirement for a DefectEntry is a defect object and the uncorrected energy (defect supercell energy - bulk supercell energy).<jupyter_code># create Defect Entry
from pymatgen.analysis.defects.core import DefectEntry

defect_energy = -225.286
bulk_energy = -240.237
parameters = {"vbm": 1.2, "band_gap": 4.039, "scaling_matrix": [3.,3.,3.]}
corrections = {} #if corrections exist, then one can import them as a dictionary to the DefectEntry
uncorrected_energy = defect_energy - bulk_energy
dentry = DefectEntry(vac, uncorrected_energy, 
                     corrections=corrections, parameters=parameters)
print("Defect Entry created for {} chg {} with uncorrected energy {}".format( dentry.name, dentry.charge, 
                                                                             dentry.uncorrected_energy))<jupyter_output><empty_output><jupyter_text>In order to intepret the thermodynamic energetics of the DefectEntry, it is neccessary to have chemical potentials. At the most basic level, this requires a PhaseDiagram to be used. If the Defect calculations were performed with the same POTCAR and identical INCAR settings to those used in the Materials Project database, then the following routine can be used to grab the chemical potentials:<jupyter_code># grab chemical potentials, using the standard PhaseDiagram method 
from pymatgen.analysis.phase_diagram import PhaseDiagram

red_comp = dentry.bulk_structure.composition.reduced_composition
elt_set = list(dentry.bulk_structure.symbol_set)
with MPRester() as mp:
    pd_ents = mp.get_entries_in_chemsys( elt_set)
    
pd = PhaseDiagram( pd_ents)
cp_dict = pd.get_all_chempots( red_comp)

print('\nResulting chemical potentials:')
for k,v in cp_dict.items():
      print("\t{}: {}".format(k,v))
# get formation energy at the valence band minimum, given a dictionary of chemical potentials

for region, cps in cp_dict.items():
    form_en = dentry.formation_energy( chemical_potentials=cps, fermi_level=0)
    print("At the VBM, for chemical potentials given by {}"
          "\n\t{} chg {} formation energy is {}".format(region, dentry.name, dentry.charge, form_en))
# get defect concentrations as a function of temperature, given the chemical potentials,
# and plot as a function of temperature.
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

try:
    cps = cp_dict['Al-AlN']
except:
    cps = cp_dict["AlN-Al"]
    
TempRange = np.arange(300, 1001, 100)
ConcenRange = []
for T in TempRange:
    ConcenRange.append( dentry.defect_concentration( cps, temperature=T, fermi_level=0))

print("Plot concentrations of {} charge {}\n (assuming fermi level"
      " is at the VBM):".format( dentry.name, dentry.charge))
plt.semilogy(TempRange, ConcenRange)
plt.xlabel("Temperature [K]")
plt.ylabel("Defect Concentration [cm$^{-3}$]")
plt.show()<jupyter_output><empty_output><jupyter_text>Example P3: Run finite-size corrections (pymatgen) The defect formation energy reflects the energy to create a single point defect in an infinite bulk system (otherwise known as the 'dilute limit'). Since most DFT calculations are performed with periodic boundary conditions, additional corrections are required for removing energetic contributions from finite-size effects.Pymatgen provides modules for performing many of these corrections on DefectEntry objects. In order to have them function properly, it is neccessary to load the DefectEntry's parameters with the neccessary metadata for performing these calcualtions. As is detailed later in this notebook, PyCDT and Atomate both provide less cumbersome approaches to loading metadata required for loading relevant parameters for all the corrections.<jupyter_code># load all parameter numbers from example json file
from monty.serialization import loadfn

corr_params = loadfn("example_params.json")
# Load up relevant parameters and apply Freysoldt Correction
from pymatgen.analysis.defects.corrections import FreysoldtCorrection

frey_params = {
    "axis_grid": corr_params["axis_grid"],
    "bulk_planar_averages": corr_params["bulk_planar_averages"],
    "defect_planar_averages": corr_params["defect_planar_averages"],
    "scaling_matrix": corr_params["scaling_matrix"],
    "initial_defect_structure": corr_params["initial_defect_structure"],
    "defect_frac_sc_coords": corr_params["defect_frac_sc_coords"]
    }

dentry = DefectEntry(vac, corr_params['uncorr_energy'], 
                     parameters=frey_params)

fc = FreysoldtCorrection( corr_params['dielectric'])
corr = fc.get_correction( dentry)
print("Freysoldt Correction determined to be: {}".format( corr))
# After evaluating the correction, the freysoldt plot can then be plotted for a given axis to access localization
# (NOTE: this example was for a very small supercell to minimize memory requirement. 
#        Plot shows a clearly delocalized defect state)
import matplotlib
%matplotlib inline

p = fc.plot( 1, title="Example")
p.show()
# Load up relevant parameters and apply Kumagai Correction
from pymatgen.analysis.defects.corrections import KumagaiCorrection

kumagai_params = {
    "bulk_atomic_site_averages": corr_params["bulk_atomic_site_averages"],
    "defect_atomic_site_averages": corr_params["defect_atomic_site_averages"],
    "site_matching_indices": corr_params["site_matching_indices"],
    "initial_defect_structure": corr_params["initial_defect_structure"],
    "defect_frac_sc_coords": corr_params["defect_frac_sc_coords"]
    }

dentry = DefectEntry(vac, corr_params['uncorr_energy'], 
                     parameters=kumagai_params)

kc = KumagaiCorrection( corr_params['dielectric'])
corr = kc.get_correction( dentry)
print("Kumagai Correction determined to be: {}".format( corr))
# After evaluating the correction, the Kumagai plot can then be plotted for a given axis to access localization
# (NOTE: this example was for a very small supercell to minimize memory requirement. 
#        Plot shows a clearly delocalized defect state)
import matplotlib
%matplotlib inline

p = kc.plot(title="Example")
p.show()
# Load up relevant parameters and apply BandFilling Correction
from pymatgen.analysis.defects.corrections import BandFillingCorrection

bandfill_params = {
    "eigenvalues": corr_params["eigenvalues"],
    "kpoint_weights": corr_params["kpoint_weights"],
    "potalign": corr_params["potalign"],
    "cbm": corr_params["cbm"],
    "vbm": corr_params["vbm"]
    }

dentry = DefectEntry(vac, corr_params['uncorr_energy'], 
                     parameters=bandfill_params)

bfc = BandFillingCorrection()
corr = bfc.get_correction( dentry)
print("Bandfilling Correction determined to be: {}".format( corr))
# Load up relevant parameters and apply BandEdgeShifting Correction
from pymatgen.analysis.defects.corrections import BandEdgeShiftingCorrection

bandedgeshift_params = {
    "hybrid_cbm": corr_params["hybrid_cbm"],
    "hybrid_vbm": corr_params["hybrid_vbm"],
    "cbm": corr_params["cbm"],
    "vbm": corr_params["vbm"]
    }

dentry = DefectEntry(vac, corr_params['uncorr_energy'], 
                     parameters=bandedgeshift_params)

bes = BandEdgeShiftingCorrection()
corr = bes.get_correction( dentry)
print("BandEdgeShift Correction determined to be: {}".format( corr))<jupyter_output><empty_output><jupyter_text>Example P4: Use DefectCompatibility to detect delocalized defect states (pymatgen) Determining the localization of a defect can sometimes be a non-trivial task. Moreover, individually performing the above corrections can be cumbersome. The DefectCompatibility module performs a set of localization analyses to determine if the defect is localized while evaluating all of the possible corrections, given the DefectEntry's parameters. The parameters used to determine this localization can be tuned by the user, but the default settings are a good starting point.<jupyter_code># create DefectEntry with all parameters from Example P3

dentry = DefectEntry(vac, corr_params['uncorr_energy'], 
                     parameters=corr_params.copy())
# instantiate Defect Compatibility will delocalization parameters
# and process defect entry fully
from pymatgen.analysis.defects.defect_compatibility import DefectCompatibility

dc = DefectCompatibility( plnr_avg_var_tol=0.1, plnr_avg_minmax_tol=0.1,
                     atomic_site_var_tol=0.1, atomic_site_minmax_tol=0.1,
                     tot_relax_tol=1.0, perc_relax_tol=20.,
                     defect_tot_relax_tol=0.1, preferred_cc='freysoldt',
                     free_chg_cutoff=2.)

dentry = dc.process_entry( dentry)
# notice that process_entry performed all corrections since 
# parameters were all adequately loaded. 

print("Defect Entry has following corrections:")
for k,v in dentry.corrections.items():
    print("\t{}: {}".format( k,v))
# Also notice that compatibility results exist for each metric

print("After DefectCompatibility analysis,"
      " is_compatible = {}".format( dentry.parameters['is_compatible']))
print("Breaking down compatibility analyses")
for k,v in dentry.parameters["delocalization_meta"].items():
    print("\t{}: {}".format(k, v["is_compatible"]))<jupyter_output><empty_output><jupyter_text>Example P5: Create and work with PhaseDiagram objects (pymatgen)<jupyter_code># load example dpd entries
from monty.serialization import loadfn

dpd_dat = loadfn("example_dpd_data.json")
# load a DefectPhaseDiagram
from pymatgen.analysis.defects.thermodynamics import DefectPhaseDiagram
from pymatgen.core import Element

cp = {Element("Al"): -3.74810318, Element("N"): -11.141733989999999,
      Element("H"): -3.4006335908333334, Element("O"): -10.672383753333335
     }
dpd = DefectPhaseDiagram( dpd_dat["entries"],
                          dpd_dat["vbm"], dpd_dat["band_gap"],
                          filter_compatible=False)
# Find stable charges and transition levels

print('Finished charges:')
for k, v in dpd.finished_charges.items():
    print('\t{}: {}'.format( k, v))
print('\nSTABLE charges (and transition levels):')
for k, v in dpd.stable_charges.items():
    print('\t{}: {}\t(t.l.: {} ) '.format( k, v, 
                                           dpd.transition_levels[k]))
# plot the formation energies for the loaded data
import matplotlib
%matplotlib inline

p = dpd.plot(cp, ylim=None, ax_fontsize=1.3, fermi_level = None)
p.show()
# suggest followup charges and supercell sizes 
# based on DefectCompatibility

print('\nNow consider charges for follow up..')
rec_dict = dpd.suggest_charges()
for defname, charge_list in rec_dict.items():
    print("\t{} needs {} charges to be run".format( defname, charge_list))
    
print('\nNow consider larger supercells for follow up..')
rec_dict = dpd.suggest_larger_supercells()
for defname, charge_list in rec_dict.items():
    print("\t{} needs {} should be re-run "
          "with bigger cell".format( defname, charge_list))
# Solve for fermi energy if the bulk_dos is supplied
from pymatgen.core import Element

temp = 600.
bulk_dos = dpd_dat["bulk_dos"]
ef = dpd.solve_for_fermi_energy( temp, cp, dpd_dat["bulk_dos"])
print("Fermi level is {} at Temperature = {} K".format( ef, temp))
print("Plot again...")
p = dpd.plot(cp, ylim=None, ax_fontsize=1.3, fermi_level = ef)
p.show()<jupyter_output><empty_output><jupyter_text>PyCDT defect-related code  PyCDT Example 1: Original command line calls to aid in defect calculation setup, parsing and analysis The original PyCDT paper (linked below) serves as a useful manual for using the command line version of PyCDT to setup defect calculations, perform corrections, parse outputs, and plot formation energies. All of those features still work in PyCDT v2.0 and are useful for users who do not want to learn how to manually make use of pymatgen and/or fireworks.https://doi.org/10.1016/j.cpc.2018.01.004As detailed extensively in the original paper, all of the command line options for pycdt are:- pycdt generate_input- pycdt parse_output- pycdt compute_corrections- pycdt compure_formation_energiesAll command line options also include useful help messages with the flag --help.For further questions or discussion about the utility of PyCDT for certain applications, consider joining the PyCDT google group: https://groups.google.com/forum/!forum/pycdt-forum  PyCDT Example 2: Create DefectEntries from file paths PyCDT Version 2.0 has a SingleDefectParser class which allows for defect identification given a file path. This includes the identification of defect calculations which were not set up without PyCDT (file paths which do not have the transformation.json file). Note that if multiple relaxation runs were required, the Vasprun file may not have the true initial_defect_structure. This can cause occasionally cause issues for defect identification - therefore, this approach without PyCDT generation should be used with caution.<jupyter_code># This example makes use of the files in pycdt/test_files/test_path_files/
# if these are not unzipped and local, then following code unzips them.
import os
if not os.path.exists( './test_path_files/'):
    from pycdt import __file__ as initfilep
    test_file_path = os.path.abspath( os.path.join(os.path.split( initfilep)[0], "../", "test_files"))
    import tarfile
    tar = tarfile.open( os.path.join( test_file_path, "test_path_files.tar.gz"))
    tar.extractall()
    tar.close()
from pycdt.utils.parse_calculations import SingleDefectParser
import numpy as np

defect_file_path = "test_path_files/sub_1_Sb_on_Ga/charge_2/"
bulk_file_path = "test_path_files/bulk/"
dielectric = 18.12 * np.array( [[1., 0, 0],[0., 1., 0],[0, 0, 1.]] )
defect_charge = 2

sdp = SingleDefectParser.from_paths( defect_file_path, bulk_file_path,
                                     dielectric, defect_charge)

print("DefectEntry created from file path!\n\n{}".format( sdp.defect_entry))
# one can then load more essential data for parsing with pymatgen and run the Compatibility Class 
# to compute all relevant corrections. 
bo = sdp.kumagai_loader()
sdp.get_stdrd_metadata()
sdp.get_bulk_gap_data()

print("now running compatibility class...")
sdp.run_compatibility()
print("\nThis produced the following corrections: {}".format( sdp.defect_entry.corrections))<jupyter_output><empty_output><jupyter_text>The sdp.defect_entry DefectEntry object can then be loaded (with others) to a DefectPhaseDiagram for the analysis covered in the previous section of this notebook.  PyCDT Example 3: Intelligently grab chemical potentials from a DefectPhaseDiagram<jupyter_code>from pycdt.core.chemical_potentials import get_mp_chempots_from_dpd

cp = get_mp_chempots_from_dpd( dpd)
print("Chemical potentials determined!")
for k,v in cp.items():
    print("\tFacet {}, has {}".format( k, v))<jupyter_output><empty_output><jupyter_text>PyCDT Example 4: Quickly calculate corrections from a file path As discussed in PyCDT Example 2, version 2.0 includes the SingleDefectParser class for quick identification of defect entry objects without the command line approach. An additional feature that makes use of this are two functions which facilitate the quick calculation and plotting of charge corrections, given a file path.<jupyter_code># plot kumagai potential alignment plot
from pycdt.core.defects_analyzer import kumagai_correction_from_paths

defect_file_path = "test_path_files/sub_1_Sb_on_Ga/charge_2/"
bulk_file_path = "test_path_files/bulk/"
dielectric = 18.12 * np.array( [[1., 0, 0],[0., 1., 0],[0, 0, 1.]] )
defect_charge = 2

kc = kumagai_correction_from_paths( defect_file_path, bulk_file_path,
                                     dielectric, defect_charge, 
                                   plot = False # set to True if correction plot printing is desired
                                  )<jupyter_output><empty_output><jupyter_text>An identical function (with same inputs) exists for the Freysoldt correction (its called freysoldt_correction_from_paths).  PyCDT Example 5: Set up and analyze phase diagram calculations for chemical potentials If defect calculations are being performed with DFT settings that differ from the Materials Project's standard implementation (for example, a different Hubbard U parameter or with the SCAN functional approach), then the phase diagram that is used for determining the chemical potentials should not be those of the Materials Project. PyCDT includes options for setting up and parsing your own PhaseDiagram calculations, using information from the Materials Project database.If you trust the stability of the phases within the GGA-PBE phase diagram, you can choose to limit the number of calculations done by only allowing the setup of phases from neighboring facets around the composition of interest (this is the default operation). Alternatively, one can choose to include all stable phases within the phase diagram (through the boolean flag full_phase_diagram) and/or one can choose to include additional unstable phases in the phase diagram, up to a given energy-above-ground-state-hull value (through adjustment of the energy_above_hull float input).<jupyter_code># Setting up phase diagram calculations
from pycdt.core.chemical_potentials import UserChemPotInputGenerator
from pymatgen.core import Composition

comp = Composition( {"Ga": 1, "As": 1})
uci = UserChemPotInputGenerator( comp)

uci.setup_phase_diagram_calculations( full_phase_diagram = False, 
                                     energy_above_hull = 0
                                    )
print("Now your structures for a phase diagram calculation have been setup "\
      "in the local folder PhaseDiagram.\nWith folder contents:")
print(os.listdir("PhaseDiagram"))<jupyter_output><empty_output><jupyter_text>Once your structure files are setup, you can manually adjust their input settings to your liking and run them. After they have finished, then you can parse them with the same PhaseDiagram/ file structure, using the UserChemPotAnalyzer class. If you trust some of the Materials Project (MP) entries, then you can choose to include any additional MP entries (beyond the calculations you performed) by setting include_mp_entries = True (the default option).<jupyter_code># Parse existing phase diagram object
# (this is an example of how you would do it, 
#  but if calculations have not been run yet then this will fail.)
from pycdt.core.chemical_potentials import UserChemPotAnalyzer

# You need to load your own bulk ComputedEntry to analyze the phase diagram.
# This can be either the supercell calculation or a seperate, converged calculation.
# You can get a ComputedEntry easily from Vasprun.get_computed_entry()
bulk_ce = None #replace this after bulk calculation is performed

uca = UserChemPotAnalyzer( bulk_ce = bulk_ce)

uca.read_phase_diagram_and_chempots( include_mp_entries=True
                                    )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08313656790311524, 'fraction_numerical': 0.013215020184324777, 'mean_word_length': 4.742182374808659, 'pattern_counts': {'":': 25, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '147696', 'license': 'MIT', 'n_tokens_mistral': 7930, 'n_tokens_neox': 7451, 'n_words': 2603, 'path': 'examples/DefectExampleNotebook.ipynb', 'repo_name': 'sthartman/pycdt', 'size': '26289'}