<jupyter_start><jupyter_text>Now we want to set up the **val_params**<jupyter_code># Start the base dictionary
val_params = {}

# All we really need to set here is the path to the validation data and the desired output path to the TFRecord.
# The rest can just be pulled from the training_params.
val_params['root_path'] = os.getcwd()[:-7]+'datasets/val/'
val_params['tf_record_path'] = 'tf_record_val'

# Make sure we've set the right number of validation parameters (not a full proof check)
if len(val_params) < 2:
    raise RuntimeError('Missing something in training_params!')
elif len(val_params) > 2:
    raise RuntimeError('Too much stuff in training_params!')
else:
    print('All set!')<jupyter_output><empty_output><jupyter_text>Now we want to repeat the same for the **dataset_params**<jupyter_code># Start the base dictionary
dataset_params = {}

# First set path for the old and new lensing parameters. If you don't mind rewriting the old file, these can be the
# same.
dataset_params['lens_params_path'] = 'metadata.csv'
dataset_params['new_param_path'] = 'new_metadata.csv'
# Also set where to save the normalization weights to
dataset_params['normalization_constants_path'] = 'norms.csv'

# Set the lens parameters to be pulled from metadata.csv.
dataset_params['lens_params'] = ['external_shear_gamma_ext','external_shear_psi_ext','lens_mass_center_x',
                                 'lens_mass_center_y','lens_mass_e1','lens_mass_e2','lens_mass_gamma',
                                 'lens_mass_theta_E']

# List of parameters that need to be converted to log space
dataset_params['lens_params_log'] = ['lens_mass_theta_E']

# List of parameters that need to be converted from gamma and angle to cartesian coordinates
dataset_params['gampsi'] = {}
dataset_params['gampsi']['gampsi_parameter_prefixes'] = ['external_shear']
dataset_params['gampsi']['gampsi_params_rat'] = ['external_shear_gamma_ext']
dataset_params['gampsi']['gampsi_params_ang'] = ['external_shear_psi_ext']

# Make sure we've set the right number of dataset parameters (not a full proof check)
if len(dataset_params) < 6:
    raise RuntimeError('Missing something in dataset_params!')
elif len(dataset_params) > 6:
    raise RuntimeError('Too much stuff in dataset_params!')
elif len(dataset_params['gampsi']) != 3:
    raise RuntimeError('Ratio and angle parameters are off!')
else:
    print('All set!')<jupyter_output><empty_output><jupyter_text>Finally, we can combine this into our configuration dictionairy and write it to a json file!<jupyter_code>json_path = os.getcwd()+'/t3.json'
config_dict = {'training_params':training_params, 'validation_params':val_params,'dataset_params':dataset_params}
model_trainer.config_checker(config_dict)
with open(json_path,'w') as json_f:
    json.dump(config_dict,json_f,indent=4)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10583684950773559, 'fraction_numerical': 0.0035161744022503515, 'mean_word_length': 5.9559902200489, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '376361', 'license': 'MIT', 'n_tokens_mistral': 898, 'n_tokens_neox': 876, 'n_words': 253, 'path': 'configs/config_factory.ipynb', 'repo_name': 'swagnercarena/ovejero', 'size': '2844'}