<jupyter_start><jupyter_text>Indexing and Integration...using _CrystFEL's_ `indexamajig` tool and several wrappers around it.What you need to begin:* your data with found peaks and beam center. This can either be a `.lst` file that contains the file names to full `HDF5` data files, or an `image_info.h5`-type file that is generated by `Dataset.compute_pattern_info` and contains peak positions (but not the actual image data). When using the former, you get integrated intensities from the predicted Bragg spots right away.* a refined unit-cell file, which you can generate in `peak_processing.ipynb`, and good geometry settings in your `.yaml` config file. If unsure about ellipticity, double check using `peak_processing.ipynb`.First, we define the list of shot list fields which should go into the output stream file of indexing. (See `indexamajig --copy-hdf5-field`) and generate a _CrystFEL_ geometry file (See `man crystfel_geometry`) from our parameter `.yaml` file.<jupyter_code>stream_fields = ['sample', 'region', 'crystal_id', 'run', 
                '_Event', '_file', 'center_x', 'center_y'] 
stream_fields = [f'/%/shots/{f}' for f in  stream_fields]

# generate geometry file for virtual geometry from yaml file parameters.
opts.load()
tools.make_geometry(opts, 'refined.geom', image_name='corrected', write_mask=True)<jupyter_output><empty_output><jupyter_text>Direct local execution...generates a shell script `im_run.sh` containing the CrystFEL call, to directly run on this machine, using a number of processes defined in the `procs` argument.All parameters for indexing are set in the `preproc.yaml` file.This can either be done using (A) the `image_info.h5` file, in which case the `.stream` file does not contain useful integrated intensities, **or** (B) directly using the processed data files from `hits_agg.lst`. You can always (re-)run the integration step later on.<jupyter_code># VERSION A: from image_info.h5
opts.load() # often reload the opts so they remain updated
!echo image_info.h5 > image_info.lst
tools.call_indexamajig('image_info.lst', 'refined.geom', script='im_run.sh', 
                       output='master.stream',  cell='refined.cell', im_params=opts.indexing_params, 
                       copy_fields=stream_fields, procs=48)
# VERSION B: from the processed data files
opts.load()
tools.call_indexamajig('hits_agg.lst', 'refined.geom', script='im_run.sh', 
                       output='master.stream',  cell='refined.cell', im_params=opts.indexing_params, 
                       copy_fields=stream_fields, procs=48)
# for the curious cats... show the command line.
!cat im_run.sh<jupyter_output>indexamajig -g refined.geom -i image_info.lst -o master.stream -j 48 -p refined.cell --indexing=pinkIndexer --integration=rings-nograd-nocen --int-radius=3,4,6 --peaks=cxi --max-indexer-threads=2 --min-peaks=15 --no-refine --no-retry --no-check-peaks --camera-length-estimate=1.58 --pinkIndexer-considered-peaks-count=4 --pinkIndexer-angle-resolution=4 --pinkIndexer-refinement-type=5 --pinkIndexer-tolerance=0.1 --pinkIndexer-reflection-radius=0.001 --pinkIndexer-max-resolution-for-indexing=2 --temp-dir=/scratch/diffractem --copy-hdf5-field=/%/shots/sample --copy-hdf5-field=/%/shots/region --copy-hdf5-field=/%/shots/crystal_id --copy-hdf5-field=/%/shots/run --copy-hdf5-field=/%/shots/_Event --copy-hdf5-field=/%/shots/_file --copy-hdf5-field=/%/shots/center_x --copy-hdf5-field=/%/shots/center_y<jupyter_text>Version for clusters...which splits up the patterns into sections of `shot_per_run`, and generates a script file that submits them independently to a SLURM queue manager. Similar to CrystFEL's `turbo-index-slurm`, but a bit more streamlined. All required files for indexing can be optionally packed into a `.tar.gz` file, which can be uploaded to a cluster right away and run there.Here, `procs` defines the number of parallel processes with which a chunk of `shots_per_run` shots is processed; additionally `threads` can be defined, which are used by _pinkIndexer_. Vs `procs`, this is especially useful to save memory.Here it is important, that the `exc` argument gets the path to the `indexamajig` executable on your cluster.Here, we run the cluster indexing using the `image_info.h5` file (A from above), as it is small and can be sent to a remote cluster. Of course, you could as well use `hits_agg.lst` -- but then you'll have to have the actual data files available on the cluster.<jupyter_code>!echo image_info.h5 > image_info.lst
opts.load()
tar, script = tools.call_indexamajig_slurm('image_info.lst', 'refined.geom', name='lyso_idx', 
                                           cell='refined.cell',
                             im_params=opts.indexing_params, procs=4, threads=2, shots_per_run=50,
                             write_tar_file=True, temp_dir='$TMP_LOCAL', copy_fields=stream_fields,
                             exc='indexamajig')<jupyter_output>Wrote self-contained tar file lyso_idx.tar.gz. Upload to your favorite cluster and extract with: tar -xf lyso_idx.tar.gz
Run indexing by calling ./im_run_lyso_idx.sh<jupyter_text>Template for sending to/receiving from a clusterShown for Robert's favorite cluster. Adjust to your likes.<jupyter_code># upload immediately to your cluster
remote = '<EMAIL>:~/SHARED/EDIFF/lyso_redo' # ...for example.
!ssh {remote.split(":")[0]} 'mkdir -p {remote.split(":")[1]}'
!scp {tar} {remote}
# !scp pxmask.h5 {remote}
# concat streams on server and transfer back
name = 'lyso_idx'
cmd = f'ssh {remote.split(":")[0]} \"cat {remote.split(":")[1]}/partitions/*.stream > {remote.split(":")[1]}/master.stream\"'
!{cmd}
!scp -r {remote}/master.stream .<jupyter_output>master.stream                                 100%  106MB  52.9MB/s   00:02<jupyter_text>(Re-)IntegrationNow we have the file `master.stream`, which contains our indexing solution -- and if you used your data files above (version B) also integrated intensities ready for merging!If you did your indexing on the info file `image_info.h5` instead (locally version A, or remote), you still need to get the integrated intensities, or you might also want to play with the parameters and recalculate them even if you have them already.We then need to run `indexamajig` a second time, this time on our actual data and using `indexing=file`.In this case, `indexamajig` will, instead of computing an indexing solution, read it from a `.sol` file that contains the reciprocal lattice vectors, refined detector shift, and lattice type for each of the indexed crystals. As the `.sol` file identifies the patterns from their file names and Event specifiers, it needs to be generated specifically for each dataset you want to integrate, even if the indexing solution is the same.  Making a `.sol` fileObviously, we first need to generate this file from our stream file.  Solution file from datasetThe usually easier option to generate a `.sol` file.Here, a Dataset that you want to integrate is loaded from disk.The stored identification data (i.e.: `sample`, `region`, `run`, `crystal_id`) are used for matching the crystals in the dataset with those in the stream.You can hence now integrate even from a totally different set of patterns (e.g. a different aggregation range, or even a set with all non-aggregated data - the crystal ID data will just repeat for each frame). This is essential for dose-fractionated sets!<jupyter_code>dsname = 'hits_agg'
ds = Dataset.from_files(dsname + '.lst', open_stacks=False)
ds.get_indexing_solution('master.stream', sol_file=dsname + '.sol')<jupyter_output><empty_output><jupyter_text>Solution file directly from streamThe other option to get a `.sol` file is to run the `stream2sol` command-line tool. This is restricted to the case that you want to integrate from the exact same images as those you used for indexing (i.e., those from `hits_agg.lst` in this example), or an `image_info.h5` generated from it. In the latter case, you have to supply some special parameters to `stream2sol`... see below.<jupyter_code>from_image_info = True
cmd = tools.make_command('stream2sol', input='master.stream', output='hits_agg_2.sol',
                         event_field='header/str//%/shots/_Event' if from_image_info else None, 
                         file_field='header/str//%/shots/_file' if from_image_info else None)
print('Running conversion command:', cmd)
!{cmd};<jupyter_output>Running conversion command: stream2sol --input=master.stream --output=hits_agg_2.sol --event-field=header/str//%/shots/_Event --file-field=header/str//%/shots/_file<jupyter_text>Have a look...at the `.sol` file, to get a feeling... the numeric rows are the 3 components of each of the 3 lattice vectors, and the x and y detector shifts in mm.<jupyter_code>!head -n 5 {dsname}.sol<jupyter_output>proc_data/LysoS1_001_00000_agg_hit.h5 entry//0 -0.07982 0.09733 0.02101 -0.09808 -0.08125 0.00782 0.04409 -0.02051 0.2581 -0.037 0.025 tPc
proc_data/LysoS1_001_00000_agg_hit.h5 entry//1 -0.02552 0.1169 0.04272 -0.1231 -0.03025 0.01265 0.04248 -0.07794 0.2461 0.017 -0.009 tPc
proc_data/LysoS1_001_00000_agg_hit.h5 entry//2 -0.1256 0.01993 -0.009939 -0.02061 -0.1243 0.008538 -0.01555 0.01706 0.2645 -0.007 -0.014 tPc
proc_data/LysoS1_001_00000_agg_hit.h5 entry//3 0.1169 -0.05005 0.007035 0.0499 0.105 -0.04993 0.02774 0.1049 0.2431 0.002 -0.006 tPc
proc_data/LysoS1_001_00000_agg_hit.h5 entry//4 0.03967 0.1013 -0.06523 0.1205 -0.04362 0.003169 -0.03912 -0.1311 -0.2271 0.02 -0.014 tPc<jupyter_text>Run the integrationNow we're all set to integrate the data set.The parameters for integration are all set in the `integration_params` structure in `preproc.yaml`; the `.sol` file name is supplied in the `fromfile_input_file` parameter.It can be well worth playing with them, especially `int-radius` and `integration`.For the latter, we recommend to stick to `rings-nograd-nocen`, if your patterns are background-subtracted.Otherwise `rings-grad-nocen` might work better.Abstain from anything with `cen` in it, as it will strongly bias high-resolution peak values.`overpredict` might help if you plan to do merging with partiality correction (though it doesn't much in our experience), but absolutely don't do it for Monte-Carlo merging.**Always keep `no-revalidate`, `no-retry`, `no-refine`, `no-check-cell` active.**After you've run the command (might take a fair bit), you'll have a stream file ready for merging. See `merging.ipynb`.<jupyter_code># %mkdir streams
copy_fields = ['sample', 'region', 'crystal_id', 'run', 
               'adf1', 'adf2', 'lor_hwhm', 'center_x', 'center_y']
copy_fields = [f'/%/shots/{cf}' for cf in copy_fields]

opts.load()
cfcall = tools.call_indexamajig(f'{dsname}.lst', 'refined.geom', 
                                output=f'streams/{dsname}.stream', 
                                cell='refined.cell', 
                                im_params=opts.integration_params, 
                                procs=40, exc='indexamajig',
                                fromfile_input_file = f'{dsname}.sol',
                                copy_fields=copy_fields)

print('--- RUN THIS ---------------')
print(cfcall)<jupyter_output>--- RUN THIS ---------------
/opts/crystfel_latest/bin/indexamajig -g refined.geom -i hits_agg.lst -o streams/hits_agg.stream -j 40 -p refined.cell --indexing=file --integration=rings-nograd-nocen --int-radius=3,4,6 --peaks=cxi --max-indexer-threads=2 --no-refine --no-retry --no-check-peaks --fromfile-input-file=hits_agg.sol --copy-hdf5-field=/%/shots/sample --copy-hdf5-field=/%/shots/region --copy-hdf5-field=/%/shots/crystal_id --copy-hdf5-field=/%/shots/run --copy-hdf5-field=/%/shots/adf1 --copy-hdf5-field=/%/shots/adf2 --copy-hdf5-field=/%/shots/lor_hwhm --copy-hdf5-field=/%/shots/center_x --copy-hdf5-field=/%/shots/center_y
{'fraction_non_alphanumeric': 0.11581178465451462, 'fraction_numerical': 0.03789741415854175, 'mean_word_length': 3.8205966489579075, 'pattern_counts': {'":': 5, '<': 28, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '391386', 'license': 'MIT', 'n_tokens_mistral': 4024, 'n_tokens_neox': 3682, 'n_words': 1228, 'path': 'indexing.ipynb', 'repo_name': 'robertbuecker/serialed-examples', 'size': '11810'}