<jupyter_start><jupyter_text>This notebook provides an example workflow for caking xray diffraction patterns using pyFAI.<jupyter_code>import pathlib
import logging
import math
import warnings

import pyFAI.azimuthalIntegrator
from pyFAI.gui import jupyter
from tqdm.notebook import tqdm
import fabio
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Visualisation of patterns Calibration imageThe calibration image can be opened and read into an array which can be directly plotted.<jupyter_code>image = fabio.open('example-calibration/DLS_CeO2_1200mm.tif')
print("image:", image)

calibration_image_array = image.data
print("calibration_image_array:", type(calibration_image_array), calibration_image_array.shape, calibration_image_array.dtype)

jupyter.display(calibration_image_array)
print("Note, this image appears mirrored from top to bottom, as the origin is at the bottom left.")<jupyter_output><empty_output><jupyter_text>Pattern imageThe pattern file can be opened in the same way.<jupyter_code>image = fabio.open("example-data/pixium_03150.tif")

pattern_image_array = image.data
print("pattern_image_array:", type(pattern_image_array), pattern_image_array.shape, pattern_image_array.dtype)

jupyter.display(pattern_image_array)
print("Note, this image appears mirrored from top to bottom, as the origin is at the bottom left.")<jupyter_output><empty_output><jupyter_text>Because the data is in NumPy arrays, we can do things like subtract the image. This could be useful for subtracting a dark image of the detector, for example.<jupyter_code>example_subtraction = pattern_image_array - calibration_image_array
jupyter.display(example_subtraction)<jupyter_output><empty_output><jupyter_text>MaskA mask of the data can easily be made to exclude any dead pixels. These dead pixels tend to have a -ve value less than zero. The mask returns a binary (true/false) array.<jupyter_code>mask = pattern_image_array < 0<jupyter_output><empty_output><jupyter_text>Example workflows for cakingFirst, we need to load the calibration, which contains information about our beamline setup like the orientationof the detector relative to the image. The calibration is stored in an `azimuthal integrator`.*The `pyFAI-calibration` notebook shows how to create a .poni calibration file from a calibration image. Alternatively, notes on creating a .poni calibration file using [Dioptas](http://www.clemensprescher.com/programs/dioptas) are found on the [LightForm Wiki](https://lightform-group.github.io/wiki/tutorials/sxrd-caking-dioptas)*<jupyter_code>ai = pyFAI.load("example-calibration/DLS_CeO2_1200mm_pyFAI.poni")
print("\nIntegrator: \n", ai)<jupyter_output><empty_output><jupyter_text>Azimuthal integrationTo perform an azimuthal integration we use the `integrate1d` function. There is also an `integrate2d` function, which is designed for caking of the data.* The number of points in 2-theta is defined by the user.* The azimuthal range runs clockwise around the image, from left to top to right to bottom. This is specified as -180${^\circ}$ to 180${^\circ}$, or -pi to pi in the azimuthal range in pyFAI, rather than 0${^\circ}$ to 360${^\circ}$ as in DAWN, as explained in the section on [cake rotation](cell_cake_rotation). An error will appear if using azimuth_range = (0,360).* *Note, the image in these notebooks appears mirrored from top to bottom due to the use of `jupyter.display()`, which plots the origin at the bottom left. However, the origin of an image is usually displayed at the top left.  This makes things confusing, appearing as if -90${^\circ}$ is bottom / South, when in actual fact it is top / North.* * An output .dat file can be saved, which contains a header of metadata. * The result is returned as a numpy array of 2-theta and intensity.An azimuthal integration can be performed like this.[TODO] - review all available options and determine those needed for our Diamond and DESY data.<jupyter_code>result = ai.integrate1d(pattern_image_array,
                        npt=5000,
                        azimuth_range=(-180,180),
                        mask = mask,
                        unit="2th_deg",
                        correctSolidAngle=True,
                        polarization_factor=0.99,
                        method='full_csr',
                        filename="example-analysis/integrated.dat")

print('Size of result numpy array =', np.shape(result))<jupyter_output><empty_output><jupyter_text>CakingThe `integrate2d` function of the azimuthal integrator object does a 2D azimuthal integration. Thisdivides the images into sections (cakes) as a function of azimuthal angle, as well as points in 2-theta angle.* If provided, the `filename` parameter allows output of the data to a file in 'edf' format.This file has the data in ascii format with some metadata. This is not directly the data format that xrdfit reads and so we will output our own custom data file later.* The `method` function allows selection of different methods of averaging. This may be important,  particularly if you are averaging the data to a low number of bins. More details about the different  methods can be found in the pyFAI documentation.* The result is returned as an `Integrate2dResult` which contains NumPy arrays with the caked data.<jupyter_code>result = ai.integrate2d(pattern_image_array,
                        5000,
                        36,
                        azimuth_range=(-180,180),
                        mask = mask,
                        unit="2th_deg",
                        polarization_factor=0.99,
                        method='full_csr',
                        filename = "example-analysis/integrated.edf")
print('The intensity data is stored in "result.intensity" and has the shape: ', np.shape(result.intensity))
print('The radial bin edges are stored in "result.radial" with shape: ', np.shape(result.radial))
print('The azimuthal bin edges are stored in "result.azimuthal" with shape: ', np.shape(result.azimuthal))<jupyter_output><empty_output><jupyter_text>If it is output, we can open the '.edf' file using FabIO.<jupyter_code>cake = fabio.open("example-analysis/integrated.edf")
print(cake.header)
print("cake:", type(cake.data), cake.data.shape, cake.data.dtype)<jupyter_output><empty_output><jupyter_text>Cake rotation More details about the orientation of the cakes is explained in the accompanying `pyFAI-sample-rotation` notebook using an example image.The relation between the cardinal directions perpendicular to the beamline and the azimuthal angle is:- WEST is -180${^\circ}$ / +180${^\circ}$ - NORTH is -90${^\circ}$- EAST is 0${^\circ}$ / 360${^\circ}$- SOUTH is +90${^\circ}$ Comparison to caking in DAWNpyFAI cakes the data clockwise which is the same direction as DAWN. However, pyFAI starts caking the data from the `West` (-180${^\circ}$) direction (compared to DAWN which starts caking from the East direction).*Note, the image in these notebooks appears mirrored from top to bottom due to the use of `jupyter.display()`, which plots the origin at the bottom left.  However, the origin of an image is usually displayed at the top left. This makes things confusing, appearing as if the caking is running anti clockwise, even though it is actually running clockwise.*This means data is caked from -180${^\circ}$ to 0${^\circ}$ to +180${^\circ}$.As shown by the result below, the minimum cake angle in pyFAI is -180${^\circ}$, and so caking will slice along the horizontal, meaning the cake is centred on -175${^\circ}$ cake for 10${^\circ}$ slices, rather than being centred on the horizontal.<jupyter_code>result.azimuthal<jupyter_output><empty_output><jupyter_text>For some of the materials we analyse, there is signal in the cardinal directions so it is desirable that thecakes are centred on the horizontal (and vertical) by applying a rotation to the detector.This rotation needs to be **negative for caking clockwise** and **positive for caking anti-clockwise**, to rotate the detector in the correct sense.<jupyter_code># rotate the detector so that the cardinal direction is in the center of the first cake.
number_of_cakes = 36
first_cake_angle = 360 / number_of_cakes
ai.rot3 = - (first_cake_angle / 2) * (math.pi / 180) # convert rotation to radians
print(np.rad2deg(ai.rot3), 'degrees')
print(ai.rot3, 'radians')<jupyter_output><empty_output><jupyter_text>Caking multiple images for `xrdfit`To iterate through multiple images we can create a loop, create an array for the data and then save it as a text file.xrdfit has a simple data format with the first column being the bin edges of the two-theta angle and the subsequentcolumns being the binned intensity data, one column for each cake.*Note that after the azimuthal integration the `np.flip` function could be used to reverse the order of the columns of intensity data, to order the cakes anticlockwise rather than clockwise.*<jupyter_code># suppress warnings when TIFFs are read
logging.getLogger("fabio.TiffIO").setLevel(logging.ERROR)

# user inputs
number_of_points = 5000
number_of_cakes = 36

# rotate the detector by half a cake width so that the cardinal direction
# is in the center of the first cake not the edge.
first_cake_angle = 360 / number_of_cakes
ai.rot3 = (first_cake_angle / 2) * (math.pi / 180) # convert rotation to radians

# get a list of the files
image_list = sorted(pathlib.Path("example-data/").glob("pixium*"))

for image_path in tqdm(image_list):
    # create empty array
    caked_data = np.zeros((number_of_cakes + 1, number_of_points))
    
    # create an image array and cake the data
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        image = fabio.open(image_path)
    pattern_image_array = image.data
    result2d = ai.integrate2d(pattern_image_array,
                              number_of_points,
                              number_of_cakes,
                              mask = mask,
                              radial_range=(2, 8),
                              unit="2th_deg",
                              polarization_factor=0.99,
                              method='full_csr')
    
    # transpose to order the intensity data
    intensity = result2d.intensity.T
    
    # flip the intensity data could be used to order cakes anticlockwise rather than clockwise
    # intensity = np.flip(result2d.intensity.T, axis=1)
    
    # reshape radial labels to 2D array so they can be attached to the intensity data.
    radial = np.reshape(result2d.radial, (-1, 1))
    
    result_array = np.hstack((radial, intensity))
    
    # write out the caked data to a text file
    np.savetxt(f"example-analysis/{image_path.stem}.dat", result_array, fmt = "%12.8f", delimiter = "\t")<jupyter_output><empty_output><jupyter_text>This caked dataset is now saved in a format that we can use in [xrdfit](https://xrdfit.readthedocs.io/en/stable/) to analyse how the peak profiles change over time. *Note, it is important to check that our input for `number_of_points` can fully resolve the peak profiles, we can check this by plotting the final cake.*<jupyter_code>plt.plot(result_array[:,0],result_array[:,2], marker = ".")
plt.xlim(3,4);<jupyter_output><empty_output><jupyter_text>Caking multiple images for MAUD To cake data in preparation for analysis in [MAUD](http://maud.radiographema.eu) we need to change the output slightly to print out the intensity versus *pixel number*. PyFAI has the option of outputting intensity versus detector distance with the option `unit="r_mm"`. We can then convert this value into a *pixel number*, by providing a pixel size in millimetres.<jupyter_code># suppress warnings when TIFFs are read
logging.getLogger("fabio.TiffIO").setLevel(logging.ERROR)

# user inputs
number_of_points = 5000
number_of_cakes = 72
pixel_size = 0.296

# rotate the detector by half a cake width so that the cardinal direction
# is in the center of the first cake not the edge.
first_cake_angle = 360 / number_of_cakes
ai.rot3 = (first_cake_angle / 2) * (math.pi / 180) # convert rotation to radians

# get a list of the files
image_list = sorted(pathlib.Path("example-data/").glob("pixium*"))

for image_path in tqdm(image_list):
    # create empty array
    caked_data = np.zeros((number_of_cakes + 1, number_of_points))
    
    # create an image array and cake the data
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        image = fabio.open(image_path)
    pattern_image_array = image.data
    result2d = ai.integrate2d(pattern_image_array,
                            number_of_points,
                            number_of_cakes,
                            mask = mask,
                            unit="r_mm",
                            polarization_factor=0.99,
                            method='full_csr')
    
    # transpose to order the intensity data
    intensity = result2d.intensity.T
    
    # flip the intensity data could be used to order cakes anticlockwise rather than clockwise
    # intensity = np.flip(result2d.intensity.T, axis=1)
    
    # reshape radial labels to 2D array so they can be attached to the intensity data.
    radial_mm = np.reshape(result2d.radial, (-1, 1))
    radial_pixel = radial_mm/pixel_size
    
    result_array = np.hstack((radial_pixel, intensity))
    
    # write out the caked data to a text file
    np.savetxt(f"example-analysis/{image_path.stem}.dat", result_array)
plt.plot(result_array[:,0],result_array[:,2], marker = ".")
plt.xlim(210,260);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08179069252894756, 'fraction_numerical': 0.014234088059591416, 'mean_word_length': 3.7763296935540684, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 57, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '654654', 'license': 'MIT', 'n_tokens_mistral': 4054, 'n_tokens_neox': 3800, 'n_words': 1449, 'path': 'pyFAI-caking-example.ipynb', 'repo_name': 'LightForm-group/pyFAI-integration-caking', 'size': '13559'}