<jupyter_start><jupyter_text>Selection of primary sampling units (PSUs)  In the sections below, we draw primary sampling units (PSUs) using probability proportional to size (PPS) sampling techniques implemented in the *Sample* class. The class *Sample* has two main methods that is *inclusion_probs* and *select*. The method *inclusion_probs()* computes the probability of selection and *select()* draws the random samples. The following will illustrate the use of samplics for sample selection. For the illustration,- We consider a stratified cluster design.- We will a priori decide how many PSUs to sample from each stratum- For the clusters selection, we demonstrate PPS methodsThis example is not meant to be exhaustif. There are many use cases that are not covered in this tutorial. For example, some PSUs may be segmented due to their size and segments selected in a subsequent step. Segment selection can be done with Samplics in a similar way as the PSUs selection, with PPS or SRS, after the segements have been created by the user.First, let us import the python packages necessary to run the tutorial.<jupyter_code>import numpy as np 
from samplics.datasets import load_psu_frame
from samplics.sampling import SampleSelection<jupyter_output><empty_output><jupyter_text>Sample Dataset The file *sample_frame.csv* - shown below - contains synthetic data of 100 clusters classified by region (East, North, South and West). Clusters represent a group of households. In the file, each cluster has an associated number of households (number_households) and a status variable indicating whether the cluster is in scope or not. This synthetic data represents a simplified version of enumeration areas (EAs) frames found in many countries and used by major household survey programs such as the Demographic and Health Surveys (DHS), the Population-based HIV Impact Assessment (PHIA) surveys and the Multiple Cluster Indicator Surveys (MICS).<jupyter_code>psu_frame_dict = load_psu_frame()
psu_frame = psu_frame_dict["data"]
psu_frame.head(25)<jupyter_output><empty_output><jupyter_text>Often, sampling frames are not available for the sampling units of interest. For example, most countries do not have a list of all households or people living in the country. Even if such frames exist, it may not be operationally and financially feasible to directly select sampling units without any form of clustering. Hence, stage sampling is a common strategy used by large household national surveys for selecting samples of households and people. At the first stage, geographic or administrative clusters of households are selected. At the second stage, a frame of households is created from the selected clusters and a sample of households is selected. At the third stage (if applicable), a sample of people is selected from the households in the sample. This is a high level description of the process; usually implementations are much less straightforward and may require many adjustments to address complexities.   PSU Probability of Selection At the first stage, we use the proportional to size (pps) method to select a random sample of clusters. The measure of size is the number of households (number_households) as provided in the psu sampling frame. The sample is stratified by region. The probabilities, for stratified pps, is obtained as follow: \begin{equation} p_{hi} = \frac{n_h M_{hi}}{\sum_{i=1}^{N_h}{M_{hi}}} \end{equation} where $p_{hi}$ is the probability of selection for unit $i$ from stratum $h$, $M_{hi}$ is the measure of size (mos), $n_h$ and $N_h$ are the sample size and the total number of clusters in stratum $h$, respectively.**Important.** The pps method is used in many surveys not just for multistage household surveys. For example, in business surveys, establishments can greatly vary in size; hence pps methods are often use to select samples. Simarly, facility-based surveys can benefit from pps methods when frames with measures of size are available.   PSU Sample sizeFor a stratified sampling design, the sample size is provided using a Python dictionary. Python dictionaries allow us to pair the strata with the sample sizes. Let's say that we want to select 3 clusters from stratum *East*, 2 from *West*, 2 from *North* and 3 from *South*. The snippet of code below demonstrates how to create the Python dictionary. Note that it is important to correctly spell out the keys of the dictionary which corresponds to the values of the variable stratum (in our case it's *region*).<jupyter_code>psu_sample_size = {"East":3, "West": 2, "North": 2, "South": 3}

print(f"\nThe sample size per domain is: {psu_sample_size}\n")<jupyter_output>The sample size per domain is: {'East': 3, 'West': 2, 'North': 2, 'South': 3}<jupyter_text>The function *array_to_dict()* converts an array to a dictionnary by pairing the values of the array to their frequency. We can use this function to calculates the number of clusters per stratum and store the result in a Python dictionnary. Then, we modify the values of the dictionnary to create the sample size dictionnary.If some of the clusters are certainties then an exception will be raised. Hence, the user will have to manually handle the certaininties. Better handling of certainties is planned for future versions of the library *samplics*.<jupyter_code>from samplics import array_to_dict

frame_size = array_to_dict(psu_frame["region"])
print(f"\nThe number of clusters per stratum is: {frame_size}")

psu_sample_size = frame_size.copy()
psu_sample_size["East"] = 3
psu_sample_size["North"] = 2
psu_sample_size["South"] = 3
psu_sample_size["West"] = 2
print(f"\nThe sample size per stratum is: {psu_sample_size}\n")
stage1_design = SampleSelection(method="pps-sys", stratification=True, with_replacement=False)

psu_frame["psu_prob"] = stage1_design.inclusion_probs(
    psu_frame["cluster"], 
    psu_sample_size, 
    psu_frame["region"],
    psu_frame["number_households_census"],
    )

nb_obs = 15
print(f"\nFirst {nb_obs} observations of the PSU frame \n")
psu_frame.head(nb_obs)<jupyter_output>First 15 observations of the PSU frame<jupyter_text>PSU Selection In this section, we select a sample of psus using pps methods. In the section above, we have calculated the probabilities of selection. That step is not necessary when using *samplics*. We can use the method *select()* to calculate the probability of selection and select the sample, in one run. As shown below, *select()* method returns a tuple of  three arrays.       * The first array indicates the selected units (i.e. psu_sample = 1 if selected, and 0 if not selected).       * The second array provides the number of hits, useful when the sample is selected with replacement.          * The third array is the probability of selection.        NB: *np.random.seed()* fixes the random seed to allow us to reproduce the random selection.<jupyter_code>np.random.seed(23)

psu_frame["psu_sample"], psu_frame["psu_hits"], psu_frame["psu_probs"] = stage1_design.select(
    psu_frame["cluster"], 
    psu_sample_size, 
    psu_frame["region"], 
    psu_frame["number_households_census"]
    )

nb_obs = 15
print(f"\nFirst {nb_obs} observations of the PSU frame with the sampling information \n")
psu_frame.head(nb_obs)<jupyter_output>First 15 observations of the PSU frame with the sampling information<jupyter_text>The default setting ```sample_only=False``` returns the entire frame. We can easily reduce the output data to the sample by filtering i.e. ```psu_sample == 1```. However, if we are only interested in the sample, we could use ```sample_only=True``` when calling *select()*. This will reduce the output data to the sampled units and ```to_dataframe=true``` will convert the data to a pandas dataframe (pd.DataFrame). Note that the columns in the dataframe will be reduced to the minimum.<jupyter_code>np.random.seed(23)

psu_sample = stage1_design.select(
    psu_frame["cluster"], 
    psu_sample_size, 
    psu_frame["region"], 
    psu_frame["number_households_census"],
    to_dataframe = True,
    sample_only = True
    )

print("\nPSU sample without the non-sampled units\n")
psu_sample<jupyter_output>PSU sample without the non-sampled units<jupyter_text>The systematic selection method can be implemented with or without replacement. The other *samplics* algorithms for selecting sample with unequal probablities of selection are Brewer, Hanurav-Vijayan (hv), Murphy, and Rao-Sampford (rs) methods. As shown below, all these sampling techniques can be specified when extentiating a *Sample* class; then call *select()* to draw samples. ```python Sample(method="pps-sys", with_replacement=True)Sample(method="pps-sys", with_replacement=False)Sample(method="pps-brewer", with_replacement=False)Sample(method="pps-hv", with_replacement=False)  Hanurav-Vijayan methodSample(method="pps-murphy", with_replacement=False)Sample(method="pps-sampford", with_replacement=False)  Rao-Sampford method```For example, if we wanted to select the sample using the Rao-Sampford method, we could use the following snippet of code.<jupyter_code>np.random.seed(23)

stage1_sampford = SampleSelection(method="pps-rs", stratification=True, with_replacement=False)

psu_sample_sampford = stage1_sampford.select(
    psu_frame["cluster"], 
    psu_sample_size, 
    psu_frame["region"], 
    psu_frame["number_households_census"],
    to_dataframe=True,
    sample_only=False
    )

psu_sample_sampford<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07275607180570222, 'fraction_numerical': 0.004751847940865892, 'mean_word_length': 5.373485868102288, 'pattern_counts': {'":': 4, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '478736', 'license': 'MIT', 'n_tokens_mistral': 2671, 'n_tokens_neox': 2548, 'n_words': 1179, 'path': 'docs/source/tutorial/psu_selection.ipynb', 'repo_name': 'samplics-org/samplics', 'size': '9470'}