<jupyter_start><jupyter_text>Definitions  HHS - Department of Health and Human ServicesCabinet level department whose mandate includes receiving refugees. Current head: [<NAME>](https://en.wikipedia.org/wiki/Alex_Azar) ORR - Office of Refugee ResettlementProgram within the HHS that settles refugees. I'll use the term HHS for work that the ORR does, but many sources make the distinction. UAC - Unaccompanied Alien ChildLegal status given to refugee children when crossing the border, traditionally in cases where a guardian is not present.  Analysis The number of detained children that I've found in articles are not apples-to-apples numbers. I've plotted them as such here because I didn't know any better before doing this. I left them this way because I think the distinctions between the reported numbers would be lost on most people unless they were paying a lot of attention to what was going on. Several events are included as vertical lines. I thought they may be influential on both the media's reporting and on the government's methods of reporting the data to fit a narrative.As can be seen below, this cursory look at the numbers doesn't make a bit of sense.<jupyter_code>plot_reported_detained()<jupyter_output><empty_output><jupyter_text>When the zero-tolerance policy that required families to be separated at the border was implemented, the only relevant number that could be obtained from the HHS was the total number of UAC. The reason for this being that once the families were split, children were considered UAC and processed the same as children who came without adults.The sharp decline at the end of May is due to this change in reporting. I assume this is because the media was more interested in children who were separated from their family. This would be logical -- it wasn't known at this point that the procedure for UAC who had not been separated from their families would soon be changing. The sharp increase in September is due to the media generally returning to a similar UAC metric (more on that).Plotting only UAC numbers makes more sense for now. It is going up a steady rate. It is clear we are imprisoning more and more children.<jupyter_code>plot_uac_detained()<jupyter_output><empty_output><jupyter_text>The executive order that Trump signed on June 20th to end family separation carried with it an extremely important caveat. It served as an excuse to dismantle something referred to as the Flores agreement, which was a set of rules that governed how both UAC and families with children could be treated.The Flores rules were primarily meant to guarantee children certain rights. Basic necessities, such as food, water, medical treatment, running water, etc. were to be provided. Additionally, they would be released and reunited with an adult as soon as possible.Courts also decided that these rights extend to children with families, and by extension the families themselves (since they were not to be separated). Families with children were to be released after twenty days. The Trump administration viewed this as a loophole that led to a "catch and release" immigration policy.In summary, doing away with Flores does several things to harm children that I'm focusing on. It removes rules on the treatment of children, so they can be kept in adult detention centers in worse conditions. It also removes the time constraints on how soon a release must happen, whether it be a UAC or a family with childrne. The Flores agreement was always meant to be superceded by a more thought-out legal framework. The only reason it has stuck around for so long is the solutions to it were hard and every attempt by previous administrations had to be rolled back. All it would take is a stroke of a pen and an alibi to remove these protections though. Not only that, since it was not rolling back established law, the framework could be replaced without entering the public record.It wasn't known until later that in addition to removing the protectons provided by Flores, the detainment procedures were altered to be more strict on the adults who were accompanying children at the border. The purported reasoning was to fight human trafficking -- but it's difficult to see it as anything other than an excuse to separate families without calling it separation.The number of UAC reported by the HHS are influenced by these policy changes. I have called the original post zero-tolerance policy for children UAC A in the plot below. This includes children who were unaccompanied by the Flores' definition, and children who were separated from their families.Under immense political pressure, the administration scrubbed through the pool of UAC A to discover which children had been separated from their parents under the new policy. Since no records of this were required and there was little transparency, there is not much reason to believe the numbers. Regardless, a number was produced. This shows up in the plot as separated children. This 'pool' of separated children was reported on for several months and their reunification has been recorded in the plot below from official HHS statements reported by the media.While the number of this pool of separated children was decreasing, the children of UAC who were not considered separated lost their legal protections. I have called this new group UAC B. UAC B continued to grow while the selected group of separated children were reunited with their families. It is important to note that UAC B continued to grow under this more strict policy on which child was unaccompanied. It seems quite likely that after the pool of separated children was made, many children joined UAC B who would have been considered separated prior to the new rules.The number of children in the UAC B group was not reported on while separated children were being reported on. I have taken the liberty of linearly interpolating the end of UAC A reporting to the beginning of UAC B reporting, while subtracting out the number of separated children where that number is available. It's also worth noting that the period of June in the plot doesn't really make sense -- the three groups plotted below should never all be active at the same time. I haven't fixed it primarily because it's not public knowledge when exactly these policies were changed and for now I'll leave it as a sort of transition zone.<jupyter_code>plot_stacked_immigration_types()<jupyter_output><empty_output><jupyter_text>The above plot fails to capture a newly created group -- families with children who can now be detained indefinitely in adult detention centers. I have not seen any reporting on what this number might be, but we can maybe approximate it.The last reported number I have for UAC A is about 10,000, and from this the pool of 2600 separated children was created. If it can be extrapolated that roughly 25% of children who would have been separated are now placed in adult detention centers with their families, then an additional 25% of all increases to UAC B will have been sent to adult detention centers.This is plotted below. I am also assuming no family releases during this time, which I don't find too unreasonable since the stated goal of the administration is to end "catch and release".<jupyter_code>plot_stacked_immigration_plus_adult()<jupyter_output><empty_output><jupyter_text>It's also now required that HHS perform a background check on child sponsors. The results of this background check are shared with ICE and without any limitations on its use -- they can arrest anyone they deem a criminal. Notably, just being undocumented is considered criminality and they have been deporting such people. It's entirely logical that this would prevent many sponsors from releasing children, even in cases where they are US citizens. This, combined with the lack of Flores rules for the prompt release of children, means that we can expect much longer wait times for release in the future.<jupyter_code>plot_sponsors_arrested()<jupyter_output><empty_output><jupyter_text>Sources Links are referred to by their bracketed number in the raw data, so each bit of data is traceable.In cases where a range is given (e.g. 4-6 months), the average of the range (5 months) is chosen.* [NPR, December 13th [1]](https://www.npr.org/2018/12/13/676300525/almost-15-000-migrant-children-now-held-at-nearly-full-shelters)* [SF Chronicle, November 9th, [2]](https://www.sfchronicle.com/nation/article/More-than-14-000-immigrant-children-are-in-U-S-13399510.php)* [Washington Post, August 30th, [3]](https://www.washingtonpost.com/local/immigration/still-separated-nearly-500-separated-migrant-children-remain-in-us-custody/2018/08/30/6dbd8278-aa09-11e8-8a0c-70b618c98d3c_story.html?utm_term=.db8790abe02a)* [CNN, December 10th, [4]](https://www.cnn.com/2018/12/10/politics/ice-potential-sponsors-arrests/index.html)* [CNN, September 14th, [5]](https://www.cnn.com/2018/09/14/politics/immigrant-children-kept-detention/index.html)* [WSJ, October 19th, [6]](https://www.wsj.com/articles/immigrant-children-are-staying-longer-in-government-custody-1539941401)* [Washington Post, May 29th, [7]](https://www.washingtonpost.com/world/national-security/trumps-zero-tolerance-at-the-border-is-causing-child-shelters-to-fill-up-fast/2018/05/29/7aab0ae4-636b-11e8-a69c-b944de66d9e7_story.html?utm_term=.831c55ba33e5)* [Guardian, September 21st, [8]](https://www.theguardian.com/us-news/2018/sep/21/ice-arrests-undocumented-immigrants-sponsoring-migrant-children)* [MoveOn, June 30th, [9]](https://act.moveon.org/event/families-belong-together_attend1/search/)* [NYT, November 25th, [10]](https://www.nytimes.com/2018/11/25/world/americas/tijuana-mexico-border.html)* [Washington Post, June 20th, [11]](https://www.washingtonpost.com/powerpost/gop-leaders-voice-hope-that-bill-addressing-family-separations-will-pass-thursday/2018/06/20/cc79db9a-7480-11e8-b4b7-308400242c2e_story.html)* [Washington Post, May 7th, [12]](https://www.washingtonpost.com/world/national-security/sessions-says-justice-dept-will-prosecute-every-person-who-crosses-border-unlawfully/2018/05/07/e1312b7e-5216-11e8-9c91-7dab596e8252_story.html)* [Justice For Immigrants, Undated, [13]](https://justiceforimmigrants.org/what-we-are-working-on/unaccompanied-children/orr-and-dhs-information-sharing-agreement-its-consequences/)  Raw Data<jupyter_code>disp_data(df)
disp_data(event_df)<jupyter_output><empty_output><jupyter_text>Code<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np
import datetime
import collections

import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [15, 8]

import copy
def make_row(
        link,
        date,
        detained=None, # Number detained children as reported by any metric
        uac_detained_def1=None, # Number detained children during zero-tolerance, = UAC + separated children
        separated_detained=None, # Number of detained children who were determined to be separated in June
        uac_detained_def2=None, # Number of detained children post zero-tolerance, = UAC - children with families,
        uac_detained_all_def=None, # Number of detained UAC, all definitions
        detention_time_official=None, # Officially stated averange detention time from HHS
        detention_time_unofficial=None, # Unreliable comments about actual detention time from people on ground
        sponsors_arrested=None, # Number of child sponsors arrested by ICE
        cost=None, # Accumulated cost post zero tolerance. Might be hard to find data.
    ):
    
    row_data = collections.OrderedDict()
    row_data['Link'] = link
    row_data['Date'] = date
    row_data['Detained'] = detained
    row_data['UAC Detained Def1'] = uac_detained_def1
    row_data['UAC Detained Def2'] = uac_detained_def2
    row_data['UAC Detained'] = uac_detained_all_def
    row_data['Separated Detained'] = separated_detained
    row_data['Detention Time Official (days)'] = detention_time_official
    row_data['Detention Time Unofficial (days)'] = detention_time_unofficial
    row_data['Sponsors Arrested'] = sponsors_arrested
    row_data['Accumulated Cost ($million)'] = cost
    return row_data

data = [
    make_row(
        1,
        datetime.date(2018, 12, 13),
        detained=14700,
        uac_detained_def2=14700,
        uac_detained_all_def=14700,
        detention_time_official=50,
        detention_time_unofficial=243.334
    ),
    make_row(
        2,
        datetime.date(2018, 11, 9),
        detained=14056,
        uac_detained_def2=14056,
        uac_detained_all_def=14056,
        cost=80
    ),
    make_row(
        3,
        datetime.date(2018, 8, 30),
        detained=497,
        separated_detained=497
    ),
    make_row(
        3,
        datetime.date(2018, 7, 31),
        detained=1200,
        separated_detained=1200
    ),
    make_row(
        3,
        datetime.date(2018, 6, 20),
        detained=2600,
        separated_detained=2600,
    ),
    make_row(
        4,
        datetime.date(2018, 7, 1),
        sponsors_arrested=0
    ),
    make_row(
        4,
        datetime.date(2018, 11, 30),
        sponsors_arrested=170
    ),
    make_row(
        5,
        datetime.date(2018, 9, 14),
        detained=12800,
        uac_detained_def2=12800,
        uac_detained_all_def=12800
    ),
    make_row(
        6,
        datetime.date(2018, 10, 19),
        detention_time_official=59,
        detention_time_unofficial=136
    ),
    make_row(
        6,
        datetime.date(2018, 6, 30),
        detention_time_official=56
    ),
    make_row(
        7,
        datetime.date(2018, 5, 29),
        detained=10073,
        uac_detained_def1=10073,
        uac_detained_all_def=10073
    ),
    make_row(
        7,
        datetime.date(2018, 4, 29),
        detained=8886,
        uac_detained_def1=8886,
        uac_detained_all_def=8886
    ),
    make_row(
        8,
        datetime.date(2018, 9, 21),
        sponsors_arrested=41
    ),
]
df = pd.DataFrame(data=data)
# label: (link #, date, color)
raw_events = {
    'Tear Gas': (
        10,
        datetime.date(2018, 11, 25),
        'red'
    ),
    'Zero Tolerance Policy': (
        11,
        datetime.date(2018, 4, 7),
        'green'
    ),
    'Mass Protest': (
        9,
        datetime.date(2018, 6, 30),
        'blue'
    ),
    'Information Sharing HHS/ICE': (
        13,
        datetime.date(2018, 4, 13),
        'green'
    ),
    'Executive Order': (
        11,
        datetime.date(2018, 6, 20),
        'green'
    )
}
event_data = collections.defaultdict(list)
for name in raw_events.iterkeys():
    event_data['Name'].append(name)
    event_data['Link'].append(raw_events[name][0])
    event_data['Date'].append(raw_events[name][1])

event_df = pd.DataFrame(
    data=event_data
)
def time_evolution(df, y_column, title, events):
    ys_mask = np.isfinite(df[y_column].values)
    ys = df[y_column].values[ys_mask]
    xs = df.Date.values[ys_mask]
    
    xs, ys = zip( *sorted( zip(xs, ys) ) )
    
    plt.xlim = (
        min(
            df.Date.unique() - datetime.timedelta(days=20)
        ),
        max(
            df.Date.unique() + datetime.timedelta(days=20)
        )
    )
    plt.ylim = (
        0.0,
        max(
            ys
        )
    )
    
    plt.plot(
        xs,
        ys,
        marker='o'
    )
    plt.title(title)
    
    def _plot_event(col_name, color, label):
        plt.axvline(
            x=df.Date[df[col_name] == True].values,
            color=color,
            label=label
        )
    
    for event in events:
        link_id, date, color = raw_events[event]
        
        plt.axvline(
            x=date,
            color=color,
            label='{0} [{1}]'.format(event, link_id)
        )

    plt.legend()
    
    ax = plt.subplot(111)
    ax.legend(loc='upper left')
def disp_data(df):
    return df.sort_values(by='Date').reset_index().drop('index', axis=1)
def plot_reported_detained():
    time_evolution(
        df, 'Detained', 'Number of Reported Detained Children By Date',
        events=['Mass Protest', 'Tear Gas', 'Executive Order']
    )
def plot_uac_detained():
    time_evolution(
        df, 'UAC Detained', 'Number of UAC By Date',
        events=['Mass Protest', 'Tear Gas', 'Executive Order']
    )
def plot_stacked_immigration_types():
    f = copy.deepcopy(df)
    
    # sorting by date eases eventual plotting, because sorted dates are needed to plot
    # and the values should be masked by sorted dates
    f = f.sort_values(by='Date').reset_index().drop('index', axis=1)
    uac1_mask = np.isfinite(f['UAC Detained Def1'].values)
    separated_mask = np.isfinite(f['Separated Detained'].values)
    uac2_mask = np.isfinite(f['UAC Detained Def2'].values)
    uac1_separated_mask = (uac1_mask == True) | (separated_mask == True)
    uac1_separated_uac2_mask = (uac1_separated_mask == True) | (uac2_mask == True)

    labels = [
        'UAC Definition 1',
        'Separated Children',
        'UAC Definition 2',
    ]
    
    # [ [uac def 1, ...], [separated, ...], [uac def 2, ...] ]
    
    # The values are deep-copied here to preserve the pre-interpolated values
    uac1_values = copy.deepcopy(f['UAC Detained Def1'].values)
    separated_values = copy.deepcopy(f['Separated Detained'].values)
    uac2_values = copy.deepcopy(f['UAC Detained Def2'].values)
    
    # The frame is interpolated so we can get a UAC all definition that exists over separated values
    uac_values = copy.deepcopy(f.interpolate()['UAC Detained'].values)
    
    # UAC1 is mutually exclusive with the others.
    uac1_values[np.isnan(uac1_values)] = 0.0
    
    # In the raw data, separated is mutually exclusive from UAC1 and UAC2 reporting
    separated_values[np.isnan(separated_values)] = 0.0
    
    # Mutually exclusive from Def 1
    uac2_values[uac1_mask] = 0.0
    
    # derived in the case where separated values are provided
    uac2_values[separated_mask] = uac_values[separated_mask] - separated_values[separated_mask]
    
    dates, uac1_val, sep_val, uac2_val = zip(
        *sorted(
            zip(
                f.Date.values[uac1_separated_uac2_mask],
                uac1_values[uac1_separated_uac2_mask],
                separated_values[uac1_separated_uac2_mask],
                uac2_values[uac1_separated_uac2_mask]
            )
        )
    )

    plt.stackplot(
        dates,
        [
            uac1_val,
            sep_val,
            uac2_val
        ],
        labels=labels
    )
    plt.legend(loc='upper left')
def plot_stacked_immigration_plus_adult():
    f = copy.deepcopy(df)
    
    # sorting by date eases eventual plotting, because sorted dates are needed to plot
    # and the values should be masked by sorted dates
    f = f.sort_values(by='Date').reset_index().drop('index', axis=1)
    uac1_mask = np.isfinite(f['UAC Detained Def1'].values)
    separated_mask = np.isfinite(f['Separated Detained'].values)
    uac2_mask = np.isfinite(f['UAC Detained Def2'].values)
    uac1_separated_mask = (uac1_mask == True) | (separated_mask == True)
    uac1_separated_uac2_mask = (uac1_separated_mask == True) | (uac2_mask == True)
    separated_uac2_mask = (separated_mask == True) | (uac2_mask == True)

    labels = [
        'UAC A',
        'Separated Children',
        'UAC B',
        'Children in Adult Detention',
    ]
    
    # [ [uac def 1, ...], [separated, ...], [uac def 2, ...] ]
    
    # The values are deep-copied here to preserve the pre-interpolated values
    uac1_values = copy.deepcopy(f['UAC Detained Def1'].values)
    separated_values = copy.deepcopy(f['Separated Detained'].values)
    uac2_values = copy.deepcopy(f['UAC Detained Def2'].values)
    
    # The frame is interpolated so we can get a UAC all definition that exists over separated values
    uac_values = copy.deepcopy(f.interpolate()['UAC Detained'].values)
    
    adult_values = np.zeros(len(uac_values))
    adult_values[separated_uac2_mask] = np.append(
        # The first data point here is a special case
        [.25 * (uac_values[separated_uac2_mask][0] - uac_values[uac1_mask][-1]) ],
        np.add.accumulate(
            .25 * (np.diff(uac_values[separated_uac2_mask]))
        )
    )

    # UAC1 is mutually exclusive with the others.
    uac1_values[np.isnan(uac1_values)] = 0.0
    
    # In the raw data, separated is mutually exclusive from UAC1 and UAC2 reporting
    separated_values[np.isnan(separated_values)] = 0.0
    
    # Mutually exclusive from Def 1
    uac2_values[uac1_mask] = 0.0
    
    # derived in the case where separated values are provided
    uac2_values[separated_mask] = uac_values[separated_mask] - separated_values[separated_mask]
    
    dates, uac1_val, sep_val, uac2_val, adult_val = zip(
        *sorted(
            zip(
                f.Date.values[uac1_separated_uac2_mask],
                uac1_values[uac1_separated_uac2_mask],
                separated_values[uac1_separated_uac2_mask],
                uac2_values[uac1_separated_uac2_mask],
                adult_values[uac1_separated_uac2_mask]
            )
        )
    )

    plt.stackplot(
        dates,
        [
            uac1_val,
            sep_val,
            uac2_val,
            adult_val
        ],
        labels=labels
    )
    plt.legend(loc='upper left')
def plot_sponsors_arrested():
    f = copy.deepcopy(df)
    start_date = raw_events['Information Sharing HHS/ICE'][1]
    
    first_arrest_data_date = f.Date[f['Sponsors Arrested'] == 0.0].values[0]
    
    for idx in f.index:
        if f.iloc[idx].Date < first_arrest_data_date:
            f.at[idx, 'Sponsors Arrested'] = 0.0
    
    time_evolution(
        f, 'Sponsors Arrested', 'Sponsors Arrested By Date',
        events=['Mass Protest', 'Tear Gas', 'Executive Order']
    )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0844399670299478, 'fraction_numerical': 0.030863632200750984, 'mean_word_length': 3.316861039731172, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 30, 'https://': 14, 'lorem ipsum': 0, 'www.': 11, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '662117', 'license': 'Unlicense', 'n_tokens_mistral': 7075, 'n_tokens_neox': 6468, 'n_words': 2171, 'path': 'MigrantChildren.ipynb', 'repo_name': 'sourkwhat/notebooks', 'size': '21841'}