<jupyter_start><jupyter_text>Examining all files in RecordSearch with the access status of 'Closed'(Harvested on 1 January 2021)This notebook attempts some large-scale analysis of files from the National Archives of Australia's RecordSearch database that have the access status of 'closed'. For a previous attempt at this, see [Closed Access](http://closedaccess.herokuapp.com/). For more background, see my [*Inside Story* article](https://insidestory.org.au/withheld-pending-advice/) from 2018.See [this notebook](harvest_closed_files.ipynb) for the code used to harvest the data and create the CSV dataset.<jupyter_code>import pandas as pd
import altair as alt
import datetime<jupyter_output><empty_output><jupyter_text>The harvested data has been saved as a CSV file. First we'll open it up using Pandas.<jupyter_code>df2020 = pd.read_csv('data/closed-20210101.csv', parse_dates=['contents_start_date', 'contents_end_date', 'access_decision_date'], keep_default_na=False)
df2020.head()<jupyter_output><empty_output><jupyter_text>How many closed files are there?<jupyter_code>df2020.shape[0]<jupyter_output><empty_output><jupyter_text>What series do the 'closed' files come from?First let's see how many different series are represented in the data set.<jupyter_code>df2020['series'].unique().shape[0]<jupyter_output><empty_output><jupyter_text>Now let's look at the 25 most common series.<jupyter_code>df2020['series'].value_counts()[:25]<jupyter_output><empty_output><jupyter_text>Series [A1838](http://recordsearch.naa.gov.au/scripts/AutoSearch.asp?Number=A1838) is familiar to anyone who's looked into the NAA's access examination process. It's a general correspondence series from DFAT, and requests for access tend to take a **long** time to be processed. Series [K60](http://recordsearch.naa.gov.au/scripts/AutoSearch.asp?Number=K60) contains repatriation files from the Department of Veterans' Affairs, so these will often been withheld on privacy grounds. We'll see more about both of these below.Let's chart the results.<jupyter_code># This creates a compact dataset to feed to Altair for charting
# We could make Altair do all the work, but that would embed a lot of data in the notebook.

# Save the series counts to a new dataframe.
series_counts = df2020['series'].value_counts().to_frame().reset_index()
series_counts.columns = ['series', 'count']
# Chart the results, sorted by number of files
alt.Chart(series_counts[:50]).mark_bar().encode(
    x=alt.X('series', sort='-y'),
    y=alt.Y('count', title='number of files'),
    tooltip=['series', 'count']
)<jupyter_output><empty_output><jupyter_text>This is only the top 50 of 686 series, so quite obviously there's a very long tail of series that have a small number of closed files.  What reasons are given for closing the files?[Section 33](http://www8.austlii.edu.au/cgi-bin/viewdoc/au/legis/cth/consol_act/aa198398/s33.html) of the Archives Act defines a number of 'exemptions' – these are reasons why files should not be opened to public access. These reasons are recorded in RecordSearch, so we can explore why files have been closed. It's a little complicated, however, because multiple exemptions can be applied to a single file. The CSV data file records multiple reasons as a pipe-separated string. First we can look at the most common combinations of reasons.<jupyter_code>df2020['reasons'].value_counts()[:25]<jupyter_output><empty_output><jupyter_text>It's probably more useful, however, to look at the frequency of individual reasons. So we'll split the pip-separated string and create a row for each file/reason combination.<jupyter_code>df2020_reasons = df2020.copy()
# Split the reasons field on pipe symbol |. This turns the string into a list of values.
df2020_reasons['reason'] = df2020_reasons['reasons'].str.split('|')
# Now we'll explode the list into separate rows.
df2020_reasons = df2020_reasons.explode('reason')<jupyter_output><empty_output><jupyter_text>Now we can look at the frequency of individual reasons. Not, of course, that the sum of the reasons will be greater than the number of files, as some files have multiple exemptions applied to them.<jupyter_code>df2020_reasons['reason'].value_counts()<jupyter_output><empty_output><jupyter_text>The reasons starting with '33' are clauses in [section 33](http://www8.austlii.edu.au/cgi-bin/viewdoc/au/legis/cth/consol_act/aa198398/s33.html) of the Archives Act. You can look up the Act to find out more about them, or [look at this list](https://www.naa.gov.au/help-your-research/using-collection/access-records-under-archives-act/why-we-refuse-access) on the NAA website. Some of the reasons, such as 'Parliament Class A' refer to particular types of records that are not subject to the same public access arrangements as other government records. Others, such as 'MAKE YOUR SELECTION' seem to be products of the data entry system!Looking at the other most common reasons:* 33(1)(g) relates to privacy* 'Withheld pending adv' is applied to files that are undergoing access examination and have been referred to the relevant government agency for advice on whether they should be released to the public. This is not a final determination – these files may or may not end up being closed. But, as any researcher knows, this process can be *very* slow.* 33(1)(a) is the national security catch-allYou might also notice that there's a blank line in the list above. This is because some closed files have no reasons recorded in RecordSearch. We can check this.<jupyter_code>missing_reasons = df2020.loc[df2020['reasons'] == '']
missing_reasons.shape[0]<jupyter_output><empty_output><jupyter_text>There are 46 closed files with no reason recorded. Here's a sample.<jupyter_code>missing_reasons.head()<jupyter_output><empty_output><jupyter_text>Let's change the missing reasons to 'None recorded' to make it easier to see what's going on.<jupyter_code>df2020_reasons['reason'].replace('', 'None recorded', inplace=True)<jupyter_output><empty_output><jupyter_text>Let's chart the frequency of the different reasons.<jupyter_code># Once again we'll create a compact dataset for charting
reason_counts = df2020_reasons['reason'].value_counts().to_frame().reset_index()
reason_counts.columns = ['reason', 'count']
# Make the Chart
alt.Chart(reason_counts).mark_bar().encode(
    x='reason',
    y=alt.Y('count', title='number of files'),
    tooltip=['reason', 'count']
)<jupyter_output><empty_output><jupyter_text>Connecting reasons and seriesIt would be interesting to bring together the analyses above and see how reasons are distributed across series. First we need to reshape our dataset to show combinations of series and reasons.<jupyter_code># Group files by series and reason, then count the number of combinations
series_reasons_counts = df2020_reasons.groupby(by=['series', 'reason']).size().reset_index()
# Rename columns
series_reasons_counts.columns = ['series', 'reason', 'count']<jupyter_output><empty_output><jupyter_text>Now we can chart the results. Once again we'll show the number of files in the 50 most common series, but this time we'll highlight the reasons using color.<jupyter_code>alt.Chart(series_reasons_counts).transform_aggregate(
    count='sum(count)',
    groupby=['series', 'reason']
# Sort by number of files
).transform_window( 
    rank='rank(count)',
    sort=[alt.SortField('count', order='descending')]
# Get the top 50
).transform_filter(
    alt.datum.rank < 50
).mark_bar().encode(
    x=alt.X('series', sort='-y'),
    y=alt.Y('sum(count)', title='number of files'),
    color='reason',
    tooltip=['series', 'reason', 'count']
)<jupyter_output><empty_output><jupyter_text>Now we can see that the distribution of reasons varies considerably across series.  How old are these files?You would think that the sensitivity of material in closed files diminishes over time. However, there's no automatic re-assessment or time limit on 'closed' files. They stay closed until someone asks for them to be re-examined. That means that some of these files can be quite old. How old? We can use the contents end date to explore this.<jupyter_code># Normalise contents end values as end of year
df2020['contents_end_year'] = df2020['contents_end_date'].apply(lambda x: datetime.datetime(x.year, 12, 31))
date_counts = df2020['contents_end_year'].value_counts().to_frame().reset_index()
date_counts.columns = ['end_date', 'count']
alt.Chart(date_counts).mark_bar().encode(
    x='year(end_date):T',
    y='count'
).properties(width=700)
alt.Chart(date_counts.loc[date_counts['end_date'] > '1890-12-31']).mark_bar().encode(
    x='year(end_date):T',
    y='count',
    tooltip='year(end_date)'
).properties(width=700)
df2020['years_old'] = df2020['contents_end_year'].apply(lambda x: round((datetime.datetime.now() - x).days / 365))
age_counts = df2020['years_old'].value_counts().to_frame().reset_index()
age_counts.columns = ['age', 'count']
alt.Chart(age_counts.loc[age_counts['age'] < 130]).mark_bar().encode(
    x=alt.X('age:Q', title='age in years'),
    y=alt.Y('count', title='number of files'),
    tooltip=['age', 'count']
).properties(width=700)
df2020['years_old'].describe()
df2020.loc[df2020['reasons'].str.contains('33(1)(a)', regex=False)]['years_old'].describe()
df2020['years_old'].quantile([0.25, 0.5, 0.75]).to_list()<jupyter_output><empty_output><jupyter_text>Dates of decisions<jupyter_code>df2020['year'] = df2020['access_decision_date'].dt.year
year_counts = df2020['year'].value_counts().to_frame().reset_index()
year_counts.columns = ['year', 'count']
alt.Chart(year_counts).mark_bar().encode(
    x='year:O',
    y='count'
)<jupyter_output><empty_output><jupyter_text>33(1)(a)<jupyter_code>df331a = df2020.loc[df2020['reasons'].str.contains('33(1)(a)', regex=False)]
df331a.head()
series_counts_331a = df331a['series'].value_counts().to_frame().reset_index()
series_counts_331a.columns = ['series', 'count']
alt.Chart(series_counts_331a[:50]).mark_bar().encode(
    x=alt.X('series', sort='-y'),
    y=alt.Y('count', title='number of files'),
    tooltip=['series', 'count']
)<jupyter_output><empty_output><jupyter_text>Withheld pending advice<jupyter_code>dfwh = df2020.loc[df2020['reasons'].str.contains('Withheld pending adv', regex=False)]
dfwh.head()
series_counts_wh = dfwh['series'].value_counts().to_frame().reset_index()
series_counts_wh.columns = ['series', 'count']
alt.Chart(series_counts_wh[:50]).mark_bar().encode(
    x=alt.X('series', sort='-y'),
    y='count'
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12188392007611798, 'fraction_numerical': 0.027878211227402475, 'mean_word_length': 6.309457579972183, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 78, 'https://': 2, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '596990', 'license': 'MIT', 'n_tokens_mistral': 3486, 'n_tokens_neox': 3205, 'n_words': 1000, 'path': 'closed-files-2020-analysis.ipynb', 'repo_name': 'GLAM-Workbench/recordsearch', 'size': '10510'}