<jupyter_start><jupyter_text>Welcome to the Domains Analysis NotebookWelcome to the Archives Unleashed Domains Analysis Jupyter Notebook. This demonstration takes the domain derivatives from the Cloud and uses Python to analyze and produce information about your collection.Please feel free to create an [issue](https://github.com/archivesunleashed/auk/issues) to let us know about any bugs you encounter or improvements you would like to see.If you have Python experience, please feel free to change the provided code to suit your own needs.We recommend that you use `File > Make a Copy` first before changing the code in the repository. That way, you can always return to the basic visualizations we have offered here. Of course, you can also just re-download the Jupyter Notebook file from your Archives Unleashed Cloud account. How Jupyter Notebooks WorkIf you have no previous experience with Jupyter Notebooks, the most important thing to understand is that that ` + ` will run the Python code inside a cell and output it to below the cell.    The cells that cover the required inputs, marked "Setup", need to be run before the rest of the notebook will work. These cells will import all the libraries and set basic variables (e.g. where your derivative files are located) for the notebook. After that, everything else should be able to run on its own.If you just want to see all results for your collection, use `Cell > Run All`.  Setup  The Collection IDThis variable is the most important, and the only variable you need to change to see a complete set of visualizations for your [Archives Unleashed Cloud derivatives](https://cloud.archivesunleashed.org/derivatives).<jupyter_code>COLLECTION_ID = '4867'  # Change to switch collections.
AUK_PATH = 'data/'  # Change value to full path to your data, including trailing slash.<jupyter_output><empty_output><jupyter_text>This section imports the AuNotebook custom class that includes some methods for accessing derivatives and producing visualizations. Also, some libraries like [pandas](https://pandas.pydata.org/), [numpy](http://www.numpy.org/), [networkx](https://networkx.github.io/), and [nltk](https://www.nltk.org/) will be imported so you can do additional analysis.<jupyter_code># Required packages.

from au_notebook import au_notebook
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>User ConfigurationThe following cell sets up the notebook functions under the name `nb`. If you want to use one of the functions in the class, you can use `nb.function_name()`. For example, `nb.get_text()` returns the data from the text derivative as a single text string. Use `nb.__dict__` to see the available settings and functions and `help(nb.function_name)` to see the documentation for a particular function. While setting up the functions, you may also want to change some of the default settings.Take a look here: are there any domains you are not interested in? How many words would you like to be shown? Do you want to filter out 404 results? Do you want to sample the data? Read the choices below carefully.<jupyter_code>nb = au_notebook(COLLECTION_ID, AUK_PATH,

# Maximum number of words to show in output.
# Jupyter will create an output error if the number is too high.

TOP_COUNT = 30,

# Domain suffixes to check non-U.S. domains so that (e.g.) www.google.co.uk will become "google".

STOP_DOMAINS = ["co", "org", "net", "edu"], # Domain suffixes to remove.

FILTERED_DOMAINS = [], # E.g ["google", "apple", "facebook"].

)<jupyter_output><empty_output><jupyter_text>Basic Analysis of DomainsDomains data provides basic information about what is crawled and how often. Change the variables in the following cell to manipulate the domain analysis. For example, you may want to exclude common domains (i.e. "google") or you may be interested in sub-domains such as ca.geocities.com.<jupyter_code>DOMAIN_BY = 'full' # Change to 'sub' if you want to include subdomains.
DOMAIN_EXCLUDE = ['google', 'facebook', 'youtube', 'apple'] # Add items to this list to exclude from the collection.
DOMAIN_FIGURE_SIZE = [10, 4] # Change the width and height of your graph plot ([wdth, hgt]).
DOMAIN_RESULTS = 30 # The number of results to plot.
DOMAIN_BAR_WIDTH = 0.35 # The width of the bars in the histogram.
DOMAIN_Y_LABEL = "Number of occurences." # The label for the y axis.
DOMAIN_TITLE = "Top domains by count." # The title of the graph.
DOMAIN_CSV_OUTPUT = 'domains.csv' # Name of .csv file for output.
domains = nb.get_domains(DOMAIN_BY) # Gets the data from the domains derivative.<jupyter_output><empty_output><jupyter_text>Add Domain Data to Pandas Dataframe and Output to CSVDomain data is usually straight forward, but you may want to add additional modifiers. Pandas dataframes are a good way of doing this.<jupyter_code>domain_array = [(key, key.split('.')[-1], val) for key, val in domains.items()]
dataframe = pd.DataFrame(domain_array, columns=['DOMAIN', 'SUFFIX', 'COUNT'])

print(dataframe.iloc[0:4])

## Removing the # on the following line will write the results 
## as a csv file to DOMAIN_CSV_OUTPUT ('domains.csv' by default).

# dataframe.to_csv(DOMAIN_CSV_OUTPUT)
print(
    dataframe.loc[dataframe['SUFFIX'] == 'com'] # Get just the .com domains.
        .sort_values(by='COUNT', axis=0, ascending=False) # Sort by count.
        .iloc[0:4] # Get first 4 items.
     )
# Count the times a domain appears and count it.

name_domains = nb.get_domains('name') # Remove prefix and suffix (ie. 'www.google.com' becomes 'google').
plt.rcParams['figure.figsize'] = DOMAIN_FIGURE_SIZE # Set the figure size for the graph.
most_common = name_domains.most_common(DOMAIN_RESULTS) 
vals = [x[1] for x in most_common if x[0] not in DOMAIN_EXCLUDE]
labs = [x[0] for x in most_common if x[0] not in DOMAIN_EXCLUDE]
ind = np.arange(len(vals))
width = DOMAIN_BAR_WIDTH
p1 = plt.bar(ind, vals, width)
plt.ylabel(DOMAIN_Y_LABEL)
plt.title(DOMAIN_TITLE)
plt.xticks(ind, labs, rotation='vertical')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0830860534124629, 'fraction_numerical': 0.004780745136828223, 'mean_word_length': 4.919024390243902, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 5, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '479049', 'license': 'Apache-2.0', 'n_tokens_mistral': 1788, 'n_tokens_neox': 1731, 'n_words': 786, 'path': 'auk-notebook-domains.ipynb', 'repo_name': 'MarieGallagher/auk-notebooks', 'size': '6066'}