<jupyter_start><jupyter_text>More Explorations with MoMA Scrape![image](img/gallery.webp)This notebook further explores web data scraped from the MoMA collection.The [previous notebook](https://github.com/tnakatani/python2_ccac/blob/master/wk_8/moma_scrape/moma_scrape.ipynb) explored scraped data of specific artists.  This time, the script was revised to randomly sample artworks from the collection in order to explore trends in the larger collection. Step 1: Data Collection The [moma_scrape.py](moma_scrape.py) script created an artwork dataset with the following steps:1. NumPy's [`randint`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.randint.html?highlight=randintnumpy.random.randint) method creates a list of random integers.2. The `ArtworkSoup` class contains a method `scrape`, which takes the integers as a parameter.  The class subsequently builds a URL with the integer and makes a HTTP request.  Both successful and unsuccessful HTTP calls are logged in a separate `scrape.log` file.3. If the HTTP request is successful, the class instantiates a `BeautifulSoup` object, extracts relevant artwork data from the HTML and builds a data structure from it.4. Once all HTTP calls are made, the resulting data structure is dumped to a CSV file.__One flaw in the method__: Each integer in the 300K list of random integer does not necessarily guarantee a a valid request.  It was not clear to me how the MoMA site assigns these artwork URL IDs, so I attempted to throw a large number of a randomized sample and see how many valid requests I can receive.  Using the scraping logs, I calculated the rate of 404 errors:```pythonwith open('scrape.log', 'r') as f:    count = 0    count_404 = 0    for line in f:        count += 1        if 'INFO:root:404' in line:            count_404 += 1    print(f'Total HTTP requests resulting in 404 response: {count_404}')    print(f'Total HTTP requests: {count}')        print(f'Ratio of 404 response versus 200: {count_404/count*100:.2f}%') Total requests resulting in 404: 17567 Total HTTP requests: 30000 Ratio of 404 response versus 200: 58.56%```Thus, more than half of the 300K requests resulted in a 404 error!  Step 2: Data ExplorationNow that we have the scraped data, we can explore it as a dataframe.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
# Import the CSV into a dataframe
df = pd.read_csv('data/artwork_data.csv', index_col=0)

# Always show all of the columns
pd.set_option('display.max_columns', None)

# View a sample of the data
df.sample(10).dropna(how='all', axis=1)<jupyter_output><empty_output><jupyter_text>While scraping the data, I noticed that the artworks had a wide range in the amount of metadata - ranging from common keys (e.g. "artist", "title") to less common ("producer", "delineator").<jupyter_code># Inspect all the features of an artwork from the 
df.keys()<jupyter_output><empty_output><jupyter_text>I also forgot to add a URL for each of the artworks.  Since we have the artwork ID, we can concatenate that with the URL template<jupyter_code># Create a column of URLs from artwork IDs
url = f'https://www.moma.org/collection/works/'
df['url'] = df['id'].apply(lambda x: url + str(x))
df['url'].head()<jupyter_output><empty_output><jupyter_text>Sorting Values by FrequencyA typical way to inspect categorical data like this is to calculate the relative occurrence of unique values.  This will answer questions like, "What artist do we most frequently see in this dataset?" and more.Using `value_counts()` and `pd.concat()`, we can view both the raw count as well as the relative frequency of each unique value.<jupyter_code># Clean up the state/variant column as it causes the loop to break because of its slash
df = df.rename(columns={'state/variant':'state_variant'})
# Print value counts for each column, then save each aggregation as a CSV file.
for k in df.keys()[1:]:
    # Assign both raw counts and relative frequencies to variables
    count = df[k].value_counts(dropna=True)
    freq = df[k].value_counts(normalize=True, dropna=True)
    
    # Concatenate the two series into a dataframe
    values_combined = pd.concat([count, freq], keys=['count','frequency'], axis=1)
    
    # Save each dataframe as its own CSV file
    values_combined.reset_index() \
                   .rename(columns={'index': k}) \
                   .to_csv(f'data/{k}.csv')
    
    # Print results here
    banner = f'{k.upper()} - Total Unique Values: {len(count)}'
    print(banner)
    print(values_combined.head(10), '\n')<jupyter_output>ARTIST - Total Unique Values: 2038
                          count  frequency
<NAME>               675   0.054291
<NAME>               311   0.025014
<NAME>                   291   0.023405
<NAME>              210   0.016891
<NAME>                  200   0.016086
Unknown Artist              164   0.013191
<NAME>         146   0.011743
<NAME>               120   0.009652
<NAME>               120   0.009652
<NAME>    118   0.009491 

TITLE - Total Unique Values: 7094
                                                    count  frequency
Vase                                                   92   0.007400
Bowl                                                   77   0.006193
Untitled fromFound Masks 1975–1978                     37   0.002976
Fabric Sample                                          29   0.002333
Untitled fromTwenty-Four Personal Attitudes Rel...     28   0.002252
The Manhattan Transcripts Project, New Yor[...]<jupyter_text>Some of the more interesting data points from the cursory view of the data:1. __<NAME>'s__ works were the most represented, with a relative frequency of ~5%.2. The most common artwork title is __"Vase"__, but the frequency distribution has quite a long tail.3. The most common manufacturer is __Braun AG__ [wiki](https://en.wikipedia.org/wiki/Braun_(company)).  [Here](https://www.moma.org/collection/works/3391) is an example of a Braun product they have in the collection.4. The most common medium is a __lithograph__, and its most common printer is [__Mourlot__](https://en.wikipedia.org/wiki/Mourlot_Studios), a commercial print shop founded in 1852 by the Mourlot family and located in Paris, France.  Grouping Dimensions for New PerspectivesThere are some data points that are easily observed by looking at value counts.  We can dive a bit deeper into the data by grouping columns together.First off, we will be querying various keywords in each column, so let's make a simple helper function so we can type less repetitive commands:<jupyter_code>def query_df(column, keyword):
    """Fuzzy match a dataframe column by a keyword
    Return a clean dataframe by dropping columns with all NaNs.
    """
    return df.loc[df[column].str.lower() \
                            .str.contains(keyword, na=False)] \
                            .dropna(how='all', axis=1)<jupyter_output><empty_output><jupyter_text>Using this function we can query data we saw in the value counts to the full artwork data. Exploring Artist WorksFor instance, let's take a look at all the artwork whose manufacturer is Braun:<jupyter_code># Query artwork manufactured by Braun
query_df('manufacturer', 'braun').head()<jupyter_output><empty_output><jupyter_text>[<NAME>](https://en.wikipedia.org/wiki/Dieter_Rams) is a famous industrial designer in the modern design world known for products he designed for the Braun company.  Let's see what artwork they have by him:<jupyter_code>dieter_rams = query_df('artist', '<NAME>')
dieter_rams.head()<jupyter_output><empty_output><jupyter_text>We can group the artworks by how they was added to the collection by referring to the "credit" column:<jupyter_code>dieter_rams['credit'].value_counts()<jupyter_output><empty_output><jupyter_text>Let's see artworks printed by the Mourlot company:<jupyter_code>mourlot = query_df('printer', 'mourlot')
mourlot<jupyter_output><empty_output><jupyter_text>There are 425 artworks printed by the Mourlot company.  An interesting angle to look at this data is to investigate __how the artworks printed by Mourlot were acquired__ (by using the `credit` column), and __how many of each artist's work were acquired from the credited source__.<jupyter_code># Group the dataframe by donor and artist, then sort the result.
group = mourlot.groupby(['credit','artist'])['id']
group.count().reset_index().sort_values(by='id', ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Based on the query, the top three artworks printed by the Mourlot company acquired from the Lous E. Stern Collection were works by <NAME>, <NAME> and <NAME>.  Step 3: Visualization This section shows the distribution of copyright owners among the scraped artwork data. Cleaning the DataBefore visualization, I noticed that the copyright claims are prefixed by the year.  Since I am only interested in the group entities, I will first clean the data by removing the copyright year string.<jupyter_code># This is how the data looks like at the time of scraping
df['copyright'].dropna().head(10)
# Remove the copyright year from the strings
cp = df['copyright'].str.extract('(?<=© [0-9]{4} )(.*)(?=/.*)')
# Concatenate the raw count and the relative frequency,
# and reset the index
cp_count = cp[0].value_counts(dropna=True)
cp_freq = cp[0].value_counts(normalize=True, dropna=True)
cp_concat = pd.concat([cp_count, cp_freq], keys=['count','frequency'], axis=1) \
              .reset_index() \
              .rename(columns={'index': 'copyright'})<jupyter_output><empty_output><jupyter_text>Now we have a workable dataframe with more meaningful value counts<jupyter_code>cp_concat.head(10)<jupyter_output><empty_output><jupyter_text>Plotting the DataWe can now transform the dataframe into a plot using matplotlib.<jupyter_code># Plotting all the data is too long of a tail, shorten to top 20
cp_20 = cp_concat.head(20)

fig = plt.figure(figsize=(5,5)) 
plt.barh(cp_20['copyright'], width=cp_20['frequency']);
plt.title('Distribution of the Top 20 Copyright Owners from Scraped Artwork Data')
plt.ylabel('Copyright')
plt.xlabel('Relative Frequency')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08786446020488574, 'fraction_numerical': 0.028368794326241134, 'mean_word_length': 3.9095744680851063, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 67, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '491207', 'license': 'MIT', 'n_tokens_mistral': 3142, 'n_tokens_neox': 2853, 'n_words': 1148, 'path': 'final_project/moma_scrape_exploration.ipynb', 'repo_name': 'tnakatani/python2_ccac', 'size': '10260'}