<jupyter_start><jupyter_text>Revision history of Wikipedia articles Inspired by <NAME>'s "Who writes Wikipedia?" Work in progress...<jupyter_code># Import classes used in this notebook

import urllib.request
import re
import pandas as pd

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn')
def GetRevisions(pageTitle):
    
    # Format is xml because it's preferred for 'rvcontinue'. See https://www.mediawiki.org/wiki/API:Revisions
    url = "https://en.wikipedia.org/w/api.php?action=query&format=xml&prop=revisions&rvprop=timestamp|user|size&rvlimit=500&titles=" + pageTitle
    revisions = [] # list of all accumulated revisions
    next = '' # information for the next request
    
    while True:
        response = urllib.request.urlopen(url + next).read().decode('utf-8') # web request, error without .decode('utf-8')
        revisions += re.findall('<rev [^>]*>', response) # adds all revisions from the current request to the list

        cont = re.search('<continue rvcontinue="([^"]+)"', response)
        if not cont: # break the loop if 'continue' element missing
            break

        next = "&rvcontinue=" + cont.group(1) # gets the revision Id from which to start the next request

    return revisions;

# Could the RegEx string parsing from result_to_dataframe() be incorporated into this already?<jupyter_output><empty_output><jupyter_text>Insert Wikipedia page title belowInsert the article title of your choosing in the GetRevisions() function below. Check you have the corroct page title! It's case sensitive and spaces are underscores. You can take the title from the page URL. For example, for https://en.wikipedia.org/wiki/Aaron_Swartz the page title is "Aaron_Swartz".So to get the revisions history for the article put it into the function in the cell below in quotation marks like so:```pythonGetRevisions("Aaron_Swartz")```Then run all subsequent cells or click 'Kernel' in the menu bar and select 'Restart Kernel and Run All Cells...'Depending on the article size it may take several seconds to gather all the data.<jupyter_code>revisions = GetRevisions("Diabetes_mellitus") 

# Wikipedia article titles are case sensitive and spaces are underscores
# Make sure the title corresponds to Wikipedia's title - you can take it from the URL
# e.g. "Diabetes" is actually "Diabetes_mellitus", "Diabetes" returns differnt results<jupyter_output><empty_output><jupyter_text>Number of article revisions to date:<jupyter_code># API returns a list of strings so the number of items in the list == number of revisions
len(revisions) 
def result_to_dataframe():
    
    """Loops over the the 'revisions' list and extracts
        user, timestamp, revision size values. 
    Returns: 
        pandas dataframe"""
    
    # Initiate an empty list of list for future dataframe
    data = [[],[],[]]
    # These will be column headers of the returned dataframe
    headers = ['user', 'time', 'size']
    
    for i in revisions:
        
        # Find username to extract 
        userMatch = re.search('user="(.+?)"', i)
        # Catch missing values throwing AttributeError
        if userMatch is not None:
            user = userMatch.group(1)
        else:
            user = ""
        # Add result to the first list of lists in 'data'
        data[0].append(user)
        
        # Find timestamp to extract
        timeMatch = re.search('timestamp="(.+?)"', i)
        # Catch missing values throwing AttributeError
        if timeMatch is not None:
            time = timeMatch.group(1)
        else:
            time = "" # how is this handled in the chart?
        # Add result to the second list of lists in 'data'
        data[1].append(time)
        
        # Find revision size to extract
        sizeMatch = re.search('size="(.+?)"', i)
        # Catch missing values throwing AttributeError
        if sizeMatch is not None:
            size = sizeMatch.group(1)
        else:
            size = "" # how to handle this is the chart?
        # Add result to the third list of lists in 'data'
        data[2].append(size)
        
    return pd.DataFrame(dict(zip(headers, data))) 
    
    # NOTE: Alternative return: pd.DataFrame(data=data).T
    # But simply transposing the dataframe with the transpose method
    # would only work if we didn't assign the headers in the same step
    # therefore the (dict(zip(____,____))) is preferable here
# Can this be incorporated into the 'return' in the function above?
df = result_to_dataframe()
df.set_index('time', inplace=True);
df.index = pd.to_datetime(df.index, infer_datetime_format=True)
df['size'] = pd.to_numeric(df['size']) # size is article length after revision, not size of revision
df.sort_index();
df['edit'] = df['size'] - df['size'].shift(-1)
# Just some formatting parameters
plt.figure(figsize=(16,9)) # Increase the plot size to 16 x 9
plt.title('Article length over time - edit by edit', color='grey', size='xx-large') 
plt.xticks(size='large', color='grey') # Grey color to show on my dark theme and also white background
plt.yticks(size='large', color='grey') # Units are bytes

y = df['size']
x = df.index

plt.plot(x,y);
df2 = df.resample('A').max()

plt.figure(figsize=(16,9)) # Increase the plot size to 16 x 9
plt.title('Article length over time - year by year', color='grey', size='xx-large') 
plt.xticks(size='large', color='grey') # Grey color to show on my dark theme and also white background
plt.yticks(size='large', color='grey') # Units are bytes

y2 = df2['size']
x2 = df2.index

plt.plot(x2,y2);

# Idea: Add four or more figures: actual, H, D, W, M, A
# Or H to A in different colour lines in the same figure<jupyter_output><empty_output><jupyter_text>Top ten vandalisms<jupyter_code>df.sort_values(by=['edit'],ascending=True, inplace=False).head(10)<jupyter_output><empty_output><jupyter_text>Reversals of vandalisms<jupyter_code>df.sort_values(by=['edit'],ascending=False, inplace=False).head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1078234704112337, 'fraction_numerical': 0.006018054162487462, 'mean_word_length': 3.6851996867658574, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '506371', 'license': 'MIT', 'n_tokens_mistral': 1821, 'n_tokens_neox': 1726, 'n_words': 633, 'path': 'MediaWikiAPIrevisions.ipynb', 'repo_name': 'janilles/wikiRevHist', 'size': '5988'}