<jupyter_start><jupyter_text>STAT 480 Fall 2019 On-Computer Assignment 2 Due Date and Submission 23:59 Wednesday, September 25, 2019 Submit by Exporting to HTML, zipping, and uploading to Compass  Students Enter Name and NetID Here Name: <NAME> NetID: yihanj2  Verify Execution and Check Spark Version<jupyter_code>print('PageRank Assignment')
print('Spark Version: ', sc.version)<jupyter_output>PageRank Assignment<jupyter_text>Students Place Access String Secret Between Quotes in Cell below<jupyter_code>access_string = "?st=2019-08-27T14%3A37%3A38Z&se=2019-12-31T14%3A37%3A00Z&sp=rl&sv=2018-03-28&sr=c&sig=chJc88%2FCIGzKaLjqVn0sIL2NtmLQvgAAk%2FYbYECHQZw%3D"<jupyter_output><empty_output><jupyter_text>Code seen in previous assignments that does not need to be changed.<jupyter_code>import numpy as np
import pandas as pd
# Enable Arrow-based columnar data transfers
spark.conf.set("spark.sql.execution.arrow.enabled", "true")

spark.conf.set( 
  "fs.azure.sas.stat480-data-blobs.stat480storageaccount.blob.core.windows.net",
  access_string)
# using _df to indicate a Spark dataframe, as opposed to a Pandas dataframe
clickstream_df = spark.read.parquet("wasbs://stat480-data-blobs@stat480storageaccount.blob.core.windows.net/WikipediaClickstream/clickstream-enwiki-2019-07.parquet")
print('The dataframe has ', clickstream_df.count(), ' records.')
print('Note: some records have text encoding issues that the instructor will address in the future.')
print('Note: some records in the original dataset may not have been handled resulting in a different record count.')
from pyspark.sql import functions
def top_df_2(df, key_col, k):
    """
    Using limit(k) and then filtering.  Handles ties OK."
    """
    num_records = df.count()
    value_at_k_rank = (df
                       .orderBy(functions.col(key_col).desc())
                       .limit(k)
                       .select(functions.min(key_col).alias('min'))
                       .first()['min'])

    return df.filter(df[key_col] >= value_at_k_rank)<jupyter_output><empty_output><jupyter_text>Students may wish to decrease the number of records processed at first for convenience and to save time. Ultimately, set number_of_records = 100000.<jupyter_code>number_of_records = 100000
clickstream_pdf = top_df_2(clickstream_df, key_col='n', k=number_of_records).toPandas()<jupyter_output><empty_output><jupyter_text>Part I: In the next cell or cells, create an adjacency matrix m using data from clickstream_pdf- This is the portion of the assignment where most of the programming work is necessary- That said, it is likely that solutions for this portion of the assignment can be 15 lines of code or fewer Notes- Restrict attention to records of type 'link'. For these, both the referrer and the resources, that is, both prev and cur, will correspond to Wikipedia pages- After the restriction has been applied, the 'link' column can be removed.- Also, the 'n' column can be removed.- Furthermore, restrict attention to pages that appear both as referrers and as resources, that is, in boht the prev and cur columns- Python, pandas, and numPy ideas to consider utilizing inlcude but are not limited to:  - drop, including specification of axis  - reset_index  - the pandas countepart to https://stat.ethz.ch/R-manual/R-devel/library/base/html/table.html in R  - list(), set()  - indexing on columns by label and on rows by label  - a way to get the values of a pandas dataframe into a numPy array  - a way to sum across an axis of a numPy array<jupyter_code>df1 = clickstream_pdf.loc[clickstream_pdf.type == "link"]
df2 = df1[['prev', 'cur']]
cur_list = df2['cur'].values
prev_list = df2['prev'].values
def intersection(lst1, lst2): 
    lst3 = [value for value in lst1 if value in lst2] 
    return lst3 
idx = intersection(cur_list, prev_list )
idx = np.unique(idx)
df3 = df2[df2['cur'].isin(idx) & df2['prev'].isin(idx)]
df = pd.crosstab(df3.prev, df3.cur)
import networkx as nx
m = nx.DiGraph()
for i, row in df3.iterrows():
    m.add_edge(row['prev'], row['cur'])<jupyter_output><empty_output><jupyter_text>Part II - Define a Python function that computes PageRank values for all pages represented as the rows (and the columns) of a square adjaceny matrix- You may write this yourself, or borrow from another source, such as https://en.wikipedia.org/wiki/PageRankPython as long as you cite the source.- The function should have or accept:  - pagerank as its name  - an adjacency matrix as the first parameter  - epsilon, the quadratic error term in the PageRank algorithm, as the second parameter  - the damping factor in the PageRank algorithm as the third parameter- The function should return a one-dimensional numPy array with PR(E), that is, the PageRank value, for each page, in the same order as the pages appeared in the rows (and the columns) of the adjacency matrix<jupyter_code>def pagerank(M, eps=1.0e-8, d=0.85):
  pr = nx.pagerank(M, alpha=d, tol=eps)
  pr = pr.values()
  return pr<jupyter_output><empty_output><jupyter_text>Part III - Invoke the pagerank function, passing it the adjacency matrix you created, displaying the results in descending order Notes- Part III should be very straightforward- Consider using these default values: eps=1.0e-8, d=0.85- Consider creating a new pandas data frame that holds results  - Consider creating a new index that is a renamed version of the index of an intermediate dataframe from Part I. This will make it easier to display the page along with its PageRank- The conventional way of displaying a dataframe is sufficient - it is OK that not all pages and their page ranks are displayed<jupyter_code>pr = pagerank(m)
def index(M, eps=1.0e-8, d=0.85):
  idx = nx.pagerank(M, alpha=d, tol=eps)
  return idx
df_result = pd.DataFrame()
idx = index(m)
idx = idx.keys()
df_result['Pages'] = idx
df_result['PageRanks'] = pr
df_PR = df_result.set_index(['Pages'])
df_PR.sort_values(["PageRanks"], ascending=False )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09130507622717374, 'fraction_numerical': 0.022784385994303904, 'mean_word_length': 4.734870317002882, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '290916', 'license': 'MIT', 'n_tokens_mistral': 1941, 'n_tokens_neox': 1817, 'n_words': 678, 'path': 'course3/.ipynb_checkpoints/STAT480_Assignment_2 (3)-checkpoint.ipynb', 'repo_name': 'Cuilie/Code-for-algorithm-stanford-coursera', 'size': '5974'}