<jupyter_start><jupyter_text>Spark Miniproject Stack Overflow is a collaboratively edited question-and-answer site originally focused on programming topics. Because of the variety of features tracked, including a variety of feedback metrics, it allows for some open-ended analysis of user behavior on the site.Stack Exchange (the parent organization) provides an anonymized [data dump](https://archive.org/details/stackexchange), and we'll use Spark to perform data manipulation, analysis, and machine learning on this data set. As a side note, there's also an online data explorer which allows you to query the data interactively.*Consider*: Do we need to use Spark to work with this data set? What are our alternatives?  Workflow Most questions can be done locally, however in some cases you may want to use cloud services. See the appropriate lecture notebooks for information on how to use cloud services.Python example:1. Edit source code in your `main.py` file, classes in a separate `classes.py` (class definitions need to be written in a separate file and then included at runtime).1. Run locally on a chunk using eg. `$SPARK_HOME/bin/spark-submit --py-files src/classes.py src/main.py data/stats results/stats/`1. Run on GCP once your testing and development are done.General tips:* Try `cat output_dir/* | sort -n -t , -k 1.2 -o sorted_output` to concatenate your output files, which will also be in `part-xxxxx` format.* You can access an interactive PySpark shell with `$SPARK_HOME/bin/pyspark`.  Accessing the data The data is available on S3 (`s3://dataincubator-course/spark-stack-data`). There are three sub-folders, `allUsers`, `allPosts`, and `allVotes`, which contain Gzipped XML with the following format:``` html```Data from the much smaller `stats.stackexchange.com` is available in the same format on S3 (`s3://dataincubator-course/spark-stats-data`). This site, Cross-Validated, will be used below in some instances to avoid working with the full data set for every question.The full schema is available as a text file:<jupyter_code>!aws s3 cp s3://dataincubator-course/spark-stats-data/stack_exchange_schema.txt .<jupyter_output>download: s3://dataincubator-course/spark-stats-data/stack_exchange_schema.txt to ./stack_exchange_schema.txt<jupyter_text>You can either get the data by running the appropriate S3 commands in the terminal, or by running this block for the smaller stats data set:<jupyter_code>!mkdir -p spark-stats-data
!aws s3 sync --exclude '*' --include 'all*' s3://dataincubator-course/spark-stats-data/ ./spark-stats-data
!aws s3 sync --exclude '*' --include 'posts*zip' s3://dataincubator-course/spark-stats-data/ ./spark-stats-data<jupyter_output>download: s3://dataincubator-course/spark-stats-data/allPosts/part-00000.xml.gz to spark-stats-data/allPosts/part-00000.xml.gz
download: s3://dataincubator-course/spark-stats-data/allPosts/part-00009.xml.gz to spark-stats-data/allPosts/part-00009.xml.gz
download: s3://dataincubator-course/spark-stats-data/allPosts/part-00008.xml.gz to spark-stats-data/allPosts/part-00008.xml.gz
download: s3://dataincubator-course/spark-stats-data/allPosts/part-00004.xml.gz to spark-stats-data/allPosts/part-00004.xml.gz
download: s3://dataincubator-course/spark-stats-data/allPosts/part-00002.xml.gz to spark-stats-data/allPosts/part-00002.xml.gz
download: s3://dataincubator-course/spark-stats-data/allPosts/part-00007.xml.gz to spark-stats-data/allPosts/part-00007.xml.gz
download: s3://dataincubator-course/spark-stats-data/allPosts/part-00006.xml.gz to spark-stats-data/allPosts/part-00006.xml.gz
download: s3://dataincubator-course/spark-stats-data/allPosts/part-00003.xml.gz to spark-stats-data/allPosts/pa[...]<jupyter_text>And to get the much larger full data set (be warned, this can take 20 or more minutes, so you may want to run it in the terminal to avoid locking up the notebook):<jupyter_code>!mkdir -p spark-stack-data
!aws s3 sync --exclude '*' --include 'all*' s3://dataincubator-course/spark-stack-data/ ./spark-stack-data<jupyter_output>download: s3://dataincubator-course/spark-stack-data/allPosts/part-00001.xml.gz to spark-stack-data/allPosts/part-00001.xml.gz
download: s3://dataincubator-course/spark-stack-data/allPosts/part-00009.xml.gz to spark-stack-data/allPosts/part-00009.xml.gz
download: s3://dataincubator-course/spark-stack-data/allPosts/part-00000.xml.gz to spark-stack-data/allPosts/part-00000.xml.gz
download: s3://dataincubator-course/spark-stack-data/allPosts/part-00005.xml.gz to spark-stack-data/allPosts/part-00005.xml.gz
download: s3://dataincubator-course/spark-stack-data/allPosts/part-00008.xml.gz to spark-stack-data/allPosts/part-00008.xml.gz
download: s3://dataincubator-course/spark-stack-data/allPosts/part-00003.xml.gz to spark-stack-data/allPosts/part-00003.xml.gz
download: s3://dataincubator-course/spark-stack-data/allPosts/part-00002.xml.gz to spark-stack-data/allPosts/part-00002.xml.gz
download: s3://dataincubator-course/spark-stack-data/allPosts/part-00010.xml.gz to spark-stack-data/allPosts/pa[...]<jupyter_text>Data input and parsing Some rows are split across multiple lines; these can be discarded. Incorrectly formatted XML can also be ignored. It is enough to simply skip problematic rows, the loss of data will not significantly impact our results on this large data sets.You will need to handle XML parsing yourself.  Our solution uses `lxml.etree` in Python.To make your code more flexible, it's also recommended to incorporate command-line arguments that specify the location of the input data and where output should be written.The goal should be to have a parsing function that can be applied to the input data to access any XML element desired. It is suggested to use a class structure so that you can create RDDs of Posts, Votes, Users, etc.``` python Command line arguments using sysv or argparse in Pythonif __name__ == '__main__':    main(ARGS.input_dir, ARGS.output_dir)```  Questions  Bad XML A simple question to test your parsing code. Create an RDD of Post objects where each Post is a valid row of XML from the Cross-Validated (stats.stackexchange.com) `allPosts` data set.We are going to take several shortcuts to speed up and simplify our computations.  First, your parsing function should only attempt to parse rows that start with `<row` as these denote actual data entries. This should be done in Spark as the data is being read in from disk, without any pre-Spark processing. Return the total number of XML rows that started with `<row` that were subsequently **rejected** during your processing.  Note that the text is unicode, and contains non-ASCII characters.  You may need to re-encode to UTF-8 (depending on your XML parser)Note that this cleaned data set will be used for all subsequent questions.*Question*: Can you figure out what filters you need to put in place to avoid throwing parsing errors entirely?<jupyter_code>bad_xml = 0

grader.score('spark__bad_xml', bad_xml)<jupyter_output><empty_output><jupyter_text>Favorites and scores We're interested in looking for useful patterns in the data.  If we look at the Post data again (the smaller set, `stats.stackexchange.com`), we see that many things about each post are recorded.  We're going to start by looking to see if there is a relationship between the number of times a post was favorited (the `FavoriteCount`) and the `Score`.  The score is the number of times the post was upvoted minus the number of times it was downvoted, so it is a measure of how much a post was liked.  We'd expect posts with a higher number of favorites to have better scores, since they're both measurements of how good the post is.Let's aggregate posts by the number of favorites, and find the average score for each number of favorites.  Do this for the lowest 50 numbers of favorites._Note:_ Before submitting, take a look at the numbers.  Do they follow the trend you expect?**Checkpoints**- Total score across all posts: 299469- Mean of first 50 favorite counts (averaging the keys themselves): 24.76<jupyter_code>favorite_score = [(0, 2.3398827696988396)]*50

grader.score('spark__favorite_score', favorite_score)<jupyter_output><empty_output><jupyter_text>Answer percentage Investigate the correlation between a user's reputation and the kind of posts they make. For the 99 users with the highest reputation, single out posts which are either questions or answers and look at the percentage of these posts that are answers: *(answers / (answers + questions))*. Return a tuple of their **user ID** and this fraction.You should also return (-1, fraction) to represent the case where you average over all users (so you will return 100 entries total).Again, you only need to run this on the statistics overflow set.  Checkpoints * Total questions: 52,060* Total answers: 55,304* Top 99 users' average reputation: 11893.464646464647<jupyter_code>answer_percentage = [(7071, 0.9107142857142857)] * 100

grader.score('spark__answer_percentage', answer_percentage)<jupyter_output><empty_output><jupyter_text>First question We'd expect the first **question** a user asks to be indicative of their future behavior.  We'll dig more into that in the next problem, but for now let's see the relationship between reputation and how long it took each person to ask their first question.For each user that asked a question, find the difference between when their account was created (`CreationDate` for the User) and when they asked their first question (`CreationDate` for their first question).  Return this time difference in days (round down, so 2.7 days counts as 2 days) for the 100 users with the highest reputation, in the form`(UserId, Days)`**Checkpoints**- Users that asked a question: 23134- Average number of days (round each user's days, then average): 30.1074258<jupyter_code>first_question = [(805, 669)] * 100

grader.score('spark__first_question', first_question)<jupyter_output><empty_output><jupyter_text>Identify veterans It can be interesting to think about what factors influence a user to remain active on the site over a long period of time. In order not to bias the results towards older users, we'll define a time window between 100 and 150 days after account creation. If the user has made a post in this time, we'll consider them active and well on their way to being veterans of the site; if not, they are inactive and were likely brief users.*Consider*: What other parameterizations of "activity" could we use, and how would they differ in terms of splitting our user base?*Consider*: What other biases are still not dealt with, after using the above approach?Let's see if there are differences between the first ever question posts of "veterans" vs. "brief users". For each group separately, average the score, views, number of answers, and number of favorites of the users' **first question**.*Consider*: What story could you tell from these numbers? How do the numbers support it?  Checkpoints * Total brief users: 24,864* Total veteran users: 2,027<jupyter_code>identify_veterans = {
    "vet_score": 0,
    "vet_views": 0,
    "vet_answers": 0,
    "vet_favorites": 0,
    "brief_score": 0,
    "brief_views": 0,
    "brief_answers": 0,
    "brief_favorites": 0
}

grader.score('spark__identify_veterans', identify_veterans)<jupyter_output><empty_output><jupyter_text>Identify veterans&mdash;full Same as above, but on the full Stack Exchange data set.No pre-parsed data is available for this question.  Checkpoints * Total brief users: 1,848,628* Total veteran users: 288,285<jupyter_code>identify_veterans_full = {
    "vet_score": 0,
    "vet_views": 0,
    "vet_answers": 0,
    "vet_favorites": 0,
    "brief_score": 0,
    "brief_views": 0,
    "brief_answers": 0,
    "brief_favorites": 0
}

grader.score('spark__identify_veterans_full', identify_veterans_full)<jupyter_output><empty_output><jupyter_text>Word2vec Word2Vec is an alternative approach for vectorizing text data. The vectorized representations of words in the vocabulary tend to be useful for predicting other words in the document, hence the famous example "vector('king') - vector('man') + vector('woman') ~= vector('queen')".Let's see how good a Word2Vec model we can train using the **tags** of each Stack Exchange post as documents (this uses the full data set). Use the implementation of Word2Vec from Spark ML (this will require using DataFrames) to return a list of the top 25 closest synonyms to "ggplot2" and their similarity score in tuple format ("string", number).The tags appear in the data as one string, you will need to separate them into individual tags. There is no need to further parse them beyond separating them.  Parameters The dimensionality of the vector space should be 100. The random seed should be 42 in `PySpark`.  Checkpoints * Mean of the top 25 cosine similarities: 0.8012362027168274<jupyter_code>word2vec = [("data.frame", 0.772650957107544)] * 25

grader.score('spark__word2vec', word2vec)<jupyter_output><empty_output><jupyter_text>Classification We'd like to see if we can predict the tags of a **question** from its body text. Instead of predicting specific tags, we will instead try to predict if a question contains one of the top ten most common tags.  To this end, we have separated out a train and a test set from the original data.  The training and tests sets were downloaded with the stats data at the beginning of the notebook.  You can also get them from S3:  * `s3://dataincubator-course/spark-stats-data/posts_train.zip`  * `s3://dataincubator-course/spark-stats-data/posts_test.zip`This will involve two steps: first, find the ten most common tags for questions in the training data set (the tags have been removed from the test set). Then train a learner to predict from the text of the question (the `Body` attribute) if it should have one of those ten tags in it - you will need to process the question text with NLP techniques such as splitting the text into tokens.Since we can't reliably pickle Spark models, instead return a list of your predictions, sorted by the question's `Id`.  This sorting is very important, as our grader expects the results to be submitted in a particular order. These predictions should be `0` if the question isn't expected to have a tag in the top ten, and `1` if it is.As an example, if our top tags include `spark` and `python`, and we had the following questions:```" ... />" ... />" ... />```We would expect to return `[0, 1, 1]` (for the order `[723, 1740, 2740]`).  You may need to do some format manipulation in your DataFrame to get this.  Checkpoints - Number of training posts with a tag in the top 10: `22525`- Number without: `19540`<jupyter_code>!unzip -d spark-stats-data/train spark-stats-data/posts_train.zip
!unzip -d spark-stats-data/test spark-stats-data/posts_test.zip

classification = [0] * 4649

grader.score('spark__classification', classification)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08466957396846696, 'fraction_numerical': 0.02972156994297216, 'mean_word_length': 4.87776025236593, 'pattern_counts': {'":': 16, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 34}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '224039', 'license': 'MIT', 'n_tokens_mistral': 4642, 'n_tokens_neox': 4188, 'n_words': 1823, 'path': 'XML_ProblemStatements.ipynb', 'repo_name': 'manoharkaranth/XML_Parsing_Spark', 'size': '14905'}