<commit_before>// Databricks notebook source exported at Mon, 22 Feb 2016 04:23:15 UTC
// MAGIC %md
// MAGIC 
// MAGIC # [Scalable Data Science](http://www.math.canterbury.ac.nz/~r.sainudiin/courses/ScalableDataScience/)
// MAGIC 
// MAGIC 
// MAGIC ### prepared by [<NAME>](https://nz.linkedin.com/in/raazesh-sainudiin-45955845) and [<NAME>](https://www.linkedin.com/in/sivanand)
// MAGIC 
// MAGIC *supported by* [![](https://raw.githubusercontent.com/raazesh-sainudiin/scalable-data-science/master/images/databricks_logoTM_200px.png)](https://databricks.com/)
// MAGIC and 
// MAGIC [![](https://raw.githubusercontent.com/raazesh-sainudiin/scalable-data-science/master/images/AWS_logoTM_200px.png)](https://www.awseducate.com/microsite/CommunitiesEngageHome)

// COMMAND ----------

// MAGIC %md 
// MAGIC # What is Scalable [Data Science](https://en.wikipedia.org/wiki/Data_science) in one picture?
// MAGIC 
// MAGIC ![what is sds?](http://www.math.canterbury.ac.nz/~r.sainudiin/courses/ScalableDataScience/sds.png "sds")

// COMMAND ----------

// MAGIC %md
// MAGIC Source: [Vasant Dhar, Data Science and Prediction, Communications of the ACM, Vol. 56 (1). p. 64, DOI:10.1145/2500499](http://dl.acm.org/citation.cfm?id=2500499)
// MAGIC 
// MAGIC ### key insights
// MAGIC * Data Science is the study of *the generalizabile extraction of knowledge from data*.
// MAGIC * A common epistemic requirement in assessing whether new knowledge is actionable for decision making is its predictive power, not just its ability to explain the past.
// MAGIC * A *data scientist requires an integrated skill set spanning* 
// MAGIC   * mathematics, 
// MAGIC   * machine learning, 
// MAGIC   * artificial intelligence, 
// MAGIC   * statistics, 
// MAGIC   * databases, and 
// MAGIC   * optimization, 
// MAGIC   * along with a deep understanding of the craft of problem formulation to engineer effective solutions.
// MAGIC   
// MAGIC Source: [Machine learning: Trends, perspectives, and prospects, <NAME>, <NAME>, Science  17 Jul 2015: Vol. 349, Issue 6245, pp. 255-260, DOI: 10.1126/science.aaa8415](http://science.sciencemag.org/content/349/6245/255.full-text.pdf+html)
// MAGIC 
// MAGIC ### key insights
// MAGIC * ML is concerned with the building of computers that improve automatically through experience
// MAGIC * ML lies at the intersection of computer science and statistics and at the core of artificial intelligence and data science
// MAGIC * Recent progress in ML is due to:
// MAGIC   * development of new algorithms and theory
// MAGIC   * ongoing explosion in the availability of online data
// MAGIC   * availability of low-cost computation (through clusters of commodity hardware in the *cloud*)
// MAGIC * The adoption of data science and ML methods is leading to more evidence-based decision-making across:
// MAGIC   * health sciences (neuroscience research, )
// MAGIC   * manufacturing
// MAGIC   * robotics (autonomous vehicle)
// MAGIC   * vision, speech processing, natural language processing
// MAGIC   * education
// MAGIC   * financial modeling
// MAGIC   * policing
// MAGIC   * marketing

// COMMAND ----------

// MAGIC %md
// MAGIC ## Standing on shoulders of giants!
// MAGIC 
// MAGIC This course will build on two other edX courses where needed.  
// MAGIC * [BerkeleyX/CS100-1x, Introduction to Big Data Using Apache Spark by <NAME>, Chancellor's Professor, UC Berkeley](https://www.edx.org/course/introduction-big-data-apache-spark-uc-berkeleyx-cs100-1x)
// MAGIC * [BerkeleyX/CS190-1x, Scalable Machine Learning by <NAME>, Ass. Prof., UC Los Angeles](https://www.edx.org/course/scalable-machine-learning-uc-berkeleyx-cs190-1x)
// MAGIC 
// MAGIC We encourage you to take these courses if you have more time.  For those of you (including the course coordinator) who have taken these courses formally in 2015, this course will be an *expanded scala version* with an emphasis on *individualized course project* as opposed to completing labs that test sytactic skills. 
// MAGIC 
// MAGIC We will also be borrowing more theoretical aspects from the following course:
// MAGIC * [Stanford/CME323, Distributed Algorithms and Optimization by <NAME>, Ass. Prof., Institute for Computational and Mathematical Engineering, Stanford Univ.](http://stanford.edu/~rezab/dao/)
// MAGIC 
// MAGIC The two recommended readings are (already somewhat outdated!):
// MAGIC * Learning Spark : lightning-fast data analytics by <NAME>, <NAME>, <NAME>, and <NAME>, O'Reilly, 2015.
// MAGIC * Advanced analytics with Spark : patterns for learning from data at scale, O'Reilly, 2015.
// MAGIC 
// MAGIC ![LS](http://www.syndetics.com/index.aspx?type=xw12&isbn=9781449358624/LC.GIF&client=ucanterburyl&upc=&oclc=) and  ![aas](http://t3.gstatic.com/images?q=tbn:ANd9GcSQs35NvHVozz77dhXYc2Ce8lKyJkR3oVwaxyA5Ub4W7Kvtvf9i "aas")
// MAGIC 
// MAGIC **These are available at [UC Library](http://ipac.canterbury.ac.nz/ipac20/ipac.jsp?profile=a&npp=30&index=.CC&term=STAT478)**

// COMMAND ----------

// MAGIC %md
// MAGIC # What should you be able to do at the end of this course?
// MAGIC 
// MAGIC ## Understand the principles of fault-tolerant scalable computing using Apache Spark
// MAGIC 
// MAGIC * in-memory and generic DAG extensions of Map-reduce
// MAGIC * resilient distributed datasets for fault-tolerance
// MAGIC * skills to process today's big data using state-of-the art techniques in Apache Spark 1.6, in terms of:
// MAGIC   * hands-on coding with real datasets
// MAGIC   * an intuitive (non-mathematical) understanding of the ideas behind the technology and methods
// MAGIC   * pointers to academic papers in the literature, technical blogs and video streams for *you to futher your theoretical understanding*.
// MAGIC 
// MAGIC # More concretely, you will be able to:
// MAGIC ### 1. Extract, Transform, Load, Interact, Explore and Analyze Data
// MAGIC 
// MAGIC #### Exploring Apache Web Logs (semi-structured data)
// MAGIC [![Databricks jump start](https://raw.githubusercontent.com/raazesh-sainudiin/scalable-data-science/master/images/dataExploreWebLogsSQL.png)](https://vimeo.com/137874931)
// MAGIC 
// MAGIC #### Exploring Wikipedia Click Streams (structured data)
// MAGIC [![Michael Armbrust Spark Summit East](http://img.youtube.com/vi/35Y-rqSMCCA/0.jpg)](https://www.youtube.com/v/35Y-rqSMCCA)
// MAGIC 
// MAGIC 
// MAGIC ### 2. Build Scalable Machine Learning Pipelines
// MAGIC 
// MAGIC ### Apply standard learning methods (we will choose from this list)
// MAGIC * Supervised Learning Mehtods: Regression /Classification
// MAGIC * Unsupervised Learning Methods: Clustering
// MAGIC * Recommedation systems
// MAGIC * Streaming
// MAGIC * Graph processing
// MAGIC * Geospatial data-processing
// MAGIC * Topic modeling
// MAGIC * Deep Learning
// MAGIC * ...
// MAGIC 
// MAGIC 
// MAGIC ####  Spark Summit 2015 demo: Creating an end-to-end machine learning data pipeline with Databricks (Live Sentiment Analysis)
// MAGIC [![<NAME>'s :ive Sentiment Analysist](http://img.youtube.com/vi/NR1MYg_7oSg/0.jpg)](https://www.youtube.com/v/NR1MYg_7oSg)

// COMMAND ----------

// MAGIC %md
// MAGIC # How will you be assessed?
// MAGIC 
// MAGIC The course is extremely hands-on and therefore gives 50% of the final grade for attending each lab and completing it. Completing a lab essentially involves going through the cells in the cloned notebooks for each week to strengthen your understanding. This will ensure that the concept as well as the syntax is understood for the learning outcomes for each week. There are additional videos and exercises you are encouraged to watch/complete.  These additional exercises will not be graded. You may use 1800-1830 hours to ask any questions about the contents in the current or previous weeks. 
// MAGIC 
// MAGIC Each student will be working on a course project and present the findings to the class in the last week or two. The course project will be given 50% of the grade. The project will typically involve applying Spark on a publicly available dataset or writing a report to demonstrate in-depth understanding of appropriate literature of interest to the student’s immediate research goals in academia or industry. Oral presentation of the project will constitute 10% of the grade. The remaining 40% of the grade will be for the written part of the project, which will be graded for replicability and ease of understanding. The written report will be encouraged for publication in a technical blog format in public repositories such as GitHub through a publishable mark-down'd databricks notebook (this is intended to show-case the actual skills of the student to potential employers directly). Group work on projects may be considered for complex projects.
<commit_msg>project cluster added<commit_after>// Databricks notebook source exported at Wed, 24 Feb 2016 04:06:27 UTC
// MAGIC %md
// MAGIC 
// MAGIC # [Scalable Data Science](http://www.math.canterbury.ac.nz/~r.sainudiin/courses/ScalableDataScience/)
// MAGIC 
// MAGIC 
// MAGIC ### prepared by [<NAME>](https://nz.linkedin.com/in/raazesh-sainudiin-45955845) and [<NAME>](https://www.linkedin.com/in/sivanand)
// MAGIC 
// MAGIC *supported by* [![](https://raw.githubusercontent.com/raazesh-sainudiin/scalable-data-science/master/images/databricks_logoTM_200px.png)](https://databricks.com/)
// MAGIC and 
// MAGIC [![](https://raw.githubusercontent.com/raazesh-sainudiin/scalable-data-science/master/images/AWS_logoTM_200px.png)](https://www.awseducate.com/microsite/CommunitiesEngageHome)

// COMMAND ----------

// MAGIC %md 
// MAGIC # What is Scalable [Data Science](https://en.wikipedia.org/wiki/Data_science) in one picture?
// MAGIC 
// MAGIC ![what is sds?](http://www.math.canterbury.ac.nz/~r.sainudiin/courses/ScalableDataScience/sds.png "sds")

// COMMAND ----------

// MAGIC %md
// MAGIC Source: [<NAME>, Data Science and Prediction, Communications of the ACM, Vol. 56 (1). p. 64, DOI:10.1145/2500499](http://dl.acm.org/citation.cfm?id=2500499)
// MAGIC 
// MAGIC ### key insights
// MAGIC * Data Science is the study of *the generalizabile extraction of knowledge from data*.
// MAGIC * A common epistemic requirement in assessing whether new knowledge is actionable for decision making is its predictive power, not just its ability to explain the past.
// MAGIC * A *data scientist requires an integrated skill set spanning* 
// MAGIC   * mathematics, 
// MAGIC   * machine learning, 
// MAGIC   * artificial intelligence, 
// MAGIC   * statistics, 
// MAGIC   * databases, and 
// MAGIC   * optimization, 
// MAGIC   * along with a deep understanding of the craft of problem formulation to engineer effective solutions.
// MAGIC   
// MAGIC Source: [Machine learning: Trends, perspectives, and prospects, <NAME>, <NAME>, Science  17 Jul 2015: Vol. 349, Issue 6245, pp. 255-260, DOI: 10.1126/science.aaa8415](http://science.sciencemag.org/content/349/6245/255.full-text.pdf+html)
// MAGIC 
// MAGIC ### key insights
// MAGIC * ML is concerned with the building of computers that improve automatically through experience
// MAGIC * ML lies at the intersection of computer science and statistics and at the core of artificial intelligence and data science
// MAGIC * Recent progress in ML is due to:
// MAGIC   * development of new algorithms and theory
// MAGIC   * ongoing explosion in the availability of online data
// MAGIC   * availability of low-cost computation (through clusters of commodity hardware in the *cloud*)
// MAGIC * The adoption of data science and ML methods is leading to more evidence-based decision-making across:
// MAGIC   * health sciences (neuroscience research, )
// MAGIC   * manufacturing
// MAGIC   * robotics (autonomous vehicle)
// MAGIC   * vision, speech processing, natural language processing
// MAGIC   * education
// MAGIC   * financial modeling
// MAGIC   * policing
// MAGIC   * marketing

// COMMAND ----------

// MAGIC %md
// MAGIC ## Standing on shoulders of giants!
// MAGIC 
// MAGIC This course will build on two other edX courses where needed.  
// MAGIC * [BerkeleyX/CS100-1x, Introduction to Big Data Using Apache Spark by <NAME>, Chancellor's Professor, UC Berkeley](https://www.edx.org/course/introduction-big-data-apache-spark-uc-berkeleyx-cs100-1x)
// MAGIC * [BerkeleyX/CS190-1x, Scalable Machine Learning by <NAME>, Ass. Prof., UC Los Angeles](https://www.edx.org/course/scalable-machine-learning-uc-berkeleyx-cs190-1x)
// MAGIC 
// MAGIC We encourage you to take these courses if you have more time.  For those of you (including the course coordinator) who have taken these courses formally in 2015, this course will be an *expanded scala version* with an emphasis on *individualized course project* as opposed to completing labs that test sytactic skills. 
// MAGIC 
// MAGIC We will also be borrowing more theoretical aspects from the following course:
// MAGIC * [Stanford/CME323, Distributed Algorithms and Optimization by <NAME>, Ass. Prof., Institute for Computational and Mathematical Engineering, Stanford Univ.](http://stanford.edu/~rezab/dao/)
// MAGIC 
// MAGIC The two recommended readings are (already somewhat outdated!):
// MAGIC * Learning Spark : lightning-fast data analytics by <NAME>, <NAME>, <NAME>, and <NAME>, O'Reilly, 2015.
// MAGIC * Advanced analytics with Spark : patterns for learning from data at scale, O'Reilly, 2015.
// MAGIC 
// MAGIC ![LS](http://www.syndetics.com/index.aspx?type=xw12&isbn=9781449358624/LC.GIF&client=ucanterburyl&upc=&oclc=) and  ![aas](http://t3.gstatic.com/images?q=tbn:ANd9GcSQs35NvHVozz77dhXYc2Ce8lKyJkR3oVwaxyA5Ub4W7Kvtvf9i "aas")
// MAGIC 
// MAGIC **These are available at [UC Library](http://ipac.canterbury.ac.nz/ipac20/ipac.jsp?profile=a&npp=30&index=.CC&term=STAT478)**

// COMMAND ----------

// MAGIC %md
// MAGIC # What should you be able to do at the end of this course?
// MAGIC 
// MAGIC ## Understand the principles of fault-tolerant scalable computing using Apache Spark
// MAGIC 
// MAGIC * in-memory and generic DAG extensions of Map-reduce
// MAGIC * resilient distributed datasets for fault-tolerance
// MAGIC * skills to process today's big data using state-of-the art techniques in Apache Spark 1.6, in terms of:
// MAGIC   * hands-on coding with real datasets
// MAGIC   * an intuitive (non-mathematical) understanding of the ideas behind the technology and methods
// MAGIC   * pointers to academic papers in the literature, technical blogs and video streams for *you to futher your theoretical understanding*.
// MAGIC 
// MAGIC # More concretely, you will be able to:
// MAGIC ### 1. Extract, Transform, Load, Interact, Explore and Analyze Data
// MAGIC 
// MAGIC #### Exploring Apache Web Logs (semi-structured data)
// MAGIC [![Databricks jump start](https://raw.githubusercontent.com/raazesh-sainudiin/scalable-data-science/master/images/dataExploreWebLogsSQL.png)](https://vimeo.com/137874931)
// MAGIC 
// MAGIC #### Exploring Wikipedia Click Streams (structured data)
// MAGIC [![Michael Armbrust Spark Summit East](http://img.youtube.com/vi/35Y-rqSMCCA/0.jpg)](https://www.youtube.com/v/35Y-rqSMCCA)
// MAGIC 
// MAGIC 
// MAGIC ### 2. Build Scalable Machine Learning Pipelines
// MAGIC 
// MAGIC ### Apply standard learning methods (we will choose from this list)
// MAGIC * Supervised Learning Mehtods: Regression /Classification
// MAGIC * Unsupervised Learning Methods: Clustering
// MAGIC * Recommedation systems
// MAGIC * Streaming
// MAGIC * Graph processing
// MAGIC * Geospatial data-processing
// MAGIC * Topic modeling
// MAGIC * Deep Learning
// MAGIC * ...
// MAGIC 
// MAGIC 
// MAGIC ####  Spark Summit 2015 demo: Creating an end-to-end machine learning data pipeline with Databricks (Live Sentiment Analysis)
// MAGIC [![<NAME>'s :ive Sentiment Analysist](http://img.youtube.com/vi/NR1MYg_7oSg/0.jpg)](https://www.youtube.com/v/NR1MYg_7oSg)

// COMMAND ----------

// MAGIC %md
// MAGIC # How will you be assessed?
// MAGIC 
// MAGIC The course is extremely hands-on and therefore gives 50% of the final grade for attending each lab and completing it. Completing a lab essentially involves going through the cells in the cloned notebooks for each week to strengthen your understanding. This will ensure that the concept as well as the syntax is understood for the learning outcomes for each week. There are additional videos and exercises you are encouraged to watch/complete.  These additional exercises will not be graded. You may use 1800-1830 hours to ask any questions about the contents in the current or previous weeks. 
// MAGIC 
// MAGIC Each student will be working on a course project and present the findings to the class in the last week or two. The course project will be done in [http://www.math.canterbury.ac.nz/databricks-projects](http://www.math.canterbury.ac.nz/databricks-projects) and count towards 50% of the grade. The project will typically involve applying Spark on a publicly available dataset or writing a report to demonstrate in-depth understanding of appropriate literature of interest to the student’s immediate research goals in academia or industry. Oral presentation of the project will constitute 10% of the grade. The remaining 40% of the grade will be for the written part of the project, which will be graded for replicability and ease of understanding. The written report will be encouraged for publication in a technical blog format in public repositories such as GitHub through a publishable mark-down'd databricks notebook (this is intended to show-case the actual skills of the student to potential employers directly). Group work on projects may be considered for complex projects.

{'fraction_non_alphanumeric': 0.10890977018000679, 'fraction_numerical': 0.02252915204347334, 'mean_word_length': 3.8602475928473177, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 26, 'lorem ipsum': 0, 'www.': 20, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2364581', 'n_tokens_mistral': 5754, 'n_tokens_neox': 5014, 'n_words': 1950}