<jupyter_start><jupyter_text>Exploring the user dataset.<jupyter_code>val user_data = sc.textFile("./u.user")
user_data.first()<jupyter_output><empty_output><jupyter_text>Since | is a regular expression you will need to split by escaping that<jupyter_code>val user_fields = user_data.map(_.split("\\|"))<jupyter_output><empty_output><jupyter_text>We will then count the number of users, genders, occupations, and ZIP codes.We can achieve this by running the following code in the console, line by line.Note that we do not cache the data, as it is unnecessary for this small size:<jupyter_code>val num_users = user_fields.map(_(0)).count()
val num_genders = user_fields.map(_(2)).distinct().count()
val num_occupations = user_fields.map(_(3)).distinct().count()
val num_zipcodes = user_fields.map(_(4)).distinct().count()
println(s"Users: $num_users, genders: $num_genders, occupations: $num_occupations, ZIP codes: $num_zipcodes")<jupyter_output>Users: 943, genders: 2, occupations: 21, ZIP codes: 795<jupyter_text>Next, we will create a histogram to analyze the distribution of user ages:<jupyter_code>case class Distribution(nBins: Int, data: List[Double]) {
  require(data.length > nBins)

  val Epsilon = 0.000001
  val (max,min) = (data.max,data.min)
  val binWidth = (max - min) / nBins + Epsilon
  val bounds = (1 to nBins).map { x => min + binWidth * x }.toList

  def histo(bounds: List[Double], data: List[Double]): List[List[Double]] =
    bounds match {
      case h :: Nil => List(data)
      case h :: t   => val (l,r) = data.partition( _ < h) ; l :: histo(t,r)
    }

  val histogram = histo(bounds, data)
}
val data = Array.tabulate(100){ _ => scala.util.Random.nextDouble * 10 }
val h = Distribution(5, data.toList).histogram
val tabulated = h.map {_.size}<jupyter_output><empty_output><jupyter_text>Exploring the movie dataset<jupyter_code>val movie_data = sc.textFile("/home/jovyan/data/u.item")
movie_data.first()
val num_movies = movie_data.count()
def convert_year<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13392857142857142, 'fraction_numerical': 0.013392857142857142, 'mean_word_length': 5.506451612903226, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '627692', 'license': 'Apache-2.0', 'n_tokens_mistral': 732, 'n_tokens_neox': 699, 'n_words': 180, 'path': 'notebooks/movie lens analysis.ipynb', 'repo_name': 'infinite-Joy/spark-scala-tutorial', 'size': '2016'}