<jupyter_start><jupyter_text>Scala Basics  IntroductionScala combines object-oriented and functional programming in one concise, high-level language. Scala's static types help avoid bugs in complex applications, and its JVM and JavaScript runtimes let you build high-performance systems with easy access to huge ecosystems of libraries.  Overview - Statically Typed (Java, C, C++, Scala)- Object-oriented programming (OOP) paradigm- Scala code results in .class files that run on the Java Virtual Machine- Like Java, Scala also uses a curly-brace syntax.- Easy to use Java libraries in Scala Creating variables in Scala. There are two types of variables in Scala. Val: creates an immutable varaibleVar creates a mutable variableThis is what declaring a variable in Scala looks like!<jupyter_code>val x = 2
var y = 3
x = 3
y = 4<jupyter_output><empty_output><jupyter_text>So here, we can see that I was unable to reassign the value of x because it was declared type val. However, if it is type val like variable y is, reassigning the value is no issue because it is mutable.  Data TypesScala contains the same data types as Java such as:- Boolean- Byte- Short- Char- Int- Long- Float- Double- StringHowever, Scala does not need to specify a type. The scala complier is similar to Python here where the complier can determine the data type. From the example above, we can see that x and y were not declared of INT type but when we ran it, the output showed both x and y as INT.  For loops examples<jupyter_code>for (i <- 0 to 5) println(i)<jupyter_output>0
1
2
3
4
5<jupyter_text>Example expression that iterates over a list of strings:<jupyter_code>val fruits = List("apple", "banana", "lime", "orange")

val fruitLengths = for {
    f <- fruits
    if f.length > 4
} yield f.length<jupyter_output><empty_output><jupyter_text>Class Examples<jupyter_code>class Person(var firstName: String, var lastName: String) {
    def printFullName() = println(s"$firstName $lastName")
}
val p = new Person("Julia", "Kern")
println(p.firstName)
p.lastName = "Manes"
p.printFullName()<jupyter_output>Julia
<NAME><jupyter_text>Scala MethodsAn example of how to create and method and how the method is being called.<jupyter_code>def add(a: Int, b: Int): Int = a + b
val add_x = add(1, 2)<jupyter_output><empty_output><jupyter_text>Examples of Scala in Spark<jupyter_code>val data = Seq((1,2,3), (4,5,6), (6,7,8), (9,19,10))
val ds = spark.createDataset(data)
ds.show()
val cubed= (s:Long) => {  s * s * s}
// Register UDF
spark.udf.register("cubed", cubed)
// Create temporary view
spark.range(1, 9).createOrReplaceTempView("udf_test")
spark.sql("SELECT id, cubed(id) AS id_cubed FROM udf_test").show()
import org.apache.spark.sql.functions._
val dataFile = "/Users/sheng/Downloads/archive/aug_test.csv"
val dataSet = spark.read.options(Map("inferSchema"->"true","delimiter"->",","header"->"true"))
  .csv(dataFile)
dataSet.printSchema()
dataSet.show()
dataSet.collect.foreach(println)
dataSet.sort(col("training_hours").desc).show(5)<jupyter_output>+-----------+--------+----------------------+------+--------------------+-------------------+---------------+----------------+----------+------------+------------+------------+--------------+
|enrollee_id|    city|city_development_index|gender| relevent_experience|enrolled_university|education_level|major_discipline|experience|company_size|company_type|last_new_job|training_hours|
+-----------+--------+----------------------+------+--------------------+-------------------+---------------+----------------+----------+------------+------------+------------+--------------+
|      17854|city_114|    0.9259999999999999|  null|Has relevent expe...|      no_enrollment|        Masters|            STEM|        15|      10000+|     Pvt Ltd|           2|           334|
|       7443|city_173|                 0.878|  Male|Has relevent expe...|      no_enrollment|        Masters|            STEM|         8|      10000+|     Pvt Ltd|           1|           334|
|      24845|city_103|                  [...]
{'fraction_non_alphanumeric': 0.18722139673105498, 'fraction_numerical': 0.024021792966815256, 'mean_word_length': 2.7853795688847236, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '468399', 'license': 'MIT', 'n_tokens_mistral': 1274, 'n_tokens_neox': 1177, 'n_words': 397, 'path': 'Project/Scala_Examples.ipynb', 'repo_name': 'ShenghaoHuang/Week9-ResearchProjects', 'size': '4043'}