<jupyter_start><jupyter_text>Install Java, Spark, and Findspark<jupyter_code>!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q http://www-us.apache.org/dist/spark/spark-2.3.2/spark-2.3.2-bin-hadoop2.7.tgz
!tar xf spark-2.3.2-bin-hadoop2.7.tgz
!pip install -q findspark<jupyter_output><empty_output><jupyter_text>Set Environmental Variables<jupyter_code>import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-2.3.2-bin-hadoop2.7"<jupyter_output><empty_output><jupyter_text>Start a Spark Session<jupyter_code>import findspark
findspark.init()<jupyter_output><empty_output><jupyter_text>Dataframe Basics<jupyter_code># Import our SparkSession so we can use it
from pyspark.sql import SparkSession
# Create our SparkSession, this can take a couple minutes locally
spark = SparkSession.builder.appName("basics").getOrCreate()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13513513513513514, 'fraction_numerical': 0.023783783783783784, 'mean_word_length': 6.8474576271186445, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '484985', 'license': 'ADSL', 'n_tokens_mistral': 375, 'n_tokens_neox': 355, 'n_words': 56, 'path': '07-Extra-Content/Big-Data-Google-Colab/day-1/05-Ins_Pyspark_Dataframes_Basics/Unsolved/spark_dataframe_basics.ipynb', 'repo_name': 'anirudhmungre/sneaky-lessons', 'size': '925'}