<jupyter_start><jupyter_text>下载（download）* 最新版本（last version)Sparkhttps://spark.apache.org/downloads.html* JDK: Amazon Corretto是Amazon提供的一个免费的、基于OpenJDK的、多平台的、生产就绪的JDK发行版https://docs.aws.amazon.com/zh_cn/corretto/latest/corretto-8-ug/downloads-list.html*  winutilshttps://github.com/steveloughran/winutils  解压缩<jupyter_code>!python -m tarfile -e c:/spark/spark-2.4.2-bin-hadoop2.7.tgz  c:/spark/
!python -m zipfile -e c:/spark/amazon-corretto-8.202.08.2-windows-x64-jdk.zip c:/spark/
!python -m zipfile -e c:/spark/winutils-master.zip c:/spark/<jupyter_output><empty_output><jupyter_text>安装pyspark和findspark<jupyter_code>!pip install findspark
#可以直接安装
#!pip install  pyspark
#网络问题，也可以下载后安装
!pip install c://spark/pyspark-2.4.2.tar.gz
import os
os.environ["HADOOP_HOME"]="c://spark/winutils-master/hadoop-2.7.1/"
os.environ["JAVA_HOME"]="c:\\spark\\jdk1.8.0_202"
import findspark
findspark.init(spark_home="c:\\spark\\spark-2.4.2-bin-hadoop2.7")
from pyspark.sql import SparkSession
spark=SparkSession.Builder().master("local[*]").getOrCreate()
spark
df=spark.sql("select 1 as a ")
df.show()
df.write.parquet("hello",mode="overwrite")
df=spark.read.parquet("hello")
print(df.toPandas())
spark.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1775244299674267, 'fraction_numerical': 0.02687296416938111, 'mean_word_length': 6.187134502923977, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '39341', 'license': 'Apache-2.0', 'n_tokens_mistral': 581, 'n_tokens_neox': 548, 'n_words': 57, 'path': 'pyspark_install_win.ipynb', 'repo_name': 'alitrack/pyspark', 'size': '1228'}