<jupyter_start><jupyter_text>创建 DataFrame```{note}可以从 RDD、从文件、从 RDBMS 创建 DataFrame 。```<jupyter_code>from pyspark.sql import SparkSession

spark = (SparkSession
         .builder
         .appName("create dataframe")
         .config('spark.executor.memory', '4g')
         .config('spark.driver.memory', '8g')
         .config("spark.executor.cores", 2)
         .config('spark.cores.max', 4)
         .getOrCreate())<jupyter_output><empty_output><jupyter_text>从 RDD 创建<jupyter_code># 创建 RDD
rdd = spark.sparkContext.parallelize([('Alice', 18), ('Bob', 20)])
# 指明列
df1 = spark.createDataFrame(rdd, ['name', 'age'])
df1.show()
# 使用 toDF 方法达到同样的效果
rdd.toDF(['name', 'age']).show()
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

# 定义 schema
schema = StructType([StructField("name", StringType()),
                     StructField("age", IntegerType())])
# 指明 schema
df2 = spark.createDataFrame(rdd, schema=schema)
df2.printSchema()<jupyter_output>root
 |-- name: string (nullable = true)
 |-- age: integer (nullable = true)<jupyter_text>读取文件创建<jupyter_code># 指明 schema，且遇到脏数据跳过加载
df = (spark.read.format("csv")
      .schema(schema)
      .option("header", True)
      .option("mode", "dropMalformed")
      .load("../data/sample.csv"))
df.show()<jupyter_output>+-----+---+
| name|age|
+-----+---+
|Alice| 18|
|  Bob| 20|
| John| 35|
| Lily|  6|
+-----+---+
{'fraction_non_alphanumeric': 0.19452449567723343, 'fraction_numerical': 0.013688760806916427, 'mean_word_length': 3.756849315068493, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '410925', 'license': 'MIT', 'n_tokens_mistral': 570, 'n_tokens_neox': 555, 'n_words': 92, 'path': 'sql/2.create df.ipynb', 'repo_name': 'newfacade/notes-on-spark', 'size': '1388'}