<jupyter_start><jupyter_text>6교시 Spark JDBC to MongoDB 목차* [1. MongoDB 접속 예제](1.-MongoDB-접속-예제)* [10. 참고자료](10.-참고자료)  1. Spark JDBC 환경구성 1.1. MongoDB 기동> 터미널에서 몽고디비를 기동되어 있다고 가정합니다<jupyter_code>from pyspark.sql import *
from pyspark.sql.functions import *
from pyspark.sql.types import *
from IPython.display import display, display_pretty, clear_output, JSON

spark = (
    SparkSession
    .builder
    .config("spark.sql.session.timeZone", "Asia/Seoul")
    .config('spark.mongodb.input.uri', 'mongodb://mongo/testdb.people')
    .config('spark.mongodb.output.uri', 'mongodb://mongo/testdb.people')
    .getOrCreate()
)

# 노트북에서 테이블 형태로 데이터 프레임 출력을 위한 설정을 합니다
spark.conf.set("spark.sql.repl.eagerEval.enabled", True) # display enabled
spark.conf.set("spark.sql.repl.eagerEval.truncate", 100) # display output columns size

# 공통 데이터 위치
home_jovyan = "/home/jovyan"
work_data = f"{home_jovyan}/work/data"
work_dir=!pwd
work_dir = work_dir[0]

# 로컬 환경 최적화
spark.conf.set("spark.sql.shuffle.partitions", 5) # the number of partitions to use when shuffling data for joins or aggregations.
spark.conf.set("spark.sql.streaming.forceDeleteTempCheckpointLocation", "true")
spark
people = spark.createDataFrame(
    [
        ("<NAME>",  50)
        , ("Gandalf", 1000)
        , ("Thorin", 195)
        , ("Balin", 178)
        , ("Kili", 77)
        , ("Dwalin", 169)
        , ("Oin", 167)
        , ("Gloin", 158)
        , ("Fili", 82)
        , ("Bombur", None)
    ], ["name", "age"]
)

people.write.format("mongo").mode("append").save()
people.show()<jupyter_output>
{'fraction_non_alphanumeric': 0.16135204081632654, 'fraction_numerical': 0.025510204081632654, 'mean_word_length': 3.5478260869565217, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '522666', 'license': 'Apache-2.0', 'n_tokens_mistral': 718, 'n_tokens_neox': 732, 'n_words': 125, 'path': 'day3/notebooks/lgde-spark-core/lgde-spark-core-7-jdbc-mongodb.ipynb', 'repo_name': 'psyoblade/data-engineering-advanced-training', 'size': '1575'}