<jupyter_start><jupyter_text>Spark on Kubernetes Preparing the notebook https://towardsdatascience.com/make-kubeflow-into-your-own-data-science-workspace-cc8162969e29  Run Spark inside Notebook https://www.sicara.ai/blog/2017-05-02-get-started-pyspark-jupyter-notebook-3-minutes<jupyter_code>import findspark
findspark.init()

import pyspark
import random

sc = pyspark.SparkContext(appName="Pi")
sc.setLogLevel("INFO")
num_samples = 100000

def inside(p):     
  x, y = random.random(), random.random()
  return x*x + y*y < 1

count = sc.parallelize(range(0, num_samples)).filter(inside).count()

pi = 4 * count / num_samples
print(pi)

sc.stop()<jupyter_output><empty_output><jupyter_text>Setup service account permissions https://github.com/kubeflow/kubeflow/issues/4306 issue with launching spark-operator from jupyter notebook Run command in your shell (not in notebook)```shellexport NAMESPACE=kubectl create serviceaccount spark -n ${NAMESPACE}kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=${NAMESPACE}:spark --namespace=${NAMESPACE}```  Python version> Note: Make sure your driver python and executor python version matches.> Otherwise, you will see error msg like belowException: Python in worker has different version 3.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` are correctly set.<jupyter_code>import sys
print(sys.version)<jupyter_output><empty_output><jupyter_text>Client Mode<jupyter_code>import findspark, pyspark,socket
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession

findspark.init()

localIpAddress = socket.gethostbyname(socket.gethostname())

conf = SparkConf().setAppName('sparktest1')
conf.setMaster('k8s://https://kubernetes.default.svc:443')
conf.set("spark.submit.deployMode", "client")
conf.set("spark.executor.instances", "2")
conf.set("spark.driver.host", localIpAddress)
conf.set("spark.driver.port", "7778")
conf.set("spark.kubernetes.namespace", "kf10")
conf.set("spark.kubernetes.container.image", "seedjeffwan/spark-py:v3.0.0")
conf.set("spark.kubernetes.pyspark.pythonVersion", "3")
conf.set("spark.kubernetes.namespace", "spark")
conf.set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")
conf.set("spark.kubernetes.executor.annotation.sidecar.istio.io/inject", "false")
sc = pyspark.context.SparkContext.getOrCreate(conf=conf)

# following works as well
# spark = SparkSession.builder.config(conf=conf).getOrCreate()
num_samples = 100000

def inside(p):     
  x, y = random.random(), random.random()
  return x*x + y*y < 1

count = sc.parallelize(range(0, num_samples)).filter(inside).count()
sc.stop()<jupyter_output><empty_output><jupyter_text>Cluster Mode  Java<jupyter_code>%%bash

/opt/spark/bin/spark-submit --master "k8s://https://kubernetes.default.svc:443" \
--deploy-mode cluster \
--name spark-pi-java \
--class org.apache.spark.examples.SparkPi  \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.namespace=spark \
--conf spark.kubernetes.driver.annotation.sidecar.istio.io/inject=false \
--conf spark.kubernetes.executor.annotation.sidecar.istio.io/inject=false \
--conf spark.kubernetes.container.image=seedjeffwan/spark:v3.0.0 \
--conf spark.kubernetes.driver.pod.name=spark-pi-java \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.6.jar 1000<jupyter_output><empty_output><jupyter_text>Python<jupyter_code>%%bash

/opt/spark/bin/spark-submit --master "k8s://https://kubernetes.default.svc:443" \
--deploy-mode cluster \
--name spark-pi \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.container.image=seedjeffwan/spark-py:v3.0.0 \
--conf spark.kubernetes.driver.pod.name=spark-pi \
--conf spark.kubernetes.namespace=spark \
--conf spark.kubernetes.driver.annotation.sidecar.istio.io/inject=false \
--conf spark.kubernetes.executor.annotation.sidecar.istio.io/inject=false \
--conf spark.kubernetes.pyspark.pythonVersion=3 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark /opt/spark/examples/src/main/python/pi.py 1000<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14197823000473261, 'fraction_numerical': 0.02129673450070989, 'mean_word_length': 5.720190779014309, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '215112', 'license': 'Apache-2.0', 'n_tokens_mistral': 1527, 'n_tokens_neox': 1400, 'n_words': 239, 'path': 'examples/notebook/spark-on-kubernetes.ipynb', 'repo_name': 'abin-tiger/eks-spark-benchmark', 'size': '4226'}