My ultimate goal is to use Jupyter together with Python for data analysis using Spark. The current hurdle I face is loading the external spark_csv library. I am using Mac OS and Anaconda as the Python distribution.In particular, the following:from pyspark import SparkContextsc = SparkContext('local', 'pyspark')sqlContext = SQLContext(sc)df = sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('file.csv')df.show()when invoked from Jupyter yields: Py4JJavaError: An error occurred while calling o22.load.: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.orgHere are more details:Setting Spark together with JupyterI managed to set up Spark/PySpark in Jupyter/IPython (using Python 3.x). System initial settingOn my OS X I installed Python using Anaconda. The default version of Python I have currently installed is 3.4.4 (Anaconda 2.4.0). Note, that I also have installed also 2.x version of Python using conda create -n python2 python=2.7.Installing SparkThis is actually the simplest step; download the latest binaries into ~/Applications or some other directory of your choice. Next, untar the archive tar -xzf spark-X.Y.Z-bin-hadoopX.Y.tgz.For easy access to Spark create a symbolic link to the Spark:ln -s ~/Applications/spark-X.Y.Z-bin-hadoopX.Y ~/Applications/sparkLastly, add the Spark symbolic link to the PATH:export SPARK_HOME=~/Applications/sparkexport PATH=$SPARK_HOME/bin:$PATHYou can now run Spark/PySpark locally: simply invoke spark-shell or pyspark.Setting JupyterIn order to use Spark from within a Jupyter notebook, prepand the following to PYTHONPATH:export PYTHONPATH=$SPARKHOME/python/lib/py4j-0.8.2.1-src.zip:$SPARKHOME/python/:$PYTHONPATHFurther details can be found here.
Use spark_csv inside Jupyter and using Python
python;apache spark;pyspark;jupyter
Assuming the rest of your configuration is correct all you have to do is to make spark-csv jar available to your program. There are a few ways you can achieve this:manually download required jars including  spark-csv and csv parser (for example org.apache.commons.commons-csv) and put them somewhere on the CLASSPATH.using --packages option (use Scala version which has been used to build Spark. Pre-built versions use 2.10):using PYSPARK_SUBMIT_ARGS environmental variable:export PACKAGES=com.databricks:spark-csv_2.11:1.3.0export PYSPARK_SUBMIT_ARGS=--packages ${PACKAGES} pyspark-shelladding Gradle string to spark.jars.packages in conf/spark-defaults.conf:spark.jars.packages    com.databricks:spark-csv_2.11:1.3.0