I'm wondering how other developers are setting up their local environments for working on Spark projects. Do you configure a 'local' cluster using a tool like Vagrant? Or, is it most common to SSH into a cloud environment, such as a cluster on AWS? Perhaps there are many tasks where a single-node cluster is adequate, and can be run locally more easily.
Local Development for Apache Spark
apache spark
Spark is intended to be pointed at large distributed data sets, so as you suggest, the most typical use cases will involve connecting to some sort of Cloud system like AWS. In fact, if the data set you aim to analyze can fit on your local system, you'll usually find that you can analyze it just as simply using pure python.  If you're trying to leverage a series of local VMs, you're going to run out of memory pretty quickly and jobs will either fail or grind to a halt.With that said, a local instance of spark is very useful for the purpose of development. One way that I've found that works is if I have a directory in HDFS with many files, I'll pull over a single file, develop locally, then port my spark script to my cloud system for execution.  If you're using AWS, this is really helpful for avoiding big fees while you're developing.