Q:

Not all Spark Workers are starting: SPARK_WORKER_INSTANCES

I have my spark-defaults.conf configuration like this.
my node has 32Gb RAM. 8 cores.
I am planning to use 16gb and 4 workers with each using 1 core.
SPARK_WORKER_MEMORY=16g
SPARK_PUBLIC_DNS=vodip-dt-a4d.ula.comcast.net
SPARK_WORKER_CORES=4
SPARK_WORKER_INSTANCES=4
SPARK_DAEMON_MEMORY=1g

When i try to start the master and workes like this, only 1 worker is being started where an i am expecting 4 workers.
start-master.sh --properties-file /app/spark/spark-1.6.1-bin-hadoop2.6/conf/ha.conf
start-slaves.sh

these commands started the master and showed that they were starting 4 workes where as only 1 worker was started.
The one worker that started using all 4 cores. 
Please let me know why my other 3 worked are not starting.

A:

Memory and core properties are for every executor. So when you say SPARK_WORKER_CORES=4 this is every executor with 4 cores.
Also you cant use all memory in your server for executors. If you want 4 executors with total 16gb memory, your properties should be like this
SPARK_WORKER_MEMORY=4g
SPARK_PUBLIC_DNS=vodip-dt-a4d.ula.comcast.net
SPARK_WORKER_CORES=1
SPARK_WORKER_INSTANCES=4

