Q:

Out of memory exception during TFIDF generation for use in Spark's MLlib

I have been running into memory overflow issues while creating TFIDF vectors to be used in document classification using MLlib's Naive Baye's classification implementation. 
http://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
Memory overflow and GC issues occur while collecting idfs for all the terms. To give an idea of scale, I am reading around 615,000(around 4GB of text data) small sized documents from HBase  and running the spark program with 8 cores and 6GB of executor memory. I have tried increasing the parallelism level and shuffle memory fraction but to no avail.
How can I solve this OOM issue?
Thanks

A:

How are you connecting your Spark RDD to HBase? How big are your HBase regions?  IF you are loading multiple HBase regions simultaneously and the sum of their size is greater than available RAM, you could blow out memory during the loading phase.  
One way to avoid this is to write your own HBase loader that specifically only scans the columns you are interested in loading rather than the whole region file, which might reduce memory consumption considerably.  You could then also tune the number of partitions you are loading based on a semantic understanding of your HBase keyspace, further tuning your memory consumption during loading.

