Q:

Hadoop Spark docker swarm where pyspark gives BlockMissingException but file is fine

Based on https://github.com/gotthardsen/docker-hadoop-spark-workbench/tree/master/swarm I have a docker swarm setup with hadoop, spark, hue and a jupyter notebook setup.
Using Hue I uploaded a file to hdfs, and I have not problem downloading or viewing the file from hue or in hdfs on the namenode. There is no missing blocks and file check says everything is fine.
But when I try to access it using pyspark in jupyter I get a:

org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block:
  BP-296583215-10.0.0.6-1542473394293:blk_1073741833_1009
  file=/20170930.csv

I know this is not about missing block but more likely something else. But I can not figure out why. The code python code from the workbook using python2 kernel is:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setAppName('hello').setMaster('spark://spark-master:7077')
sc = SparkContext(conf=conf)

from pyspark.sql import SparkSession
# Read from HDFS
sparkSession = SparkSession(sc)
df_load = sparkSession.read.csv('hdfs://namenode:9000/20170930.csv')
df_load.show()

The problem comes at:
df_load = sparkSession.read.csv('hdfs://namenode:9000/20170930.csv')

I can't see anything in the different logs that gives me a clue. There are only INFO lines
Anybody who can give be a clue what to look for?

A:

The point is that we cannot publish the ports if we use endpoint-mode=dnsrr (and we have to use dnsrr or there will be some problems for Hadoop).
So to expose the ports, we can start another container to publishes ports and routes the traffic to HDFS nodes. One available tool is socat.
Of course, you still have to set the hosts on that node.
see https://blog.newnius.com/setup-distributed-hadoop-cluster-with-docker-step-by-step.html

