Q:

Spark - How to deal with columns that have blank space in the name

I would like to know how to access an attribute from a Row that has a blank space in the name.
For example, I have this Row object
Row(ONE CATEGORY=u'category') 

How can I access the ONE CATEGORY value. Normally I would use row.oneCategory to access it, but in this case that's not possible because of the blank space. If possible, I prefer the suggestions in Python.
Thanks

A:

In Python can use getattr function:
row = Row("ONE CATEGORY")("category")
row
## Row(ONE CATEGORY='category')
getattr(row, u"ONE CATEGORY")
## 'category'

or Row.asDict method:
row.asDict()["ONE CATEGORY"]
## 'category'

Since you cannot use dot syntax in Scala it is not really an issue, but if you want to access fields by name you can use Row.getAs
val row = sc.parallelize(Tuple1("category") :: Nil).toDF("ONE CATEGORY").first
row.getAs[String]("ONE CATEGORY")

or Row.getValuesMap:
row.getValuesMap[String](Seq("ONE CATEGORY"))("ONE CATEGORY")

In both Python and Scala, you can access value by index:
## row[0]
'category'

row(0)
// Any = category
row.getString(0)
// String = category

Finally you can use alias method during select to avoid the issue completely:
df.select(col("ONE CATEGORY").alias("ONE_CATEGORY"))

