Q:

Spark: How to prevent DataFrameWriter from dropping the partitioning columns on call to write.partitionBy?

we are using below method to save a spark sql dataframe 
public DataFrameWriter<T> partitionBy(scala.collection.Seq<String> colNames)

from the docs:

Partitions the output by the given columns on the file system. If
  specified, the output is laid out on the file system similar to Hive's
  partitioning scheme. As an example, when we partition a dataset by
  year and then month, the directory layout would look like:
  - year=2016/month=01/
  - year=2016/month=02/

our code looks like this:
df.repartition(new Column("day")).write.partitionBy("day").mode(SaveMode.Overwrite).parquet(outputPath)

we get the directory layout we expect, but we find the parquet files don't have the day column in them anymore. we would like the parquet files to have the day column. how can we do this? 

A:

the best solution is to add a duplicate day column to the dataframe which would get dropped in the parquet file but the original day column will still be there. 

