Q:

Is there a way to write variables created during a for loop to a dataframe in pyspark?

I am creating four variables in for loop, I am trying to write these variables into a dataframe in pyspark. Then I want to write this dataframe into a parquet and csv file. Is there an efficient way to do this. If so can you please explain me with pyspark code.

A:

An elegant and efficient way to achieve this is to translate your for loop logic into pure Spark code. 
For example:
(given four functions f1, f2, f3, f4, taking one int argument and given spark a SparkSession instance)
The following for loop:
for i in range(1000000):
    v1, v2, v3, v4 = f1(i), f2(i), f3(i), f4(i)
    [...]

can be translated into:
from pyspark.sql.functions import udf
df = spark.range(1000000)\
  .withColumn("v1", udf(f1).apply("id"))\
  .withColumn("v2", udf(f2).apply("id"))\
  .withColumn("v3", udf(f3).apply("id"))\
  .withColumn("v4", udf(f4).apply("id"))
[...]

