return sum(col(c).isNotNull().cast('integer')).alias(c)
exprs = [count_not_null(c) for c in df.columns]
df.agg(*exprs).show()
Count number of non-NaN entries in each column of Spark dataframe with Pyspark