For any choice of activation function (fix the choice for all the hidden nodes for both the following DNNs) do we know of functions which some $k$ (hidden layer) DNN can compute but a $(k-1)-$DNN can't? (for any $k$ and with no restriction on width on any of the two DNNs)  I am equally happy to know of results which pertain to width restrictions on either side or vary the activation function between the two sides. I am not asking about approximation ability gaps like what have recently been proven by Elden and Shamir between $2-$DNN and $1-$DNN. 
What precisely is the extra power afforded by using deeper nets?
cc.complexity theory;circuit complexity;machine learning;lg.learning;interaction nets