Also, when you are on the ground, YOU are more confident so he gains confidence from you (mirrors your emotion).
When you are on his back, you feel more nervous (because he is nervous) and that compounds his nervousness.