Our physical actions are what helps us build up our spirit, and we can use our physical actions for this purpose because Christ made it possible by becoming physical Himself.
The 'visible' or 'physical' Church as you put it also serves the function to help a person develop their spirit.