he had served so much and so well that he was being allowed an early retirement in the years during which i grew up , an early withdrawal from the social life that whirled around the imperial palace , though i did not realize this at the time .