Stately progress: the gardens of Holker Hall And pretty much killed himself in the process, Hugh adds. 'The reason my father died of a heart attack at 54 was because his father had been a sort of Edwardian gentleman who didn't understand about money. He had 35 indoor staff. When my father was a child here, there was a footman outside the nursery. The train took him and his sisters and their ponies from Cark to London for the Season. They also built the railway, which notched up a few more debts as its construction required the diversion of the River Leven. 'By the time my father took over,' Hugh continues, 'all that vanished into, "How am I going to pay the wages on Friday?" ' Hugh's book, A Time to Plant: Life and Gardening at Holker, was intended as a history of the house and gardens, but became a hybrid: part impassioned horticultural guide and part memoir about his childhood, banishment to boarding school and the icy winds of his mother's coldness. The story of Holker, Hugh says, is about the people involved with it. Many, such as his father's sisters, known collectively as the Aunt Heap, and the horticultural legacy of his paternal grandmother, Lady Moyra Cavendish, helped cultivate Hugh's passion for gardening. At Cothill, his Oxfordshire prep school, and then at Eton, Hugh was 'dreadfully homesick' for the gardens. What to his mother, Lady Pamela, were 20 acres of 'rank desolation' and to his father another money- and life-sapping concern, were to the young boy 'my wilderness domain', 'my kingdom'. And then came banishment. 'After the age of seven,' he writes, 'I had never seen the magnolias, the azaleas, the rhododendrons or the majority of the flowering shrubs.' He hadn't seen much love from his mother either. 'She took it in turns to be quite nasty to all us siblings [three girls and two boys] except for Ed [Hugh's younger brother]. She wrecked each of us in different ways. All of us went down with drinks or drugs or something. It is fitting that as part of their revamping of the formal gardens, first designed in the 1720s, Hugh and Grania chose to build a slate labyrinth in 2002 after successfully creating a wildflower meadow, itself inspired by his Aunt Sybil, who once mentioned that his grandmother grew martagon lilies here 40 years earlier. They grow on this spot today, bringing the past and the present full circle. An ancient symbol of pilgrimage and journeying, the labyrinth is a modern gesture by a couple whose own journey has been somewhat rocky. They have had to work hard to find their place at Holker and to ease the estate into the 21st century while preserving its heritage. Halfway through our stroll around the formal gardens we enter the Elliptical Garden, designed to be as welcoming as a front hall, which leads on to the summer garden and the wildflower meadow beyond. 'Look, that's our handkerchief tree we put in. It takes 14 years to flower. It's all about the long term,' he continues. My grandmother also planted a handkerchief tree. That's what the modern financier doesn't understand. Tell them something will come into its own in 100 years' time and they will say, "You're a dinosaur." Over there,' he says, 'there is a statuesque chestnut tree that was part of the 1750 planting. 'When I was working in the nuclear industry I had an Irish PR,' he continues, referring to his two-year stint as a government whip in the early 1990s. I talked about leaving a legacy for posterity and the PR man said, "What did posterity ever do for us?" He roars with laughter. Hugh marvels that Baroness Thatcher 'rather set her mind' on him, despite his refusal to offer his lifelong allegiance to the Conservative Party when grilled by her Chief Whip, Tim Renton, 'a man I famously didn't get on with when we had known each other in the City'. Thatcher made Hugh a life peer shortly before her resignation in 1990. As we approach the Neptune Cascade, a limestone and slate water feature inspired by one of Hugh and Grania's visits to Rajasthan, Hugh tells me that it is unofficially dedicated to the woman who 'released us'. We would not be the prosperous, expanding, fairly modern estate employing 250 people without her era bringing things slightly under control. She made hideous mistakes - everyone knows that - but she released the entrepreneurial world. She was also phenomenally kind,' he adds. 'Warts and all, I think she's a phenomenal lady.' The revival of the estate is in itself fairly phenomenal given how 'terribly lazy' Hugh insists he is and what an 'undisciplined mind' he is saddled with. The slate quarries have been a particular challenge. 'There have certainly been peaks and troughs economically,' Hugh says as he leads me up to the sundial on the wildflower meadow, made from blue-grey Kirkby slate in 1992 to a design by Sir Mark Lennox-Boyd. The slate market, always capricious, had collapsed. We had 150 highly skilled craftsmen on our payroll. It was a very bad moment. We built the sundial to keep them busy. It cost £100,000 in today's terms, but it did a lot for morale. The men are very proud of it. Along with economic vagaries, 'the health and safety police' are Hugh's other preoccupation. The charm of the garden is lost because they make you check every tree for possible falling branches. We have to have tree surgeons going around regularly in case something falls on someone's head. But if you want romance in the garden there are going to be hazards. Hugh knows that from his own childhood, when he spent hours climbing trees or swimming in the old reservoir. It was here, in the natural landscape, that he felt at ease. The gardens he first loved were those of his maternal grandmother, Gwendoline Lloyd Thomas, who lived at Compton Beauchamp (then in Berkshire, now Oxfordshire). Then, after leaving Eton at the age of 18, he did a year's work experience in the forestry department of Lord Lonsdale's Lowther Castle nearby in Penrith. His love of nature, 'a dormant seed' for so long, suddenly blossomed, coinciding with his first weekends at home. 'It was the only period of my life when I had a good relationship with my mother,' he says. Meanwhile, a darker seed was germinating: alcoholism, a disease also suffered by his father. 'When people ask me when [my addiction] started,' Hugh says, 'I'm rather fond of saying, "From birth, actually." ' He grew up, he says, 'with all those negative characteristics that go with addiction: ego, anger, believing everybody else is having a better time. Until that was resolved I don't think I was very happy with myself. I went on functioning for years. A lot of people didn't realise I was a drunk.' He was drinking full throttle when, aged 31, he inherited Holker 'overnight'. Lady Pamela immediately abdicated responsibility for the entire estate, though her presence would threaten to strangle it. 'Of the remaining 35 years of her life I can remember no single occasion when I enjoyed her approval or benefited from her encouragement,' Hugh writes. Hugh and Grania had married only two years earlier, and their son, Freddy, was a babe in arms. Their daughter Lucy was born a year later, followed by Emily in 1980. My mother-in law moved to the back of the house, then to one on the estate. We made our own quarters to begin with,' Grania recalls. We went up to the top floor because I wanted to be near the children. The set-up was quite old-fashioned. In those days nurseries were as far away as they could be. One always had a nanny. I had an au pair if I was lucky. We did start modernising pretty quickly. But when they first took over the reins, they were too nervous to ask for guidance, and there was none forthcoming from Lady Pamela. 'I was terrified of the gardens,' Hugh writes. Their first five years of gardening were a 'disaster'. One visitor even asked to be directed to the gardens when she was standing right in them. Thank goodness for the Aunt Heap, three of Lord Cavendish's five sisters, Diana, Mary and Sybil, who in their later lives returned to live on the estate. 'Diana was the cleverest and most permanent,' says Hugh, who saw a lot of her in London when she was, for a time, companion to Lady Astor at 100 Eaton Square. Following the death of her third husband, Lord Gage, she returned to live in a house on the estate (now occupied by Emily, an addiction counsellor, and her son, Hector). It was heaven having them back. They showed us that you could absorb change without being frightened of it. They gave us great confidence - and they were such fun. Holker first opened to the public in 1950 - only the second private house to do so after Longleat - with nothing but a converted loo and a basic guidebook. The key to its transformation has been twofold. In the garden, Hugh and Grania have followed their instincts. They talk about how they listen to the plants, trees and shrubs, and talk to them, taking care not to offend. 'If something's not doing well,' Grania says, 'you've got to be careful not to say too much in front of it.' 'Otherwise they do just die,' Hugh says. They need the right neighbours too. And space. Ours are given all the love and space they need. Not something the National Trust excel at, he remarks later when he shows me how a path has been redirected to avoid moving a beloved shrub. 'They wouldn't dream of doing this at the National Trust. They're all about preserving the wrong things.' He and Grania also listen to what the public wants. Vistas, benches for the elderly, paths for the less ambulant, a Pagan Grove, the Neptune Cascade, more planting; there are always new ways to give pleasure. We probably have higher standards in the house and garden than at any time in our history, largely because the public is so critical and demanding and has a great appetite for change. What I know for certain is that the public appreciates being invested in. Hence the attention to detail, whether it's the car-park, designed by the architect Kim Wilkie, or the lavatories in the car-park of Cartmel Racecourse, part of the estate. 'Everyone said, don't bother,' Grania says. And what did it do? Won Top Loo of the Year! With Holker in robust health, they are looking to its future, and their own. Four years ago Lucy agreed to step into her father's shoes. Freddy, who manages a sheep farm nearby, 'preferred, I think, to have an easier lifestyle elsewhere,' Grania says. 'He's quite alternative. He doesn't believe in conventional farming. He lives down a two-mile drive with potholes the size of this room and a river and no electricity. It is lovely. There's no light pollution, and no noise. But it's pretty odd.' Lucy is a painter. She divides her time between London, where she teaches at the Prince's Trust, and the Garden House, once lived in by Aunt Sybil. She started taking on some responsibilities for the estate two years ago. 'I won't look back,' Grania says. 'Holker's not mine. I don't think it is my home necessarily, even now.' 'I quite look forward to living somewhere without inherited disadvantages,' Hugh adds. We never really had time to look at this house and say, how would it be right for us? That's what I hope Lucy will have a chance to do. It's been great fun and we've had a very good time, but I'm ready to do a bit less. It is a fact that one is never off-duty. If one was minded to complain, there might be things to complain about. But that was a lesson from the Aunt Heap. Did they complain about having plentiful servants before the war, of having independence, quality of life and everything so plentiful in a gentle political age before the brutality and change that followed? They did not. It is fatal to look at the downside of this sort of job. There is so much plus. 'A Time to Plant: Life and Gardening at Holker' (Frances Lincoln, £25) is available from Telegraph Books for £23 plus £1.25 p&p (0844-871 1515; books.telegraph.co.uk). Lord Cavendish will be signing copies of the book at the Holker Garden Festival (June 8-10, 015395-58838; holkerfestival.co.uk) This article is from the Telegraph Magazine, free every Saturday with the Daily Telegraph