After Charles's death, his widow Lady Stuart de Rothesay inherited the castle.
She died in 1867 and left the property to Louisa her younger daughter as Charlotte, the elder daughter, had predeceased her.
Louisa had been born in 1818 in Paris.
The family moved into Highcliffe Castle in 1834  and lived there and also in their  London residence.
Louisa wrote some of her memories of her life at Highcliffe Castle which were recorded in a book by Hare.

In 1842 Louisa married Henry, Marquis of Waterford and became the Marchioness of Waterford.
The couple lived at Curraghmore, County Waterford for many years, and during this time Louisa became an accomplished painter whose work is still valued today.
Her husband died in 1859 and she moved to the estate left to her in Ford in Northumberland.
It was here that she built a school hall and on the walls painted Biblical scenes.
These paintings survive today in what is now called Lady Waterford Hall in Ford.

In 1867 her mother died and she inherited Highcliffe Castle.
She spent the summers at Highcliffe and returned to Ford Castle in the winter.
 She had many visitors at the castle one of whom was Augustus Hare, a fellow painter.
He recounts a typical day at Highcliffe Castle.

“The late breakfast in the pleasant room which looks across the sunlit sea to the Isle of Wight through the leaves of large magnolia trees was always a happy meeting, at which Lady Waterford brimmed over with the anecdotes and reminiscences evoked by her morning letters.
Immediately afterwards she would read aloud from the newspapers in the porch-room, then she went to write in her own room, and her guests dispersed till she summoned them back by playing and singing loudly with wide-open windows.
Then all adjourned to the library, where one of the party read aloud, whilst Lady Waterford painted.
After luncheon there was a walk, generally on the sands, often to the Haven House and its wild fir wood.".

She entertained the Prince and Princess of Wales (later King Edward and Queen Alexandra) at Highcliffe Castle.
In 1880 she recounts one of their visits.

"I have had a great surprise in the unexpected visit of the Prince and Princess of Wales, with their two sons and the three little Princesses, landing on the beach, coming up to tea, and drawing nets on the shore till evening — the young Princes swimming about in flannels, and finally the Prince and two sons swimming away to join the steam-tug."
A picture of the royal family at around the same time as this visit is shown on the left.

In 1891 Lady Waterford died and as she had no children she left Highcliffe Castle to a very distant cousin Edward Stuart Wortley.
After Charles's death, his widow Lady Stuart de Rothesay inherited the castle. She died in 1867 and left the property to Louisa her younger daughter as Charlotte, the elder daughter, had predeceased her. Louisa had been born in 1818 in Paris. The family moved into Highcliffe Castle in 1834  and lived there and also in their  London residence. Louisa wrote some of her memories of her life at Highcliffe Castle which were recorded in a book by Hare.
In 1842 Louisa married Henry, Marquis of Waterford and became the Marchioness of Waterford. The couple lived at Curraghmore, County Waterford for many years, and during this time Louisa became an accomplished painter whose work is still valued today. Her husband died in 1859 and she moved to the estate left to her in Ford in Northumberland. It was here that she built a school hall and on the walls painted Biblical scenes. These paintings survive today in what is now called Lady Waterford Hall in Ford.
In 1867 her mother died and she inherited Highcliffe Castle. She spent the summers at Highcliffe and returned to Ford Castle in the winter.  She had many visitors at the castle one of whom was Augustus Hare, a fellow painter. He recounts a typical day at Highcliffe Castle.
“The late breakfast in the pleasant room which looks across the sunlit sea to the Isle of Wight through the leaves of large magnolia trees was always a happy meeting, at which Lady Waterford brimmed over with the anecdotes and reminiscences evoked by her morning letters. Immediately afterwards she would read aloud from the newspapers in the porch-room, then she went to write in her own room, and her guests dispersed till she summoned them back by playing and singing loudly with wide-open windows. Then all adjourned to the library, where one of the party read aloud, whilst Lady Waterford painted. After luncheon there was a walk, generally on the sands, often to the Haven House and its wild fir wood.".
She entertained the Prince and Princess of Wales (later King Edward and Queen Alexandra) at Highcliffe Castle. In 1880 she recounts one of their visits.
"I have had a great surprise in the unexpected visit of the Prince and Princess of Wales, with their two sons and the three little Princesses, landing on the beach, coming up to tea, and drawing nets on the shore till evening — the young Princes swimming about in flannels, and finally the Prince and two sons swimming away to join the steam-tug." A picture of the royal family at around the same time as this visit is shown on the left.
In 1891 Lady Waterford died and as she had no children she left Highcliffe Castle to a very distant cousin Edward Stuart Wortley.
Highcliffe Castle