However, from the reign of James I the castle ceased to be a royal residence and it fell into decay.
After Charles I came to the throne, the castle was granted to William Cecil, 2nd Earl of Salisbury.
Much of the castle was now in ruins, having not been used since Tudor times.
The castle was leased to Sir William Harrington of Hertingfordbury Park, who carried out work to restore the buildings.
He then assigned his lease to Sir William Cowper, Collector of Customs at the Port of London.
 From this time the castle remained in the ownership of the Cecil family, who leased it out to a succession of occupiers, many of them successors to William Cowper.
Around 1790 the south wing was added to the gatehouse.
The windows of the existing gatehouse were all remodelled, and the parapet added with its brick battlements.
Around 1800 a new gateway and lodge were built by the Marquess of Downshire.

Between 1805 and 1809, the castle became the home of the East India Company College.
In 1822, a general dispensary was established at the castle by the Rev. Thomas Lloyd, who was a prime mover in charitable work in the town.

In 1911, the Hertford Corporation leased the gatehouse of the castle (which was all that then remained) from the Marquess of Salisbury to house its administration.
The grounds became a public garden.
 In the 1930s, the north wing was added to the gatehouse and, in the late 20th century, Lord Salisbury gave what was left of the castle to the town.
However, from the reign of James I the castle ceased to be a royal residence and it fell into decay. After Charles I came to the throne, the castle was granted to William Cecil, 2nd Earl of Salisbury. Much of the castle was now in ruins, having not been used since Tudor times. The castle was leased to Sir William Harrington of Hertingfordbury Park, who carried out work to restore the buildings. He then assigned his lease to Sir William Cowper, Collector of Customs at the Port of London.  From this time the castle remained in the ownership of the Cecil family, who leased it out to a succession of occupiers, many of them successors to William Cowper. Around 1790 the south wing was added to the gatehouse. The windows of the existing gatehouse were all remodelled, and the parapet added with its brick battlements. Around 1800 a new gateway and lodge were built by the Marquess of Downshire.
Between 1805 and 1809, the castle became the home of the East India Company College. In 1822, a general dispensary was established at the castle by the Rev. Thomas Lloyd, who was a prime mover in charitable work in the town.
In 1911, the Hertford Corporation leased the gatehouse of the castle (which was all that then remained) from the Marquess of Salisbury to house its administration. The grounds became a public garden.  In the 1930s, the north wing was added to the gatehouse and, in the late 20th century, Lord Salisbury gave what was left of the castle to the town.
Hertford Castle