FitzHamon was fatally injured at the Battle of Tinchebray in 1106 and died shortly afterwards.
Henry I then gave the castle in 1122 to Robert of Gloucester, the king's illegitimate son and the husband of FitzHamon's daughter, Mabe.
After the failed attempt of Robert Curthose, duke of Normandy, William the Conqueror's eldest son, to take England from Henry I, the duke was moved here in 1126 from imprisonment at Devizes and remained imprisoned in the castle until his death in 1134.
Robert of Gloucester held the castle during the troubled years of the Anarchy in England and Wales, and passed it on to his son, William Fitz Robert.
Around the middle of the century, possibly under Robert of Gloucester, a 77-foot (23 m) wide, 30-foot (9 m) high shell keep was constructed on top of the motte, along with a stone wall around the south and west sides of the inner bailey.
The polygonal shell keep has architectural links to a similar design at Arundel Castle.
The building work was probably undertaken in response to the threat posed following the Welsh uprising of 1136.

Tensions with the Welsh continued, and in 1158 Ifor Bach raided the castle and took William hostage for a period.
A further attack followed in 1183.
By 1184 town walls had been built around Cardiff, and the West Gate to the town was constructed in the gap between the castle and the river.
William died in 1183, leaving three daughters.
One of these, Isabel, Countess of Gloucester, was declared the sole heir to the estate by Henry II.
This was contrary to legal custom in England, and was done in order that Henry could then marry her to his youngest son Prince John and thus provide him with extensive lands.
John later divorced Isabel, but he retained control of the castle until she married Geoffrey de Mandeville in 1214.

Upon Isabel's death in 1217 the castle passed through her sister to Gilbert de Clare, becoming part of the Honour of Clare, a major grouping of estates and fortifications in medieval England.
The castle formed the centre of the family's power in South Wales, although the de Clares typically preferred to reside in their castles at Clare and Tonbridge.
Gilbert's son, Richard de Clare, 6th Earl of Gloucester, carried out building work at the castle in the late 13th century, constructing the Black Tower that forms part of the southern gateway seen today.
On the ground floor the tower contained the Stavell Oged and Stavell Wenn chambers, with three rooms constructed above them.
Richard was also probably responsible for rebuilding the northern and eastern walls of the inner bailey in stone.
The inner bailey was reached through a gatehouse on the eastern side, protected by two circular towers and later called the Exchequer Gate.
The defensive work may have been prompted by the threat posed by the hostile Welsh leader Llywelyn ap Gruffudd, Prince of Wales.

Richard's grandson, Gilbert de Clare, the last male de Clare, died at the Battle of Bannockburn in 1314 and the castle was given to Hugh Despenser the Younger, the controversial favourite of Edward II.
Poor harvests and harsh governance by the Despenser family encouraged a Welsh rebellion under Llywelyn Bren in 1316; this was crushed and Llywelyn was hanged, drawn and quartered in Cardiff Castle in 1318 on Hugh's orders.
The execution attracted much criticism from across both the English and Welsh communities, and in 1321 Hugh arrested Sir William Fleminge as a scapegoat for the incident, first detaining him in the Black Tower and then executing him in the castle grounds.
Conflict between the Despensers and the other Marcher Lords broke out soon after, leading to the castle being sacked in 1321 during the Despenser War.
The Despensers recovered the castle and retained it for the rest of the century, despite the execution of Hugh Despenser for treason in 1326.
Under a 1340 charter granted by the Despensers, the castle's constable was made the de facto mayor of Cardiff, controlling the local courts.
FitzHamon was fatally injured at the Battle of Tinchebray in 1106 and died shortly afterwards. Henry I then gave the castle in 1122 to Robert of Gloucester, the king's illegitimate son and the husband of FitzHamon's daughter, Mabe. After the failed attempt of Robert Curthose, duke of Normandy, William the Conqueror's eldest son, to take England from Henry I, the duke was moved here in 1126 from imprisonment at Devizes and remained imprisoned in the castle until his death in 1134. Robert of Gloucester held the castle during the troubled years of the Anarchy in England and Wales, and passed it on to his son, William Fitz Robert. Around the middle of the century, possibly under Robert of Gloucester, a 77-foot (23 m) wide, 30-foot (9 m) high shell keep was constructed on top of the motte, along with a stone wall around the south and west sides of the inner bailey. The polygonal shell keep has architectural links to a similar design at Arundel Castle. The building work was probably undertaken in response to the threat posed following the Welsh uprising of 1136.
Tensions with the Welsh continued, and in 1158 Ifor Bach raided the castle and took William hostage for a period. A further attack followed in 1183. By 1184 town walls had been built around Cardiff, and the West Gate to the town was constructed in the gap between the castle and the river. William died in 1183, leaving three daughters. One of these, Isabel, Countess of Gloucester, was declared the sole heir to the estate by Henry II. This was contrary to legal custom in England, and was done in order that Henry could then marry her to his youngest son Prince John and thus provide him with extensive lands. John later divorced Isabel, but he retained control of the castle until she married Geoffrey de Mandeville in 1214.
Upon Isabel's death in 1217 the castle passed through her sister to Gilbert de Clare, becoming part of the Honour of Clare, a major grouping of estates and fortifications in medieval England. The castle formed the centre of the family's power in South Wales, although the de Clares typically preferred to reside in their castles at Clare and Tonbridge. Gilbert's son, Richard de Clare, 6th Earl of Gloucester, carried out building work at the castle in the late 13th century, constructing the Black Tower that forms part of the southern gateway seen today. On the ground floor the tower contained the Stavell Oged and Stavell Wenn chambers, with three rooms constructed above them. Richard was also probably responsible for rebuilding the northern and eastern walls of the inner bailey in stone. The inner bailey was reached through a gatehouse on the eastern side, protected by two circular towers and later called the Exchequer Gate. The defensive work may have been prompted by the threat posed by the hostile Welsh leader Llywelyn ap Gruffudd, Prince of Wales.
Richard's grandson, Gilbert de Clare, the last male de Clare, died at the Battle of Bannockburn in 1314 and the castle was given to Hugh Despenser the Younger, the controversial favourite of Edward II. Poor harvests and harsh governance by the Despenser family encouraged a Welsh rebellion under Llywelyn Bren in 1316; this was crushed and Llywelyn was hanged, drawn and quartered in Cardiff Castle in 1318 on Hugh's orders. The execution attracted much criticism from across both the English and Welsh communities, and in 1321 Hugh arrested Sir William Fleminge as a scapegoat for the incident, first detaining him in the Black Tower and then executing him in the castle grounds. Conflict between the Despensers and the other Marcher Lords broke out soon after, leading to the castle being sacked in 1321 during the Despenser War. The Despensers recovered the castle and retained it for the rest of the century, despite the execution of Hugh Despenser for treason in 1326. Under a 1340 charter granted by the Despensers, the castle's constable was made the de facto mayor of Cardiff, controlling the local courts.
Cardiff Castle