The Bronze Age settlers left evidence of several small oval houses with thick stone walls and various artefacts including a decorated bone object. The Iron Age ruins include several different types of structure including a broch and a defensive wall around the site. The Pictish period provides various works of art including a painted pebble and a symbol stone. The Viking-age ruins make up the largest such site visible anywhere in Britain and include a longhouse; excavations provided numerous tools and a detailed insight into life in Shetland at this time. The most visible structures on the site are the walls of the Scottish period fortified manor house, which inspired the name "Jarlshof" that first appears in an 1821 novel by Walter Scott.
The name Jarlshof meaning "Earl's Mansion" is a coinage of Walter Scott, who visited the site in 1814 and based it on the Scottish period name of "the laird's house". It was more than a century later before excavations proved that there had actually been Viking Age settlement on the site, although there is no evidence that a Norse jarl ever lived there.
The remains at Jarlshof represent thousands of years of human occupation, and can be seen as a microcosm of Shetland history. Other than the Old House of Sumburgh (see below) the site remained largely hidden until a storm in the late 1800s washed away part of the shore, and revealed evidence of these ancient buildings. Formal archaeological excavation started in 1925 and Jarlshof was one of two broch sites which were the first to be excavated using modern scientific techniques between 1949–52. Although the deposits within the broch had been badly disturbed by earlier attempts, this work revealed a complex sequence of construction from different periods. Buildings on the site include the remains of a Bronze Age smithy, an Iron Age broch and roundhouses, a complex of Pictish wheelhouses, a Viking longhouse, and a mediaeval farmhouse. No further excavations have been undertaken since the early 1950s and no radiocarbon dating has been attempted.
There is also evidence of a cattle stall with a waste channel leading to a tank in a courtyard and a whale vertebra set into a wall that may have been used as a tethering post. Broken moulds from the smithy indicate that axes, knives, swords and pins were produced there and a bronze dagger was found at the site. The objects indicate the smith was trained in the Irish style of working. Bone pins and awls also survive and an extraordinary bone "plaque". This latter object is long, has three holes bored into the ends and is decorated with various linear patterns. Its function is unknown. The Bronze Age structures are overlain with sterile sand, suggesting a break in occupation prior to the next phase of building.
The inhabitants of the Iron Age built part of their settlement on top of the Bronze Age one. The structures include a complex roundhouse, replaced at a later stage by an "aisled roundhouse". Neither have been dated although artefacts found at this level include querns that suggest the latter may have been constructed prior to 200 BC.
It is in this period that the broch was built. Part of the structure has been lost to coastal erosion, and modern sea defences have been erected. The tower was probably originally 13 metres (40 feet) or more high and as with many broch sites the position would have commanded fine views of the surrounding seas. During this period archaeological sites in Shetland usually exhibit defensive fortifications of some kind, and Jarlshof is no exception. An outer defensive wall associated with the broch contained a substantial (although rather poorly constructed) house and byre at one time. This wall was utilised at a later stage to build a large roundhouse in the lee of the broch.
The wheelhouse complex came later, and post-dates the 1st century BC–2nd century AD profusion of these structures in the Western Isles by several centuries. Its construction used the stones of the broch itself and two of the four main structures are amongst the best examples of their type. Three successive periods of construction were undertaken, and the best preserved retains a significant proportion of the stone part of its roof and displays a series of corbelled bays. One structure was built as a circular building and the radial piers were inserted afterwards. This may have been an earlier, less stable design. In one case the piers are alternately rectangular and V-shaped, in another all are to the latter design, again suggesting a developing style. Unlike many wheelhouses elsewhere in Scotland that are built into the earth, the Jarlshof structures seem to have been built from ground level upwards.
Amongst the artefacts dated to the later Pictish period is a bone pin with a rounded head probably used as a hair or dress pin. It has been dated to AD 500–800. "Painted pebbles" are associated with more than two dozen Pictish sites and one such stone was unearthed at Jarlshof. This rectangular slate fragment had a cross painted onto it and two small "S" shaped scrolls suggesting an association with Christian beliefs. One of only two Pictish symbol stones found in Shetland was found here, exhibiting a double disc shape and a Z-rod. Pottery finds include buff ware from the period after AD 10, including bowls with flat rims. The quality of the pots appears to decline in the period prior to Viking settlement, becoming thinner-walled and generally more crude in design.
There are seven Norse-era houses at Jarlshof, although no more than two were in use at one time. There were several outbuildings, including a small square structure with a large hearth that may have been a sauna and which was later replaced by two separate outhouses. The largest house from this period is a by rectangular chamber with opposing doors, timber benches along the long sides, and a hearth in the centre. Unlike the earlier structures that had conical thatched roofs, those of the Norse buildings had ridged timber frames. At a later period this large structure was also used to shelter domesticated animals (at which stage it had a paved centre and animal stalls along the sides) and later still may have become an outbuilding. The door to the byre puzzled archaeologists as it appeared to be too narrow to admit a cow. The mystery was solved when a byre door was excavated at Easting on Unst which had a narrow base similar to Jarlshof's but which widened out to become cow-shaped. Another outbuilding has been interpreted as a corn-drying room. Later houses were built at 90 degrees to the longhouse and these are of a type and size that is similar to croft houses that were common in Shetland until the mid-nineteenth century.
One hundred and fifty loom weights were found suggesting wool was an important aspect of Norse-era life. Line weights from the later Norse period and associated evidence from elsewhere in Shetland indicates that deep-water fishing was also a regular undertaking. The Jarlshof site also produced ample evidence of the use of iron tools such as shears, scissors, sickles, and a fish-hook and knife. The ore was locally obtained bog iron. Hazel, birch and willow grew in the area at this time but the pine and oak must have been driftwood or imported timber.
Drawings scratched on slate have been found of dragon-prowed ships, portraits of an old man and of a young, bearded man and of a four-legged animal. The drawings were found in the Viking levels but are Pictish in style and may either pre-date the arrival of the Norse or indicate a continuity of art and culture from one period to the next. Similarly, although the rectangular shape of the Norse-era buildings are quite unlike the earlier rounded Pictish style, the basement courses of the two periods are constructed in the same way. The Viking-style loom weights, spindle whorls and other vessels were found with stone discs and other objects of a Pictish design. A bronze-gilt harness mounting made in Ireland in the 8th or 9th centuries has also been found and many items from this period are in the Shetland Museum. Jarlshof contains the most extensive remains of a Viking site visible anywhere in Britain.
The castle, now known as Jarlshof House, was built during the Scottish period. Originally a medieval stone farmhouse, it was converted into a fortified house during the 16th century, by Robert Stewart, 1st Earl of Orkney after Scotland annexed Shetland. The building was named "New Hall" at this time. It was further modernised in the early 17th century by his son Patrick Stewart, 2nd Earl of Orkney who re-named it the "Old House of Sumburgh" but it was abandoned in the late 17th century. The structure was also formerly known as "The laird's house" and "Stewart Mansion".