withdrew from Britain in about AD 410, the Germanic Anglo-Saxons began their series of invasions of the island. As Wikipedia puts it:

Traditionally, it was believed that a mass invasion by various Anglo-Saxon tribes largely displaced the indigenous British population in southern and eastern Great Britain (modern-day England with the exception of Cornwall).

However, recent genetic studies show that there was not really a mass displacement at all. Rather, the relatively small groups of Anglo-Saxon invaders had a large cultural impact on the existing British population. The most notable theory to explain why the Anglo-Saxon culture had such a profound effect on the indigenous people is that the political and economic dominance of the Anglo-Saxons made their culture more desirable. Of course, what exactly gave the Anglo-Saxons an advantage isall the way into Wales, possibly because the Romans had not built as much infrastructure there.
The Romans were never able to invade much of Scotland, possibly because it was hard to fight the tribes in the Highlands. Eventually the Romans gave up their land in southern Scotland, and the territory was dominated by native tribes, such as the Picts in the north. After the Romans withdrew from Britain, the Gaels from Ireland invaded Scotland in the west, and the Picts withdrew to the east. Eventually, gaelicization (the influence of the Gaels), transformed Scotland, and the Gaelic and Pictish crowns merged. The strength of this new kingdom is what kept Scotland independent from the English in the south, and that is another topic that deserves further study by the