The first mention of a castle at Cavers is a reference in 1311, in which it is mentioned in a list of castles with English constables. At this point, and for about 40 years previously, the lands of Cavers were held by the Balliol family; Alexander de Balliol of Cavers was the uncle of the unfortunate King John who had been first elected and then deposed by Edward I of England in the 1290s.
In 1321, a safe conduct was issued at Cavers by Sir James “the good” Douglas, brother-in-arms of King Robert the Bruce, but no mention is specifically made of the castle. Following the death of Bruce, it seems likely that Cavers fell back under English control in the 1330s, and it was not until 1358 that Cavers reappears in records, in the hands of the Earl of Mar. In 1368 the Earl of Douglas was recorded as Lord of the Barony of Cavers, perhaps in right of his wife, who was Mar’s sister.
The earl of Douglas died in 1384 of a fever, and his son James succeeded him as 2nd earl, only to be killed at the Battle of Otterburn four years later. Leaving no legitimate children, his mother Margaret was Countess of Mar and Douglas until 1391, undoubtedly having held the Barony of Cavers. She was succeeded by her daughter Isabel, and in her time a massive restructuring of the Douglas estates took place since much of the estate was entailed and could not be held by her. Cavers, however, was not in this entail, and in the early 15th century she granted it to her illegitimate nephew Archibald, one of two sons of the 2nd earl of Douglas. This was in dispute until 1412, when Archibald received a charter to the lands from King James I.
It is likely that Archibald, who held Cavers until 1432, and lived to at least 1455, built the earliest parts of the old tower at Cavers. He was an adherent of George Douglas, earl of Angus, and was not involved in the rebellion of the earls of Douglas which resulted in their extinction as a political force in the 1452-1455 war. As such, one might consider him a “Red” Douglas rather than a “Black” one, although the term was really reserved for reference to the earls.
Archibald and his successors remained heritable sheriffs of Roxburgh/Teviotdale (the terms were used interchangeably) and were intimately involved with much of the history of the central Borders during the 15th and 16th centuries. In 1547, James Douglas (VI) of Cavers signed allegiance to King Edward VI of England at Roxburgh along with many other border lairds including the Kerrs of Ferniehirst and Cessford, to whom he was related. Although the tower of the Douglases has in more recent times been referred to as “The Warden’s Tower”, it was the Kerrs who held the title of Wardens of the March, not the Douglases of Cavers.
Indeed, throughout the period, there is little mention of the castle, which is first mentioned as a “castrum” in 1511, perhaps because the sheriff was responsible for domestic issues, and the wardens for cross-border ones. There were numerous raids by the English armies into Teviotdale, and the tower was damaged in 1545. In 1596 the little town of Cavers was destroyed, never to be rebuilt. An image of the tower from c1590 was drawn by Timothy Pont, showing the view from the main approach (presumably from the north-east) as a tall tower with a courtyard wall along with the old kirk and perhaps an image of the town before its destruction.
From 1603 there is no evidence that Cavers suffered military damage again, and the tower was extended to the north to form a large rectangular house, with wings added on the western side to form a courtyard. Continuing to be held by the Douglases, it was repeatedly remodelled, with the earliest photographs from about 1860 showing that it had an Italianate mansion look, with all military styling removed. It is likely this took place in the earlier parts of the 19th century, and certainly after General Roy’s map of c1750 which shows it as a rectangular building only.
The fashion amongst the latter 18th and 19th centuries for bachelorhood meant that the estate of Cavers passed repeatedly to younger brothers and nephews, until in 1878 it passed to Mary Palmer Douglas, niece of the 21st laird. Under the Palmer Douglas family, the house was remodelled in baronial style and the wings and internal layout were substantially restructured. During this work, a mid to late 13th century piscina was discovered, and cemented into place in the external wall of the old tower, which has caused the degradation of the piscina, now in urgent need of conservation.
The Palmer Douglas family moved out of the castle into Midgard Lodge, and soon after Mary Palmer Douglas died, her grandson James took the decision to deroof the building after failing to secure a buyer. In the end, he reluctantly took the decision to allow the building to be used for explosives practice, leading to the demolition of the majority of the building. The old tower survived, along with part of the adjacent wing, and was eventually sold by James as the last surviving parts of the estate.