Very little survives of the castle that gave Castleton its name. The surviving horseshoe shaped mound is cut through by the road that leads north out of the village. Its shape suggests that the castle was a shell keep, a form of castle that had stone walls built on top of a bank, with a hollow space in the centre. In many cases the shell keep was later filled in to become a motte, concealing its original shape.
Castleton Castle was built by Robert de Brus. He was given the Forest and Lordship of Danby by William Rufus in c.1089, and the castle was probably built soon after this. Adam de Brus lost the castle to Henry II, before regaining it in 1200. In 1271 the castle passed into the Latimer family when William Latimer acquired it through marriage.
The Latimers owned Castleton for most of the fourteenth century, until the last lord Latimer died without a son in 1381. His estates passed to his daughter Elizabeth, wife of John Neville of Raby, and at this point the castle probably began to be neglected in favour of the Neville's own castles.
Elizabeth was widowed in 1388. From then until 1396 Castleton was in her hands, and her son with Neville, another John, didn't inherit until after her death in 1396. This John Neville was not his father's oldest side - John Neville senior was much older than Elizabeth, and his heir, Ralph Neville, was some thirty years older than John junior. The core of John junior's estates were thus the lands he inherited from his month, which included Castleton and Danby.
At some point during this period Castleton Castle was abandoned in favour of the newly built Danby Castle, a typical late 14th century rectangular castle. We don't know exactly when the change took place - the new castle might have been built by John Neville of Raby to provide his new young son with a home, by the widowed Elizabeth, or by the younger John Neville after he finally came of age in 1400. Whichever of the three was responsible for the change, they probably used much of the stone from the old building at Castleton in their new castle, leaving just the earthworks behind.
Grid Reference: NZ 688 081
On eastern side of the road running north out of the village towards the station