Skelton Castle was probably a very large fortress, but it was destroyed late in the eighteenth century, and little is known about its original layout. The castle was probably built soon after the conquest by the de Brus family. It is first mentioned in 1216, when it was still in the same family. The castle was probably used as a fortress and prison during the reign of King John In 1265 the castle was surrendered to Henry III by Peter de Brus, who the king suspected of being a support of his son Prince Edward (the future Edward I).
In 1272 the lands of the third Peter de Brus were split, and Skelton went to Walter de Fauconberg. The castle remained in the Fauconberg family for over two centuries, but by the end of the fourteenth century insanity stalked the family. In the 1349 inquisition post mortem (record of estates after the death of a landowner) of one of the Fauconbergs the castle was described as expensive to maintain. William Nevill, Lord Fauconberg, held the castle from his wife until his death in 1462-3. It was then held in her name until her death in 1490, after which it was inherited by her grandson William Conyers. The castle was described as ruinous and of no value in her inquisition post mortem.
The castle passed into the hands of the Hall family, who despite two changes of name held it until it was demolished. In the 1730s the current Hall adopted the name Hall Stevenson. One of his descendents, John Hall Stevenson, a friend of Stern, wrote a poem describing the castle, which he called 'Crazy'. This poem included mention of a round tower being used as a dovecote and a nearly ruined keep, with most of the castle built up on a terrace.
After that the castle was inherited by the Trotter family and then the Hall family. The Halls became the Hall Stevensons in the mid eighteenth century and then the Wharton family towards the end of the century, in both cases simply by changing their name. The last Hall Stevenson adopted the name Wharton, and it was he who was responsible for the demolition of the castle and its replacement with a mock gothic house.
The exact layout of Skelton castle is unknown. It was built on a spur of land to the south of Skelton Beck, separated from the village of Skelton itself by a small beck. Our best evidence comes in the form of a letter from a clergyman recorded in Ord's History of Cleveland (1846), accompanied by a sketch of the castle. Our clergyman claims to have watched the castle being demolished in 1788, sixty years before Ord was writing. His sketch shows the castle as being on a terrace, with a low curtain wall, possibly square in form. Square towers are show at two of the visible corners, a round tower (also said to be a dovecote) at the front-right. A keep is shown on the left.