The early castle at Rothbury stood on a mound on the north bank of the Coquet, just to the west of the church. It was probably built in the 12th century. It stayed occupied until the mid-19th century, but in 1869 its walls and foundations were removed to extend the churchyard. There are now no remains to be seen.