The original stone keep at Yester, built before 1267, is generally considered to be by Sir Hugo de Giffard. A grandson of the first Laird of Yester, he served as a guardian of the young Alexander III of Scotland, and was by repute a magician and necromancer, known by many as "The Wizard of Yester". Little remains of the castle today, but the subterranean Goblin Ha' (Gobiln Hall) is still largely intact. Yester House is described as having 85 palatial rooms, set within 500 acres. In 2008, it was offered for sale with a price tag of £15 million, which made it then the most expensive house offered for sale in Scotland, three times the prevailing record, held by nearby Seton House, which sold for £5 million in 2007.
Sir Hugo de Giffard was known as the 'Wizard of Yester', and was considered to be a powerful warlock and necromancer. It is in the undercroft of the castle that he was thought to practise his sorcery. 14th century chronicler John of Fordun mentions the large cavern in Yester Castle, thought locally to have been formed by magical artifice. Major upgrades were made to Yester Castle in the thirteenth century. No later than 1267 a Stone Keep was built at the site probably by Sir Hugo de Giffard, grandson of the original owner. The upgrades included construction of the fine vaulted chamber known as the Goblin Ha' - a title deriving from the rumours that Hugo was a wizard whose armies of goblins had built Yester Castle. Legend also supposed that Hugo was able, via a pact with the Devil, to raise a magical army to his aid, and use them to carry out his will.
Goblin Ha', or Goblin Hall, lies to the east of the walled area once enclosed by Yester Castle, and is reached by a flight of steps. This subterranean hall is about 37 feet long, 13 feet wide, built of ashlar, and having a high pointed vaulted roof, 19 feet high. Another staircase leads down to a well and bolt-hole in a nearby gulley.