Birkwood Castle, a former psychiatric hospital near Lesmahagow, Lanarkshire, was being transformed in a multimillion-pound project and was due to open as a hotel next spring.
However the developers, who had backing from American investors and proposed a "stunning boutique hotel", went into administration in January and the site was put for sale.
A structural inspection has now revealed that the crumbling B-listed building is a major safety risk and recommended that much of it is flattened.
The hospital closed in 2002 and it fell into disrepair. In 2015, a year after it was bought by the developers, a tower and part of a wall fell down in a mystery collapse. It made such a noise that locals reported an explosion.
The administrators of Birkwood Estates Limited have submitted a planning application to South Lanarkshire Council asking for permission to demolish the vast majority of the structure.
The castle's bell tower and its front entrance will be retained after they were made safe by building contractors.
A report submitted to the local authority said: "Following a partial collapse of the castle, the proposal is to part demolish the remaining structure- retaining the Bell Tower and porte-cochere feature at the front entrance- in a controlled manner in conjunction with the downtaking of the structural scaffolding restraint system.
"Following extensive site investigation work, the structural engineers, concluded there are significant areas of the building beyond repair and consider the building, due to the recent deterioration in its condition, to be an unsafe working environment in which to carry out future stabilisation/development works."
Donald McNaught, of Johnston Carmichael, joint administrator of Birkwood Estates, said they were still hopeful of finding a buyer to develop the site.
He added: "I can confirm that as joint administrators of Birkwood Estates Ltd, we continue to explore all options to realise value for the site and welcome contact from any potentially interested party."
The £80million masterplan for the site included the creation of a 56-room hotel, holiday chalets and village with more than 150 residential homes.
The 86-acre estate originally belonged to the MacKirdy family and the castle was used as a private residence until it was donated to the local authority in 1923.
Former owners, residents and staff claim that the site is stalked by spectres, including a cigar-smoking spook and the spirit of a man stabbed through the throat.
Past incidents at the castle have included strange smells, lights turning themselves off and the sound of footsteps. According to local folklore, a boy called Michael is said to stalk the grand spiral staircase where he fell and died.
And it has even been claimed the figure of a doctor who suffered a heart attack and died while working at the hospital has been seen at windows.