Last week the businessman was forced to give up the historic 14th century building after reportedly failing to pay the mortgage.
He and his former wife Kim purchased the 35-acre property near Grantown-on-Spey for £720,000 in 2006 with a 110 per cent mortgage of £800,000.
But she moved out after their marriage fell apart, and he was refusing to pay the £7,000-a-month mortgage for two years.
The Bank of Scotland raised a legal action against him last year after he failed to make the repayments from 2012.
The battle came to an end at Inverness Sheriff Court last Tuesday when Whyte finally agreed to repossession.
Now, estate agents Strutt & Parker are looking for a new owner for the castle, although they admit that a lot of work needs to be done.
Kevin Maley of Strutt & Parker’s Inverness office said: “It is clear that a number of improvements have been made, however a large part of the property remains undeveloped and a significant amount of investment will be required to finish this off.
“Castle Grant is a wonderful property which is absolutely ripe for development which could lead to it being one of the premier private houses in the UK.”
Castle Grant is the ancient ancestral seat of the the Chiefs of the Grant Clan.
The castle has had a number of additions added over the years, with the largest expansion taking place in the 1750s.
This latter expansion enclosed the main building, and attached the two outer buildings to the south, creating “wings” on the south side which now enclose the courtyard.
Mr Whyte sought help from his father Tom who supervised workmen as they cleared the castle of his belongings.
The clear-out included reportedly the siphoning off of oil from the central heating system and ripping chandeliers from the ceiling to be transported back to his London home.